Luminous Landscape Forum

Raw & Post Processing, Printing => Digital Image Processing => Topic started by: bwana on March 14, 2012, 10:05:41 am

Title: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 14, 2012, 10:05:41 am
As my coarse understanding of digital photgraphy is still new, please be patient with me. The image sensor of a digicam sits behind a bayer filter. Each image pixel is derived from four photosites. Each photosite behind a colored filter so that a pixel is constituted by 4 photosites-2 green, 1 red and 1 blue. This pixel is then the fundamental data unit transmitted to the pc. When displaying to a screen, (which consists of red, green and blue phosphors - or other type of color element) the pixel is broken down into color elements by the video driver. These are then sent to the screen for display.

What if the raw image file was used so that the data from each photosite was converted to a pixel. Of course, all color information would be lost since the only data each photosite could yield is luminance. But for black and white photography, only luminance is needed. The computer would then receive data from 4x as many pixels since we have now changed the mapping from 4 photosites-> 1 pixel to 1 photosite -> 1 pixel.

I tried to read about raw file formats, TIFF, but the literature is rather opaque. Is this concept valid or flawed?
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: deejjjaaaa on March 14, 2012, 12:54:30 pm
The computer would then receive data from 4x as many pixels since we have now changed the mapping from 4 photosites-> 1 pixel to 1 photosite -> 1 pixel.

actually (in general, certain special demosaicking methods excluded) one sensel is contributing to 4 resulting pixels, not to 1 pixel... 
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 14, 2012, 01:55:19 pm
I think of the sensor as the cookie dough and the pixel as the cookie cutter. To make properly shaped cookies, you cant overlap the cookie cutter placement. However I get the sense that you are saying that pixels are derived from overlapping photosite sampling. In other words, are you saying that each photosite contributes data to ALL the surrounding pixel groups? I dont really know what is actually being done during demosaicising in terms of the specific algorithm.  Even so, that would diminish resolution since you never have 1 photosite->1 pixel mapping.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: EricV on March 14, 2012, 02:14:42 pm
When a digital color image is mapped from raw pixels to processed pixels, the mapping is 4 pixels (R,G,B,G) -> 4 pixels (RGB,RGB,RGB,RGB).  Yes, there is necessarily some loss in resolution, due to the color interpolation, but less than a factor of four.  If you simply treat each raw pixel value as a luminance, rather than a color, you will get very strange color effects.  For example, a bright red apple will light up only one pixel out of four, which is hardly the result you want.  To really achieve better resolution for B&W, the proper solution is to remove the Bayer filter, so that every pixel sees all colors. 
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 14, 2012, 03:31:00 pm
ok, i see your point-yes the image might look pepperred with luminance fluctuations that would ruin the extra resolution. OTOH, if you knew how much luminance each color of the Bayer filter absorbed, you could compensate for that. For example, assume the green bayer filter absorbs 10%, the red 20% and the blue 20%. You could adjust the luminance measured by each photosite according to the bayer filter in front of it. So, this seems straightforward, cool if i could write a javascript implementation-that way it could run in a browser.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: John Nollendorfs on March 14, 2012, 06:07:32 pm
Or, you could just average all the luminance values, but the end result would hardly qualify as increasing resolution. The problem with your suggestion of course is the Bayer filters. And since that's integral in the manufacturing process of the sensor, it can't be removed. Spend your time dreaming  about a 41 MP phone camera instead! ;-)
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 14, 2012, 06:35:42 pm
Or, you could just average all the luminance values, ...

what? I dont get it. i said nothing about averaging the luminance values. My point is that the averaging that is being used in the Bayer demosaicising algoritm is what decreases resolution. By eliminating averaging and going to 1 photosite-> 1 pixel, we increase resolution.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Nigel Johnson on March 14, 2012, 08:37:37 pm
…if you knew how much luminance each color of the Bayer filter absorbed, you could compensate for that. For example, assume the green bayer filter absorbs 10%, the red 20% and the blue 20%. You could adjust the luminance measured by each photosite according to the bayer filter in front of it. So, this seems straightforward…

The thing that you are missing is that the absorption of each of the filters is dependent upon the colour of the light at each of the pixels. For example the green filter lets most green light pass through, but little magenta light - therefore a low luminance at the pixel behind the green filter can be caused by, amongst many other possibilities, a dark green or a bright magenta. Similar problems apply to the red and blue filters. Thus, for a single pixel, luminance cannot be calculated without knowing the colour of the light (to be fully accurate without knowing the filter's spectral response, the pixel's spectral response and the spectrum of the light falling on the pixel). The de-mosaicing process uses data from several pixels to estimate the colour and luminance at each of the pixels; for most sensors this process also has to account for the effect of the anti-aliasing filter which spreads the incident light between adjacent pixels.

Regards
Nigel

P.S. Whilst sensors without Bayer filters are available for scientific imaging purposes I do not believe that any new normal photographic cameras are currently available without such filters; although such cameras have been offered in the past. They tended to sell in small numbers and be expensive and their utility has reduced with higher resolution and more sensitive sensors and better de-mosaicing (the sensitivity comment is due to the fact that the omission of the Bayer filters allowed for higher sensitivity - important when sensors were very noisy at even low ISO).
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 14, 2012, 09:42:13 pm
tnx for the example and helping me follow through on my flawed logic. put another way, if i shine red light  through a bayer filter, i will not get a homogeneous gray tone on the sensor - . The filter absorbs different light frequencies differently. i need to know the frequency of the light so i can adjust the luminance of what's coming through each of the individual bayer filter-lets. And with a real image, with all those different light frequencies at the different locations in the image, that's impossible.

Even if I assumed that incident light on the bayer filter was 6500 K (when in fact it is not), the net result of the calculated luminance would appear as a speckled image.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: AlanPezzulich on March 14, 2012, 11:23:25 pm
Look at it this way. Each photosite produces one pixel. Only one color is measured at the site. The other two colors are calculated from adjacent photosites.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Bart_van_der_Wolf on March 15, 2012, 04:50:11 am
Even if I assumed that incident light on the bayer filter was 6500 K (when in fact it is not), the net result of the calculated luminance would appear as a speckled image.

The net result is not an image ..., it's a collection of color filtered sample data. The band-pass samples are complemented for the color info that was not sampled at that position, in a process called Bayer CFA demosaicing. Therefore each sample position will end up with a full RGB representation at each sample position and which is partially based on measured data, and partially based on advanced interpolation from many (sometimes more than 8) of the immediately surrounding data samples.

Luminance interpolation can be very accurate because each sample position records some luminance, whereas chroma interpolation will have reduced accuracy due to the lower sampling density. Fortunately, chroma information in real image scenes doesn't fluctuate as rapid as luminance information, so a Bayer CFA can produce very good images with high luminance resolution (which is the more important feature for the human visual system).

Cheers,
Bart
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Nigel Johnson on March 15, 2012, 04:39:57 pm
…i need to know the frequency of the light so i can adjust the luminance of what's coming through each of the individual bayer filter-lets.

A further complication is that very little natural light is monochromatic (ie only contains light of a single frequency) the majority of light contains a mixture of many frequencies either smoothly varying in intensity with frequency (eg natural light, incandescent 'tungsten' bulbs) or having a very spiky spectrum with most energy coming from a few frequencies (eg sodium vapour lights, most fluorescent lights), much rarer are monochromatic light sources (eg lasers, most LEDs). In addition objects illuminated by the light will also reflect the spectrum with varying degrees of smoothness meaning that the light picked up by the camera will contain varying proportions of different frequencies of light; the light from a highly saturated colour will tend to have most of the energy close to a single frequency (only a fully saturated colour would show light of a single frequency and these are extremely rare) whilst a less saturated colour will have a much wider spread of frequencies for a single colour (an example of this can be seen on the Cambridge in Colour site at http://www.cambridgeincolour.com/tutorials/color-perception.htm (http://www.cambridgeincolour.com/tutorials/color-perception.htm). This site includes other useful tutorials.

Regards
Nigel
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Nigel Johnson on March 15, 2012, 04:46:41 pm
For those who don't count in smileys, the third sentence in Bart's message should have read:

Therefore each sample position will end up with a full RGB representation at each sample position and which is partially based on measured data, and partially based on advanced interpolation from many (sometimes more than 8) of the immediately surrounding data samples.

Regards
Nigel

(Bart, not certain if you are aware that ticking the 'Don't use smileys.' check box under 'Additional Options…' in the Post Reply window prevents text being converted to smileys.)
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Bart_van_der_Wolf on March 15, 2012, 07:30:44 pm
(Bart, not certain if you are aware that ticking the 'Don't use smileys.' check box under 'Additional Options…' in the Post Reply window prevents text being converted to smileys.)

Hi Nigel,

Thanks for catching that. I just didn't have/take the time to read my own response.

Cheers,
Bart
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 15, 2012, 08:31:33 pm
thank you all for your extensive and complete replies. so, is there a piece of software that will let me see what is on the image sensor, unprocessed? I am just curious. I tried imageJ (from the NIH) but it doesnt read common raw photo file formats.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: ejmartin on March 16, 2012, 12:18:08 am
thank you all for your extensive and complete replies. so, is there a piece of software that will let me see what is on the image sensor, unprocessed? I am just curious. I tried imageJ (from the NIH) but it doesnt read common raw photo file formats.

http://www.rawdigger.com/
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 16, 2012, 09:00:29 am
http://www.rawdigger.com/

Thank you ejmartin. I've been looking for something like this for a while to avoid the layers of crap processing that obscure the original data on the sensor. In addition, the intelligent and honest presentation of the histogram and it's limitations is enlightening. Perhaps it is the Russian psyche that enables deep thinking and clear analysis- something that millions of native English speakers who inhabit the net have not matched. Or perhaps my knowledge is so limited that I do not know if these thoughts have been articulated previously. Another source I find enlightening in an introductory level is cambridgeincolour. As well as specific discussions by unique individuals on this site, as yourself.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: joofa on March 16, 2012, 03:46:51 pm
thank you all for your extensive and complete replies. so, is there a piece of software that will let me see what is on the image sensor, unprocessed? I am just curious. I tried imageJ (from the NIH) but it doesnt read common raw photo file formats.

Photoshop Mac users can use a freely available Raw Import plugin from the website in my signature that will let you (1) see the actual raw data (2) separate out the raw data into 4 R,G,G,B images, or (3) do a rudimentary demosaic outside of ACR. For more information please see the following link:

http://www.luminous-landscape.com/forum/index.php?topic=20388.msg483963#msg483963

Joofa
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Damir on March 17, 2012, 07:17:02 am
As my coarse understanding of digital photgraphy is still new, please be patient with me. The image sensor of a digicam sits behind a bayer filter. Each image pixel is derived from four photosites. Each photosite behind a colored filter so that a pixel is constituted by 4 photosites-2 green, 1 red and 1 blue. This pixel is then the fundamental data unit transmitted to the pc. When displaying to a screen, (which consists of red, green and blue phosphors - or other type of color element) the pixel is broken down into color elements by the video driver. These are then sent to the screen for display.

What if the raw image file was used so that the data from each photosite was converted to a pixel. Of course, all color information would be lost since the only data each photosite could yield is luminance. But for black and white photography, only luminance is needed. The computer would then receive data from 4x as many pixels since we have now changed the mapping from 4 photosites-> 1 pixel to 1 photosite -> 1 pixel.

I tried to read about raw file formats, TIFF, but the literature is rather opaque. Is this concept valid or flawed?

It will be valid if there is no Bayer filter. As Bayer filter already distort data e.g. no information in blue if you take photo of yellow flower, information in blue photosite need to be restored from red - green photosite, therefore you need to do demosiac.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: t6b9p on March 17, 2012, 11:58:21 am
Quote
The thing that you are missing is that the absorption of each of the filters is dependent upon the colour of the light at each of the pixels.

True, but if you work with 830nm B&W IR things change and this is a topic I have been interested in for a while with regards to IR 830nm converted DSLR. Beyond this wavelength the Bayer dyes react much more uniformly to changing wavelength as they become more transparent to IR. The suggestion of a "fixed" ratio correction for the Bayer mosaic in an undemosiaced image seems to be more of a possibility in this application.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 17, 2012, 12:27:30 pm
tnx pointing me to your raw import plugin in your sig, djjoofa.

I am using Mac OSX SnowLeopard and PS cs5 and importing Fuji raf files taken with a Fuji x10 camera.
This camera has the EXR sensor which simultaneously captures two offset 6 mp exposures whose ev differ  by one stop. It recombines these exposures to allow increased dynamic range.
Interestingly, photoshop creates 5 different images when i open up one raf file.Find it here:
http://dl.dropbox.com/u/29887336/fuji%20x10%20test%20image/DSCF1215.RAF

One large image is full frame, 2848x2144 pixels, wherein you can clearly see the bayer array when viewing the image at 400% magnification. Then there are 4 additional images at 1424x1072, which probably represent the individual GRGB sets of the CFA.

I find many interesting things comparing these images-for example the smooth clear blue sky actually is represented by pixels whose luminance varies. Even when I open up the the processed jpg (http://dl.dropbox.com/u/29887336/fuji%20x10%20test%20image/DSCF1215.JPG),
 the sky is 'mottled'-but only of course if you 'pixel peep'.

I was hoping to get 'increased resolution' by averaging out the luminance fluctuation of the bayer array filter-lets-eliminating the screen door effect but keeping the 'pristine' photo information. Alas, as pointed out by others above, the actual luminance fluctuation depends on the frequency of the incident light, so there is no simple way to get this. In fact the processed jpg (when desaturated to give a b&w image) is better than the raw image. The concept of looking at IR frequencies to minimize spectral absorption differences of the CFA elements is interesting as just pointed out. But I imagine this would require an IR illminator and a wratten 87a filter. Without the illuminator, the photographs would be rather dark and require long exposures. Of course, with todays new sensors capable of iso 25600 with low noise(take that tri-X!), one might imagine some awesome pics. Anyway, it would make an interesting photograph printed large. Thank you
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: t6b9p on March 17, 2012, 12:54:37 pm
Quote
But I imagine this would require an IR illminator and a wratten 87a filter. Without the illuminator, the photographs would be rather dark and require long exposures.

DSLR sensors are inherently sensitive to IR, remove the ICF/AA and replace it with the appropriate thickness 830nm filter and exposure times are fine. IR "illuminator" = sunlight/daylight

Outdoor in sunlight, using an 830nm converted D200, ISO 200, f/8, ~1/40s.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: bwana on March 17, 2012, 04:31:57 pm
well, after seeing what's really on the sensor, i realize how confused i really was. A D800 only has 36 million photosites so there's no way the resolution can be increased. the pixel averaging of demosaicising may add chroma info where there was none before. no new pixels can be made.and not much luminance accuracy is lost.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: joofa on March 18, 2012, 01:35:25 am
tnx pointing me to your raw import plugin in your sig, djjoofa.

You are most welcome.

Quote
Interestingly, photoshop creates 5 different images when i open up one raf file.Find it here:
http://dl.dropbox.com/u/29887336/fuji%20x10%20test%20image/DSCF1215.RAF

One large image is full frame, 2848x2144 pixels, wherein you can clearly see the bayer array when viewing the image at 400% magnification. Then there are 4 additional images at 1424x1072, which probably represent the individual GRGB sets of the CFA.

Yes, the 2848x2144 pixels image is the actual unDemosaiced raw image data. The additional four 1424x1072 images are the separate GRGB planes from the CFA. If you just want to get the unDemosaiced raw image data without the 4 CFA planes you can turn off the "RGBG" checkbox in the RawImport dialog box.

Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: RFPhotography on March 27, 2012, 09:36:10 am
You won't get a quadrupling of resolution.  No one has mentioned the Foveon sensor.  If memory serves, it shows about a 1/3 to 1/2 increase in detail resolution over Bayer sensors (e.g., a 12MP Foveon shows resolution equivalent to about 16 to 18MP Bayer sensor). 
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Bart_van_der_Wolf on March 27, 2012, 10:59:49 am
You won't get a quadrupling of resolution.  No one has mentioned the Foveon sensor.  If memory serves, it shows about a 1/3 to 1/2 increase in detail resolution over Bayer sensors (e.g., a 12MP Foveon shows resolution equivalent to about 16 to 18MP Bayer sensor). 

Hi Bob,

The Sigma cameras with a Foveon sensor don't use an AA-filter. That's what causes the higher signal level in the highest spatial frequencies. Resolution is determined by the sampling density and the System MTF.

Cheers,
Bart
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: RFPhotography on March 27, 2012, 11:06:34 am
Very true, Bart!  And I should have said that too.  Apologies for the confusion and thanks for the clarification.
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: hjulenissen on April 02, 2012, 05:52:06 am
Hi Bob,

The Sigma cameras with a Foveon sensor don't use an AA-filter. That's what causes the higher signal level in the highest spatial frequencies. Resolution is determined by the sampling density and the System MTF.

Cheers,
Bart
If your scene contains lots of uncorrelated information in the color channels, a Foveon-style sensor should be able to resolve more luminance information than a similar pitched Bayer-style sensor, regardless of AA-filter. If your scene is truely wide-band "colorless", then Bayer should have no additional loss of spatial resolution.

I guess that it is possible to remove the CFA if you contact a company like maxmax, at least for some cameras? This should give a significant increase in luminance resolution on top of the increase coming from removing the aa-filter. This sounds similar to the idea that the thread starter had in mind?

-h
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Bart_van_der_Wolf on April 02, 2012, 07:03:54 am
I guess that it is possible to remove the CFA if you contact a company like maxmax, at least for some cameras? This should give a significant increase in luminance resolution on top of the increase coming from removing the aa-filter. This sounds similar to the idea that the thread starter had in mind?

Hi h,

My experiments (http://bvdwolf.home.xs4all.nl/main/foto/bayer/bayer_cfa.htm) point to the conclusion that only a modest (<7%) increase in luminance resolution can be expected.

Cheers,
Bart
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: hjulenissen on April 02, 2012, 07:22:42 am
Hi h,

My experiments (http://bvdwolf.home.xs4all.nl/main/foto/bayer/bayer_cfa.htm) point to the conclusion that only a modest (<7%) increase in luminance resolution can be expected.

Cheers,
Bart
Am I right that your test is purely a computer simulation with equal modulation in all color channels?

If you had repeated the test with highly decorrelated color channels, or an extremely hot/cold spectral balanced scene, I suspect that you would have gotten a larger improvement.

-h
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Bart_van_der_Wolf on April 02, 2012, 08:30:46 am
Am I right that your test is purely a computer simulation with equal modulation in all color channels?

Yes, that's correct. Luminance, and in particular that of edges, is the most important feature for human visual acuity. In addition to that, in nature it is common for chrominance information to fluctuate slower than Luminance. Just look at the channels of a Lab mode image to confirm that.

Quote
If you had repeated the test with highly decorrelated color channels, or an extremely hot/cold spectral balanced scene, I suspect that you would have gotten a larger improvement.

Sure, but how relevant would such an unlikely combination of color and(!) brightness be? I know how to devise a worst case scenario, I even created a Blue/Red version of my test chart (http://www.openphotographyforums.com/forums/showthread.php?t=13217) (see the links in the P.S. of the first message) that attempts to create such a situation for RGBG Bayer CFAs. But again, that would be looking for an improbable case because in nature we rarely see colors from opposite ends of the spectrum side-by-side, and even then they need to be of equal luminosity to cause a problem for CFA demosaicing.

Each of the R/G/B filtered sensels contributes (although weighted) to the luminance info that is used for the demosaicing process. That can be verified by looking at the R/G/B MTF curves of a demosaiced image, the curves align almost perfectly for luminance input. Here is a copy from output of a test on the Imatest website (look at the (RGB) read-out in Cy/Pxl ):
(http://www.imatest.com/images/MTF_EOS-40D_EF1785_26mm_f56.gif)

Cheers,
Bart
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: hjulenissen on April 02, 2012, 09:19:04 am
Sure, but how relevant would such an unlikely combination of color and(!) brightness be?
And therein lies the success of the Bayer sensor, I guess: despite possible theoretical shortcomings, it tends to work well for the kind of images that we tend to care about.

It is nevertheless interesting to be aware of the conditions where Bayer is less ideal.

-h
Title: Re: remove the Bayer data and quadruple your resolution in B&W?
Post by: Christoph C. Feldhaim on April 02, 2012, 12:34:33 pm
The principle of a filtered sensel array is ingenious for a simple reason:
It allows to capture a great amount of information with a minimum of data junk.

We perceive resolution mainly through the luminance information, especially from the green channel.
This lies in the physiology of our visual system (retina rods are manly green sensitive).
The bayer patter exploits this and the result is a lot of information with a reasonably small file size at the RAW stage.

Just imagine all your raw files being three times as big, which would be the case if each sensel would record all color information, like the foveon sensor. The reasons above are also the reason why foveon is NOT equivalent to three times the resolution of a Bayer patterned sensor as has been discussed to death.

In the essence the filtered color arrays (not saying Bayer here, since Fuji has other patterns with the same effect) do a sort of information compression, which is similar to jpegs: reduce color information, but keep luminance information.

Thats just totally awesome.

~Chris