-
Foveon technology was something I wanted to win vs bayer many years ago. It seemed like a more elegant solution to color photography. Now they are so far behind cameras like the D800 that you could downrez your shots 50% for detail retaining a far bigger image for the print than the dp2m. Its too little too late IMO.
So my question is about foveon vs chips with no color filters. Yes, they are made, many small chips are used with telescopes for long astro-photo shots. They have a filter wheel so you could take 7 shots for each image. A pure luminance, R,G,B,C,M,Y as the motor run wheel advances, giving you what must be perfect color capture at every point.
When you absolutely, positively, must have the most perfect color why would people choose foveon vs a technical camera with filter wheel and colorless chip?
Forget moving subjects, foveon is slow. We are talking landscapes, studio, architecture shots here. Thoughts?
-
When you absolutely, positively, must have the most perfect color why would people choose foveon vs a technical camera with filter wheel and colorless chip?
There are filterless camera with filter wheels (or more sophisticated stuff for color bandwidth selection) by many manufacturers out there. Normally, they are outside the range of photographers monetarily, and more importantly, perhaps ordinary photographers would not find very many uses of them to justify the cost.
-
Thoughts?
They're just getting started with the DP2M. It's far from too late, it's just beginning.
-
Sigma is up to something as they are refreshing many of their lenses it seems. I would love to see a full sized foveon as I believe in the technology. Still if sigma is the only camera Company to market it i don't believe they enough market share for it to take off. Time will tell.
Paul
-
The foveon patent is almost expired. Anything they do now would be to advance a new one or get other companies to adopt their product. If other companies like the tech they could soon advance their own update based on their R&D.
-
A quick google of "foveon patent" shows Sony, Canon, Fuji, Panasonic (just page 1) are all advancing a 3 layer sensor of their own.
-
A calculated expiration date by Patent Lens is June 18, 2021. :(
Eduardo
A quick google of "foveon patent" shows Sony, Canon, Fuji, Panasonic (just page 1) are all advancing a 3 layer sensor of their own.
-
You should specify which patent you are talking about.
I pulled one up by "foveon"
US 5965875 (granted patent)
Inventors
Richard Billings Merrill, Woodside CA
Assignees
Foveon, Inc., Santa Clara CA
Filing Date
Apr 24, 1998
Publication Date
Oct 12, 1999
Predicted Expiry Date
Apr 24, 2018
There were at least 16 pages of returns with many years listed. The above is one I opened for 1999 at random.
-
I have a hard time seeing the obsession with getting rid of Bayer. It has given us high spatial resolution in well-balanced sensors at moderate cost. Nothing suggests that the evolution of Bayer sensors will stop anytime soon.
Say that (in a few years) you could have a 40MP or 200MP or 2000MP Bayer sensor; in practice, any human-designed lense would eventually be the effective bottleneck in terms of spatial resolution, and the OLPF could be dropped. Bayer attributed artifacts would tend to be shifted to a size that would warrant meter-size prints and a loupe to be seen. With the usual good color fidelity, high DR at base ISO, good high-ISO performance. What is wrong with such an offering? Why would you want something different?
-h
-
I have a hard time seeing the obsession with getting rid of Bayer. It has given us high spatial resolution in well-balanced sensors at moderate cost. Nothing suggests that the evolution of Bayer sensors will stop anytime soon.
Say that (in a few years) you could have a 40MP or 200MP or 2000MP Bayer sensor; in practice, any human-designed lense would eventually be the effective bottleneck in terms of spatial resolution, and the OLPF could be dropped. Bayer attributed artifacts would tend to be shifted to a size that would warrant meter-size prints and a loupe to be seen. With the usual good color fidelity, high DR at base ISO, good high-ISO performance. What is wrong with such an offering? Why would you want something different?
-h
I'm sure the R&D teams knew the theoretical limits when they started giving us 1MP cameras. They just dole it out one carrot at a time. Clearly they are making tiny pixels for point and shoots with no issues at very low cost. They could have given you 200MP 135 format years ago.
-
I have a hard time seeing the obsession with getting rid of Bayer. It has given us high spatial resolution in well-balanced sensors at moderate cost. Nothing suggests that the evolution of Bayer sensors will stop anytime soon.
+1
Don't get it.
-
I have a hard time seeing the obsession with getting rid of Bayer. I... Why would you want something different?
I agree that in practice, no rival to the mainstream CFA approach (one color measured per sensor location) has made a case for superiority, so it could be like the theoretical advantages of rotary engines never winning out over the practical superiority of the conventional but vigorously developed reciprocating engine approach. Partly because, as you indicate, issues like color moiré can be handled by further development of CFA sensors, like a suitable blend of "oversampling" (deliberate use of very small photosites and diffraction) and sharpening algorithms.
But let me at least mention one potential advantage of X3, at least if a better approach than Sigma/Foveon's arrives: improved sensitivity through measuring more of the light arriving at each photosite, by measuring light of the "other two colors" rather than ignoring it. This could in principal gain about one stop or more in sensitivity.
-
I agree that in practice, no rival to the mainstream CFA approach (one color measured per sensor location) has made a case for superiority, so it could be like the theoretical advantages of rotary engines never winning out over the practical superiority of the conventional but vigorously developed reciprocating engine approach. Partly because, as you indicate, issues like color moiré can be handled by further development of CFA sensors, like a suitable blend of "oversampling" (deliberate use of very small photosites and diffraction) and sharpening algorithms.
But let me at least mention one potential advantage of X3, at least if a better approach than Sigma/Foveon's arrives: improved sensitivity through measuring more of the light arriving at each photosite, by measuring light of the "other two colors" rather than ignoring it. This could in principal gain about one stop or more in sensitivity.
Just switching from 3 colors to a 4 pixel sub-array of RGB+clear would increase sensitivity and dynamic range. The white pixel would be like the fuji mini-pixel.
-
But let me at least mention one potential advantage of X3, at least if a better approach than Sigma/Foveon's arrives: improved sensitivity through measuring more of the light arriving at each photosite, by measuring light of the "other two colors" rather than ignoring it. This could in principal gain about one stop or more in sensitivity.
Hi,
Unfortunately that doesn't work that way. Instead of actually sampling the 2 missing channels in a Bayer CFA, they are interpolated. No sensitivity is lost, the interpolated signal (which is slightly less accurate) is added to the luminosity.
Cheers,
Bart
-
That doesn't sound right. In foveon, if the color is there it is collected. In Bayer it is the same except only partial light of non-primary colors is collected. For a neutral color close to white or a grey, foveon would collect all 3 colors. Bayer would reject light at a rate of 2/3rds at every point.
Foveon is not as good at keeping the electronics out of the light path as an Exmor so in practice bayer may be more sensitive.
Sound right?
-
Hi,
I would just add that the Foveon doesn't have color filters. It separates color based on diffusion depth in the sensor. So filter design cannot be optimized. I would presume that it takes aggressive math to reconstruct color.
By the way, the real "fovea" is very similar to the "Bayer" pattern. It is also RGBG, by and large.
Best regards
Erik
Hi,
Unfortunately that doesn't work that way. Instead of actually sampling the 2 missing channels in a Bayer CFA, they are interpolated. No sensitivity is lost, the interpolated signal (which is slightly less accurate) is added to the luminosity.
Cheers,
Bart
-
Unfortunately that doesn't work that way. Instead of actually sampling the 2 missing channels in a Bayer CFA, they are interpolated. No sensitivity is lost ...
Of course some sensitivity is lost by discarding about 60% of the light arriving at a particular location by absorbing it with a color filter. This leads to lower photon counts than, for example, a monochrome sensor and so worse ratio pf signal to photon shot noise than would be possible if an imagined perfect X3 sensor could count almost all of the incoming photons, binned into counts for three spectrall ranges, "red", "green", and "blue". But the Foveon approach to X3, relying on the fact that silicon has different degree of transparency to different wavelengths of light, loses even more information to absorption of light by the silicon, and so ends up with worse SNR than a good CFA sensor. Particularly for the color measured at the deepest level of the Foveon sensor: I forget whether that is blue or red.
Edit: it is red at the bottom. In more detail, the Foveon approach (not used by the other X3 approaches for which I have seen patents) really takes three measurements with a mix of colors, at three levels in the silicon:
- top: mostly blue, some green, a little red
- middle: mostly green, some red, a little residual blue
- bottom: mostly red, some residual green, a very little residual blue.
Then these mixed signals have to be mathematically unravelled, which increases noise and potentially causes color errors, metamerism and such. Also, the red signal is quite weak: only a rather small proportion of light of any color gets that far through the silicon.
-
Of course some sensitivity is lost by discarding about 60% of the light arriving at a particular location by absorbing it with a color filter.
Now how it that? If we use a strong red filter to capture only the red rays, it is the non-red wavelengths we are rejecting which we do not want in the first place. So where is 60% of the light lost? That 60% can not be used to describe red, as it is not red.
-
Now how it that? If we use a strong red filter to capture only the red rays, it is the non-red wavelengths we are rejecting which we do not want in the first place. So where is 60% of the light lost? That 60% can not be used to describe red, as it is not red.
Are the only colors you see rgb? No, there are other colors that are combinations. Remember Newton and his prism?
-
Of course some sensitivity is lost by discarding about 60% of the light arriving at a particular location by absorbing it with a color filter.
That is true. Presented below are two actual images with and without a Bayer CFA with the same sensor and optical parameters. There is a lot of light loss in a Bayer CFA:
(http://djjoofa.com/data/images/bw_col_both.jpg)
-
in color, please..
-
Are the only colors you see rgb? No, there are other colors that are combinations. Remember Newton and his prism?
So what you good people are suggesting is using the total spectrum energy to catch certain wavelength only. As far as I know this (recording one color) can be done only by filtering the spectrum to isolate the wanted frequency. It does not matter if we use RGB or not. Picking even more specific colors than R G an B would mean even more narrow filters "wasting" even more of the poor photons. Is somebody forgetting basic physics? Me, maybe? Please inform.
-
Well, lets see. What colors would the bayer array absorb from a yellow sandy beach scene? Red from some spots, green from some spots. Blue almost 100% regection. The foveon would take both so the exposure would be 50% higher.
You take a picture of classical greek buildings. They are a light grey marble, almost white. The foveon would absorb photons on all 3 channels. the bayer would take 1/3 at each.
What you really want are micro prisms over angled RGB so you can split out what is there over smaller sub-pixels. Easy to say, it was probably very difficult to make in many an R&D lab. You almost need curved rings of pixels.
-
Now how it that? If we use a strong red filter to capture only the red rays, it is the non-red wavelengths we are rejecting which we do not want in the first place. So where is 60% of the light lost? That 60% can not be used to describe red, as it is not red.
Joofa covered this nicely. "That 60%" can be used to describe "green" or "blue". Why would we not want those described in a general camera?
If e.g. 60% of the photons hitting a sensor are reflected or absorbed in color filters, this is real information about the scene that is wasted. Clearly, an ideal sensor would not throw away information.
-h
-
You take a picture of classical greek buildings. They are a light grey marble, almost white. The foveon would absorb photons on all 3 channels. the bayer would take 1/3 at each.
and interpolate and add the missing 2/3rds of the spectrum when demosaicing.
In both cases the green channels record similar (say 1/3rd of the total) amounts of light (give or take quantum efficiency differences) at each corresponding spatial sampling position, the red channels record similar amounts of light at their corresponding positions, and the blue channels record similar amounts of light at their corresponding positions. The only difference is that the Demosaicing will fill in the blanks on the image taken with a Bayer CFA.
Cheers,
Bart
-
and interpolate and add the missing 2/3rds of the spectrum when demosaicing.
In both cases the green channels record similar (say 1/3rd of the total) amounts of light (give or take quantum efficiency differences) at each corresponding spatial sampling position, the red channels record similar amounts of light at their corresponding positions, and the blue channels record similar amounts of light at their corresponding positions. The only difference is that the Demosaicing will fill in the blanks on the image taken with a Bayer CFA.
Cheers,
Bart
The Bayer image would be based on fewer total counted photons. (given certain, idealized conditions that we seem to take for granted, but certainly not true for real Foveon sensors).
Given a smooth, feature-less scene, sampling its value using 1/3 of the photons would give a higher uncertainty ("more noise"), no matter what interpolation and demosaic is used.
-h
-
If e.g. 60% of the photons hitting a sensor are reflected or absorbed in color filters, this is real information about the scene that is wasted. Clearly, an ideal sensor would not throw away information.
-h
If we want red color information from the scene, we ARE throwing away a lot of information. Same for any color. I see no way around it with present technology. This hypothetical sensor with 100% efficiency would work only for luminance = B&W photographs.
-
If we want red color information from the scene, we ARE throwing away a lot of information. Same for any color. I see no way around it with present technology. This hypothetical sensor with 100% efficiency would work only for luminance = B&W photographs.
Do you want to image just the "red" information in a scene? Or do you want to take general images? If you (like) me want a general image of a usual scene, those scenes tends to contain most information (=hard-to-predict sharp edges, fractal-like structures) in the luminance-channel, and after white-balancing, all primary color channels tends to contribute to the printed image.
Camera tech like the prism used in "3-CCD" video cameras could (in principle) capture all of the photons projected by the lense. In practice, I am sure that such prisms have losses, non-ideal wavelength selectivity etc. The point is that instead of converting "photons of the wrong color" to heat, they are instead diverted to the sensor that can properly count them. An ideal Foveon-like sensor would do the same.
I tend to believe that in the competitive camera market, manufacturers tends to make the kind of products that is possible today, that minimize R&D/production-cost and maximize customer willingness to pay. I really don't have the economical-technical knowledge to claim that they are wrong.
-h
-
The Bayer image would be based on fewer total counted photons. (given certain, idealized conditions that we seem to take for granted, but certainly not true for real Foveon sensors).
Given a smooth, feature-less scene, sampling its value using 1/3 of the photons would give a higher uncertainty ("more noise"), no matter what interpolation and demosaic is used.
Indeed, for total luminosity, but not lower sensitivity as some seem to believe. The (broadly speaking) 1/3rd of the spectrum caught by a Bayer CFA at a given sampling position is the same 1/3rd of the spectrum recorded by a Foveon like sensor. By adding the interpolated missing data, the full RGB luminosity is restored (with a slightly lower accuracy).
Given that fewer photons in total are actually counted, also allows to utilize the full available well depth for the spectral band that was recorded. It doesn't have to share the silicon real-estate with 2 other spectral bands for the same sampling position. It also allows to write the Raw data to a file faster (because there is less data, just a single band) and file sizes are much smaller.
Cheers,
Bart
-
Indeed, for total luminosity, but not lower sensitivity as some seem to believe.
By "sensitivity" do you mean the number of electrons delivered to a hypothetical analog front-end/ADC for a given lense/sensor area?
Comparing e.g. the Leica M9 to the M9-m (or whatever the called the monochrome (achromatic?) version) sharing the same sensor and electronics, different only in the latter having no color filter. Then each sensel would receive e.g. 3 times as much light as a CFA filtered sensel, assuming that the CFA removes 2/3 of the photons hitting it, and that exposure parameters are held constant.
-h
-
By "sensitivity" do you mean the number of electrons delivered to a hypothetical analog front-end/ADC for a given lense/sensor area?
By sensitivity I mean required exposure time at a given ISO setting. Some are suggesting that up to a stop can be gained by not filtering out 2/3rd of the spectrum at a given sampling position, which is not true because the 2/3rds are added through interpolation instead of being sampled directly. The Bayer CFA has a reasonably high transparency at the spectral band pass wavelengths although some loss takes place, but the Foveon also has less than 100% efficiency, lots of non-sensitive areas per photosite due to connectors/gates.
Comparing e.g. the Leica M9 to the M9-m (or whatever the called the monochrome (achromatic?) version) sharing the same sensor and electronics, different only in the latter having no color filter. Then each sensel would receive e.g. 3 times as much light as a CFA filtered sensel, assuming that the CFA removes 2/3 of the photons hitting it, and that exposure parameters are held constant.
Sure, without filters, no filter loss at all, but also no colors.
Cheers,
Bart
-
but the Foveon also has less than 100% efficiency, lots of non-sensitive areas per photosite due to connectors/gates.
what about microlenses ? Foveon shall have those and when you chemically wash off (unless that was a monochrome sensor by design) your CFA from non Foveon Bayer CFA sensor you also wash off microlenses that sit on top of CFA....
-
Now how it that? If we use a strong red filter to capture only the red rays, it is the non-red wavelengths we are rejecting which we do not want in the first place. So where is 60% of the light lost? That 60% can not be used to describe red, as it is not red.
Of course it canot be used to describe red, but it could be used to describe blue or green at that location, which would add useful information, and that it what an X3 sensor does. In an X3 sensor, that light of other colors is also recorded, giving blue and green signals along with the red signal at that location. This avoids the need to interpolate those colors in from nearby locations. So if you compare sensors with the same number of photosite locations (like about 16 million on the current Foveon sensors vs a 16MP bayer CFA sensor) the avoidance of interpolation increases resolution, while if instead the CFA sensor has more photosites to equalize resolution (very roughly 32MP in my example) then the larger area of the X3 photosites are potentially gathering more light at equal exposure index, reducing the effects of photon shot noise on the overall S/N ratio. (Comparing sensors of equal size, of course.)
Of course, this makes the big assumption of comparable quantum efficiency in each color channel, whereas the current Foveon approach seems distinctly worse than the Bayer CFA "state of the art" in that respect, cancelling out the potential for lower noise at high exposure index. And for all I know, lower "QE per color" may be an unavoidable disadvantage of measuring with three vertically stacked detectors.
P. S. As Erik pointed out, the actual human fovea uses single color photodectors more like a CFA sensor: cones each of whcih gives a signal for one of red, blue, or green ... along with a few pure luminosity signals from rods. Not quite the
GR
BW
tried by Kodak and Sony at times, but closer to that than to Foveon X3.
-
By adding the interpolated missing data, the full RGB luminosity is restored (with a slightly lower accuracy).
That "slightly lower accuracy" due to interpolation is in fact quite substantial if comparing with an equal number of photosite locations. The current "16MP X 3” Foveon sensor has distinctly more resolution than a 16MP CFA sensor. On the other hand, given our eyes' far lower resolution of color detail than luminosity detail, this advantage for X3 might be less in practice than in theory. And certainly far less than some Foveon fanboys make out with their trick of red-blue resolution charts!
Given that fewer photons in total are actually counted, also allows to utilize the full available well depth for the spectral band that was recorded.
That is a good point: at base sensitivity where you can get close to filling the wells at highlight locations, using a single monochromatic well at each location might work better. I do not know enough about the methods of constructing these three layer sensors with readouts from three levels; that could introduce even more constraints that make things even worse for any X3 sensor architecture.
-
That "slightly lower accuracy" due to interpolation is in fact quite substantial if comparing with an equal number of photosite locations.
Actually, on some of the Bayer CFA effect on resolution (http://bvdwolf.home.xs4all.nl/main/foto/bayer/bayer_cfa.htm) tests I did many moons ago I found that the loss in luminosity resolution is relatively limited, some 6.4%.
The current "16MP X 3” Foveon sensor has distinctly more resolution than a 16MP CFA sensor.
The main reason for that stems from comparing the lack of an Optical Low-Pass Filter (OLPF), which potentially creates other issues, to a Anti-Aliasing filtered image (an often not Capture sharpened correctly either). Add a few demosaiced Bayer CFA pixels and apply proper sharpening, then compare sharpness and artifacts again ...
On the other hand, given our eyes' far lower resolution of color detail than luminosity detail, this advantage for X3 might be less in practice than in theory. And certainly far less than some Foveon fanboys make out with their trick of red-blue resolution charts!
While that's correct, there is a small benefit to full RGB sampling at each output pixel position, especially when magnifying the image and when per pixel color accuracy is very important. But that also requires other factors to be exactly right, e.g. noise and color separation, which is a bit of an issue with the Foveons.
Cheers,
Bart
-
Indeed, for total luminosity, but not lower sensitivity as some seem to believe. The (broadly speaking) 1/3rd of the spectrum caught by a Bayer CFA at a given sampling position is the same 1/3rd of the spectrum recorded by a Foveon like sensor. By adding the interpolated missing data, the full RGB luminosity is restored (with a slightly lower accuracy).
Given that fewer photons in total are actually counted, also allows to utilize the full available well depth for the spectral band that was recorded. It doesn't have to share the silicon real-estate with 2 other spectral bands for the same sampling position. It also allows to write the Raw data to a file faster (because there is less data, just a single band) and file sizes are much smaller.
Cheers,
Bart
Ok, but how much is slightly lower accuracy? Having real data vs guesstimated would be a big deal on fine changing detail. It would be irrelevant on something like a car shot that is all basically the same color.
In the current season a landscape of mountainside with snow covered trees would be a big difference in the detail of the shot. Bayer shots like this have always looked artificial to me. Typically you have to downsample much more on natural textures.
-
in color, please..
Color is just a perception of mind, a figment of imagination ...
P. S. As Erik pointed out, the actual human fovea uses single color photodectors more like a CFA sensor: cones each of whcih gives a signal for one of red, blue, or green ... along with a few pure luminosity signals from rods.
The "luminosity" signal in human vision comes mostly from cones, not rods, under ordinary conditions, unless the illumination is really low.
-
The "luminosity" signal in human vision comes mostly from cones, not rods, under ordinary conditions, unless the illumination is really low.
Thanks for the clarification: that is what I was trying to indicate with the word "few". For one thing, AFAIK the fovea does not have many rods. All of which makes the fovea even closer to being a three color CFA sensor.
-
Actually, on some of the Bayer CFA effect on resolution (http://bvdwolf.home.xs4all.nl/main/foto/bayer/bayer_cfa.htm) tests I did many moons ago I found that the loss in luminosity resolution is relatively limited, some 6.4%.
That test is with a pure grayscale subject, so that CFA pixels of any color are near perfect proxies for luminance measurements, making it an unnaturally easy case for retaining resolution in demosaicing. On real world issues is that sharp luminance boundaries are likely to go with shifts in color, causing more oclor moiré issues and so need a stronger OLPF or stronger postprocessing to avoid aliasing artifacts. I prefer the rather consistent subjective observstion with natural subject matter that it takes about twice as many Bayer CFA photosites as X3 photosites to get comparableperceived sharpness.
By the way, what is your resolution measure? 50% MTF? Extinction resolution? Something else?
The whole game will shifts of course if we get to the regime of oversampling, with resolution limited almost entirely by the lens, not the sensor. Then the main potential advantage of X3 over CFA is the effect on low-light performance of counting most of the received photons versus only about 40% of them.
-
Ok, but how much is slightly lower accuracy? Having real data vs guesstimated would be a big deal on fine changing detail. It would be irrelevant on something like a car shot that is all basically the same color.
It depends on the level of detail in the original signal, and the amount of (photon shot) noise. Light doesn't have an absolute brightness, only an average, there is always noise involved. Detail doesn't have an absolute level, there is also a lens/diffraction/(de)focus involved and subject motion and camera shake, and usually an OLPF to reduce the tendency for aliasing inherent in discrete sampling. So when it's not easy to say what the signal was, it's also not so easy to say how much exactly the signal was off.
So the 'slightly lower' can only be answered in a statistical sense, or compared to an ideal laboratory setup, and the answer will not be a simple number but rather something like an MTF curve which needs interpretation.
In the current season a landscape of mountainside with snow covered trees would be a big difference in the detail of the shot. Bayer shots like this have always looked artificial to me. Typically you have to downsample much more on natural textures.
Looks are hard to comment on, but from my experience it often has to do with inadequate Capture sharpening (assuming good technique was used to get the shot). Digital cameras just record what's thrown at them, and the sensor technology just makes a difference in which artifacts are tolerated or suppressed, but they all produce artifacts, no exception.
Bayer CFA sensors have a slightly lower Chrominance than Luminance resolution, but Chrominance usually has a lower level of signal detail anyway compared to Luminance (just look at the color channels of an image in e.g. in Lab mode). Also, the different sampling density between Red/Blue and Green can cause False color artifacts.
Sensors without an OLPF by definition exhibit more aliasing artifacts (although DOF/defocus can function as a Low-pass filter), and sensors that sample R/G/B for each pixel will only have Luminance aliasing. In the Foveon desgn especially, the channel separation is not that simple (the Raw sensor data is almost grayscale), and it requires significant processing which boosts noise and limits high ISO robustness, also because of the small well depth per channel.
TANSTAAFL (http://en.wikipedia.org/wiki/TANSTAAFL)
Cheers,
Bart
-
That test is with a pure grayscale subject, so that CFA pixels of any color are near perfect proxies for luminance measurements, making it an unnaturally easy case for retaining resolution in demosaicing.
Only a bit easier, and luminance is the most important aspect of visual resolution. Besides, don't forget that even a pathological case (extremely unlikely to occur in real nature, next to each other from opposite sides of the spectrum) of a Blue/Red target has a significant luminance contrast difference. Only the even more unlikely situation of a Red tint and a Blue tint with approx. equal luminance contribution, would create a worst case scenario for resolution. To satisfy a few masochists, I've tried to create such a target. A link can be found at the first P.S. of this post (http://www.openphotographyforums.com/forums/showthread.php?t=13217).
On real world issues is that sharp luminance boundaries are likely to go with shifts in color, causing more oclor moiré issues and so need a stronger OLPF or stronger postprocessing to avoid aliasing artifacts. I prefer the rather consistent subjective observstion with natural subject matter that it takes about twice as many Bayer CFA photosites as X3 photosites to get comparableperceived sharpness.
That would presumably include using an OLPF on the Bayer CFA and no OLPF on the Foveon, and relatively large sensels, and no access to more than 16MP sensors. Hardly a realistic assumption, when you e.g. look at what a D800 produces compared to a D800E. Visibly more False color artfacts (but not impossible to correct in post-processing) and 1% more luminance (extinction) resolution for the AA-less version.
By the way, what is your resolution measure? 50% MTF? Extinction resolution? Something else?
Limiting visual (extinction) resolution. Both sensors types would resolve close to the physical limit of the Nyquist frequency.
The whole game will shifts of course if we get to the regime of oversampling, with resolution limited almost entirely by the lens, not the sensor. Then the main potential advantage of X3 over CFA is the effect on low-light performance of counting most of the received photons versus only about 40% of them.
As explained before, that's not how the math works out, and the proof is in the differences in high ISO performance where photons are a limited commodity.
Cheers,
Bart
-
P. S. As Erik pointed out, the actual human fovea uses single color photodectors more like a CFA sensor: cones each of whcih gives a signal for one of red, blue, or green ... along with a few pure luminosity signals from rods. Not quite the
GR
BW
tried by Kodak and Sony at times, but closer to that than to Foveon X3.
I am not sure that this line of reasoning have much relevance.
Our human senses may have all kinds of limitations, defects, nonlinearities etc. The goal of a camera/sound recorder/... is usually not to _mimic_ those senses, but to recreate reality in such a way as to _fool our senses_. Often, those two goals are overlapping (if our hearing only extends to 20kHz, then a recording system does not need to recreate anything > 20kHz), but at times, they may not (e.g. we may perceive a very dark scene, viewed physically, as "noisy", but if a camera/display system adds noise on top of that, the total perceived noise may be unrealisstic).
-h
-
About my comparisons between the fovea, Foveon sensors and CFA sensors
I am not sure that this line of reasoning have much relevance.
Quite right, except my comment was not a line of reasoning at all; it was just a comment on the ironical innacuracy of the brand name "Foveon". I see no reason to think that functioning more or less like the human eye makes a sensor better or worse.
-
Bart,
Which math are you referring to? Your measurement of a mere 6% difference in extinction resolution with a pure gray scale target is not of much interest to me, though at least gray scale is far less bad than the other extreme of pure red/blue resolution charts. Do you have comparisons based on MTF50 or similar?
And please, let us separate the poor low light performance of Foveon's particular approach from the more general potential of other approaches of "X3" detection.
-
Bart,
Which math are you referring to? Your measurement of a mere 6% difference in extinction resolution with a pure gray scale target is not of much interest to me, though at least gray scale is far less bad than the other extreme of pure red/blue resolution charts. Do you have comparisons based on MTF50 or similar?
Hi,
The Math is about only sampling a single channel does not mean a loss of sensitivity, requiring a longer exposure to compensate. The not sampled channels are added by interpolation.
There is nothing sacred about MTF50, it's just the spot in a system MTF curve at 50% modulation. The level of detail at that position depends on the the sensor (sensel pitch and number of sensels) and the subject contrast, and thus varies in absolute Cycles/mm.
The simplest way of comparing resolution is by inspection of the limiting visual resolution of a sinusoidal star target (http://www.openphotographyforums.com/forums/showthread.php?t=13217). The sinusoidal grating is important to avoid aliasing atitfacts in discrete sampling devices such as our sensor arrays. When we can no longer see the smaller detail, then its contrast has been reduced to zero. BTW contrast (a function of Lens MTF and sensor MTF) can be boosted in post processing, e.g. by wavelet decomposition or high pass sharpening for specific levels of detail, provided that there is some contrast (Signal > Noise) left to process.
Let's have a look at the following example, based on the central areas of the star targets from the above link:
A central crop of the regular RGB star target, and one of the special '~same luminance Red/Blue' version:
(http://bvdwolf.home.xs4all.nl//main/foto/bayer/bayer_files/JTF144cy-300pxcrop_RGB.png) (http://bvdwolf.home.xs4all.nl//main/foto/bayer/bayer_files/JTF144cy-300pxcrop_RB.png)
In Photoshop, I applied a Gaussian blur with radius 0.70 (which simulates a very good lens at its optimum aperture) to each,
and I added a Multiply blending layer with a RGGB pattern, which produces these Bayerized versions:
(http://bvdwolf.home.xs4all.nl//main/foto/bayer/bayer_files/JTF144cy-300pxcrop_RGB+070GB+CFA.png) (http://bvdwolf.home.xs4all.nl//main/foto/bayer/bayer_files/JTF144cy-300pxcrop_RB+070GB+CFA.png)
I then used a program (PixInsight) that allows to demosaic a Bayer CFA image with the generic VNG demosaicing algorithm
(much better than bilinear or bicubic interpolation, but maybe not the best that is possible), which produced the following:
(http://bvdwolf.home.xs4all.nl//main/foto/bayer/bayer_files/JTF144cy_300pxcrop_RGB_070GB_VNG.png) (http://bvdwolf.home.xs4all.nl//main/foto/bayer/bayer_files/JTF144cy_300pxcrop_RB_070GB_VNG.png)
I added a Red circle to mark the Nyquist frequency (@ 92 pixels diameter) , and a Green circle to mark the visual limiting resolution. The limiting resolution as I saw it for the RGB demosaiced version was at 96 pixels blur diameter, which was 4.2% (92/96) below Nyquist. Some could prefer to draw the limit at 98 pixels blur diameter, it's a bit arbitrary visually, which would be 6.1% below Nyquist. That's pretty consistent with my findings (http://bvdwolf.home.xs4all.nl/main/foto/bayer/bayer_cfa.htm) some 9 years ago, based on experiments with another Demosaicing algorithm.
The pathologically unlikely case of the adjacent colors from both opposite ends of the visual spectrum with about equal luminance to put the Bayer CFA at its most disadvantage, indeed produces the expected result. I've put the visual resolution blur limit at a diameter of 184 pixels (green circle), which is half (92/184) of the Nyquist frequency. That's exactly consistent with half the sampling density of the Red / Blue sensels compared to the Green sensels.
That demonstrates the potential effects of Demosaicing only(!). A Foveon like sensor also cannot exceed the Nyquist frequency. The constructed 'R/B' only Bayer CFA version has zero luminance contribution in the green channel, which is also extremely unlikely, because of the CFA filter characteristics (they are not perfect bandpass filters) and the effect of a possible OLPF (which spills some signal to adjacent sensels). It's also useful to realise that Demosaicing algorithms in general, use luminance differences to boost the Red and Blue resolution to virtually the same level as the Green resolution.
That's the 'Math' involved, straightforward and verifiable by all.
Cheers,
Bart
-
That is interesting if a bit vague. Are you saying that the last picture with the large cyan circle is a picture from a shot you took? Sorry, I have a bit of trouble following the post. I may need more coffee.
What I think I can interpret is that in de-bayering most detail comes from luminance information. The circle from a black white star pattern is very close to nyquist while the circle from your red blue is about double. So the ability to get a color right is much worse than getting luminance right, so color accuracy on fine color detail is suspect. That is the point of the thread right? To say getting real color data at a point is much better than guesstimating.
Bayer is great when you have sections of similar tones. Any fine random or fractal type pattern would look poor. I think that matches with our experience using the cameras. You need a few similar pixels for it to guess the color.
-
Another experiment we can all easily do is take a picture of a color checker or even better, IT8 target, then take another when the patches are far enough away to be about the size of 1 or 2 pixels. Will the measurement of the colors be the same? Take several, 1 pixel, 2x2, 3x3, 4x4. Then you know in the real world what length of lens you need to get proper color on the detail of a shot.
-
That is interesting if a bit vague. Are you saying that the last picture with the large cyan circle is a picture from a shot you took? Sorry, I have a bit of trouble following the post. I may need more coffee.
No cameras involved, only mosaicing and demosaicing, to show the influence of only sampling part of the color info versus all info (the originals at the top). Coffee might help to see what's the actual loss and what is not.
What I think I can interpret is that in de-bayering most detail comes from luminance information.
Correct, and color also carries a luminance component, so luminance is not only supplied by the Green filtered sensels.
The circle from a black white star pattern is very close to nyquist while the circle from your red blue is about double. So the ability to get a color right is much worse than getting luminance right, so color accuracy on fine color detail is suspect.
Not exactly. Only colors with virtually no luminance contrast will have reduced resolution, all others will not suffer as much from the partial color sampling per sensel. There may be some false color artifacts depending on the demosaicing algorithm, due to the different sampling densities between Blue/Red and Green (and thus different Nyquist limits and aliasing artifacts).
That is the point of the thread right? To say getting real color data at a point is much better than guesstimating.
The thread is about the different trade-offs, IMO.
Bayer is great when you have sections of similar tones. Any fine random or fractal type pattern would look poor. I think that matches with our experience using the cameras. You need a few similar pixels for it to guess the color.
I'd say it's amazing how good the color is, given the limited sampling density (and resulting filesize and thus capture speed).
Cheers,
Bart
-
Bart,
Thanks for the details, though unfortunately they have the familiar limitations of a purely mathematical attempt to quantify a subject that involves the complications of the human visual system. All we get for sure is that a Bayer CFA sensor can retain some slight resolution at extremely low MTF (so that features might often by invisible in practice) almost up to Nyquist, while color details might be limited to as low as half that, while an X3 sensor can be close to Nyquist regardless of color issues, though the significance of "color resolution" is unclear, due to the way our eyes work.
Perhaps the easiest way out is to accept that sensor resoluton is heading towards being abundant ("oversampling is coming"), so that other comparisons like low light handling, dynamic range, and color accuracy are far more important.
-
The Noise Ninja Calibration target is a good one to use. Colors are fairly randomly spread out.
How's about someone with a Canon, Nikon, Sony, Pentax, Olympus each try shots with these colors near 1x1 to 5x5 pixels. You can take pictures of your screen. The absolute color doesn't matter, the color difference from size to size does.
It would also be interesting to see the differences from one raw converter to another if any.
-
Bart,
Thanks for the details, though unfortunately they have the familiar limitations of a purely mathematical attempt to quantify a subject that involves the complications of the human visual system. All we get for sure is that a Bayer CFA sensor can retain some slight resolution at extremely low MTF (so that features might often by invisible in practice) almost up to Nyquist, while color details might be limited to as low as half that, while an X3 sensor can be close to Nyquist regardless of color issues, though the significance of "color resolution" is unclear, due to the way our eyes work.
Exactly, although the worst case scenario is very unlikely to occur. We can get an idea of what the relative importance of color is for resolution by looking at the 'ab' channels of a 'Lab' colorspace image. Color resolution doesn't fluctuate as rapidly as luminance does. That's why the Bayer CFA works as good as it does.
Perhaps the easiest way out is to accept that sensor resoluton is heading towards being abundant ("oversampling is coming"), so that other comparisons like low light handling, dynamic range, and color accuracy are far more important.
Yes, oversampling will make color resolution a moot issue and it will increase luminance resolution even further, and other issues like dynamic range are easier to solve with a Bayer CFA because of the larger silicon real estate for deep wells at a given sampling density. Only a single band of colors needs to be stored in a well, instead of 3 wells that are 1/3rd the size.
Cheers,
Bart
-
Exactly, although the worst case scenario is very unlikely to occur. We can get an idea of what the relative importance of color is for resolution by looking at the 'ab' channels of a 'Lab' colorspace image. Color resolution doesn't fluctuate as rapidly as luminance does. That's why the Bayer CFA works as good as it does.
Not quite 'lab', but perhaps sufficient for this discussion:
http://en.wikipedia.org/wiki/Y%27CbCr
(http://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Barns_grand_tetons_YCbCr_separation.jpg/220px-Barns_grand_tetons_YCbCr_separation.jpg)
"A color image and its Y, CB and CR components. The Y image is essentially a greyscale copy of the main image."
-
Not quite 'lab', but perhaps sufficient for this discussion:
http://en.wikipedia.org/wiki/Y%27CbCr
(http://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Barns_grand_tetons_YCbCr_separation.jpg/220px-Barns_grand_tetons_YCbCr_separation.jpg)
"A color image and its Y, CB and CR components. The Y image is essentially a greyscale copy of the main image."
Hi,
Exactly, and that BTW is also why such colorspaces compress so efficiently (even at high quality settings), because 2/3rd of the image has low frequency low modulation data which requires fewer bits to encode the per pixel differences.
Cheers,
Bart
-
Hi,
Exactly, and that BTW is also why such colorspaces compress so efficiently (even at high quality settings), because 2/3rd of the image has low frequency low modulation data which requires fewer bits to encode the per pixel differences.
Cheers,
Bart
It's actually that nature of scalar quantization in the usual compression formats that make dependence on a color space important. Effectively, the color channels in the various color spaces are the directions they point in. Some directions compress more with scalar quantization. However, if vector quantization is used, which typically offers more compression than scalar quantization, then directionality doesn't help. And effectively, RGB, YCbCr, or another matrix-derived color space should compress the same.
-
It's actually that nature of scalar quantization in the usual compression formats that make dependence on a color space important. Effectively, the color channels in the various color spaces are the directions they point in. Some directions compress more with scalar quantization. However, if vector quantization is used, which typically offers more compression than scalar quantization, then directionality doesn't help. And effectively, RGB, YCbCr, or another matrix-derived color space should compress the same.
I'd suggest that YCbCr combined with spatial downsampling of Cb/Cr channels (e.g. "4:2:2", "4:2:0" etc) is used because it:
1. Is low cost computationally
2. Performs bandwidth reduction before a codec (that may have a high processing cost per input pixel)
3. Maps reasonably well to perceptual correlates, meaning that you can reduce precision (quantize) with fairly low complexity while still having reasonable rate/distortion trade-offs
4. Maps reasonable well to redundancy in real-world images, meaning that the bandwidth that is dropped often contains little information
I imagine that all of this can be done with vector quantization, but I imagine that you'd spend lots of cycles and complexity achieveing what can, in practice be had a lot cheaper. It has been a while since I looked at vector quantization but as I remember it, it was essentially the solution to every problem - given that you could afford to build sufficiently large vectors, something that often is not realtistic?
-h
-
I'd suggest that YCbCr combined with spatial downsampling of Cb/Cr channels (e.g. "4:2:2", "4:2:0" etc) is used because it:
1. Is low cost computationally
2. Performs bandwidth reduction before a codec (that may have a high processing cost per input pixel)
3. Maps reasonably well to perceptual correlates, meaning that you can reduce precision (quantize) with fairly low complexity while still having reasonable rate/distortion trade-offs
4. Maps reasonable well to redundancy in real-world images, meaning that the bandwidth that is dropped often contains little information
I imagine that all of this can be done with vector quantization
Yes. Especially, if you do VQ on each color channel separately. In that case, like scalar quantization, the directions in which color channels point is important, and different color spaces will compress differently. YCbCr is just one set of such directions. Not the optimal. Optimal depends upon image content. However, if we want to pick a fixed transformation for all images, much like YCbCr, there exists better transforms than YCbCr that result in more compression. YCbCr is not bad, but there are better choices. And, such choices will work better even with scalar quantization.
-
Astro images: as long as you have accurate tracking, you will be able to shoot the same field of stars, and the deep sky objects (stars and nebulae) don't move in relation to each other. Just align the R, G, and B (or other filter) shots and process!
Landscapes: features move with relation to each other. In the case of a triad of R, G, and B capture, fusing it with any intervening motion is going to be problematic and require interpolations - similar to the Bayer demosaicing process (not computationally, but spatially).
-
...YCbCr is just one set of such directions. Not the optimal. Optimal depends upon image content. However, if we want to pick a fixed transformation for all images, much like YCbCr, there exists better transforms than YCbCr that result in more compression. YCbCr is not bad, but there are better choices....
Given that the (perhaps) main feature of "4:2:0" is that it lays the ground for 10:1 or 100:1 lossy compression in codecs that seem to have been tuned to the characteristics of YCbCr (BT 601 or 709) in the way that it trades visual errors for bandwidth reduction in a way that can only be reliably measured using largish panels of viewers, how would you go about to make a replacement, and confirm that it improves the end-to-end characteristics significantly more than the measurement uncertainty?
-h
-
By sensitivity I mean required exposure time at a given ISO setting. Some are suggesting that up to a stop can be gained by not filtering out 2/3rd of the spectrum at a given sampling position, which is not true because the 2/3rds are added through interpolation instead of being sampled directly.
I am having a little trouble understanding how 2/3rds of the information would be made up digitally in post. Aren't we confusing exposure with brightness, like when an underexposed image's brightness is increased through Compensation in post? The difference can be found in the noise - even just in the luminance channel.
Simplifying, assuming natural daylight and 3 contiguous sensels of the same size from the two sensors above (RGB for the one with CFA and L1L2L3 for the one without) with the same exposure so that the Ls are below saturation, aotbe the raw count in the RGB sensels is going to be 1/3 that of the L pixels, as can be seen in the histograms in joofa's post. Sure, one can increase their values digitally in post (let's call it 'interpolation':-) so that they are very similar, but their noise (SNR) would still be 1.6 stops worse.
If on the other hand one wanted similar results off the bat in terms of raw values and SNR performance, one would have to increase the exposure time of the sensor with CFA by three times aotbe... that would mean that the non-CFA sensor performs at ISO 300 how the CFA sensor does at ISO 100, a significant 1.6 stop noise advantage :-)
Jack
-
I am having a little trouble understanding how 2/3rds of the information would be made up digitally in post. Aren't we confusing exposure with brightness, like when an underexposed image's brightness is increased through Compensation in post? The difference can be found in the noise - even just in the luminance channel.
Hi Jack,
No we are not confusing exposure with brightness, at least I am not.
Let's zoom in on one single pixel, and for the sake of simplicity let's assume it records values from 0 to 255 for each channel, and let's disregard gamma. From an RGB Foveon type of sensor we may get a Raw recorded data reading of [128,128,128] because 3 channels are sampled, and from a Bayer CFA filtered we may get [128,0,0], or [0,128,0], or [0,0,128] depending on the color of the filter. So for the Bayer CFA we have 2/3rds missing, but the corresponding channel does record something.
Now we do a Bayer CFA demosaicing, which has nothing to do with amplification! The Bayer CFA demosaicing (a very clever interpolation) will use the surrounding sensel positions to estimate the most likely value for the missing channels. When we happen to be watching a uniform patch of gray, then the interpolation between surrounding Green filtered sensels will read 128 all around our pixel, so the interpolation decides that the missing Green channel should probably also be 128. Thus, after one interpolation, we get either [128,128,0], or [0,128,0], or [0,128,128] depending on the color data that was actually sampled. Likewise the interpolation from neighboring pixels suggests that the Red channels that were not sampled are probably 128, which gives us [128,128,0], or [128,128,0], or [128,128,128]. And after interpolating Blue from the surrounding Blue filtered sensels, we'd get [128,128,128], or [128,128,128], or [128,128,128] depending on the filter color.
As you can see, the interpolation guessed right regardless of the filter color (because a uniform patch is simple to interpolate) regardless of the amount of light that was recorded through the filter and the colors that were absorbed by the filter. The missing channel data and the level were interpolated/reconstructed from the surrounding pixels.
So despite of only really sampling 1/3rd of the light at each pixel position, the reconstruction by interpolation gives the same RGB output brightness for both types of sensor.
Cheers,
Bart
P.S. When you zoom in on the earlier synthesized CFA images in the middle row, you'll see exactly what I described (single channel colors, either R, G, or B), but there things have more detail. The original brightness of the originals in the first row, has been reconstructed by interpolation.
-
Given that the (perhaps) main feature of "4:2:0" is that it lays the ground for 10:1 or 100:1 lossy compression in codecs that seem to have been tuned to the characteristics of YCbCr (BT 601 or 709) in the way that it trades visual errors for bandwidth reduction in a way that can only be reliably measured using largish panels of viewers, how would you go about to make a replacement, and confirm that it improves the end-to-end characteristics significantly more than the measurement uncertainty?
-h
For objective criterion stuff such as distortion measure or bitrate allocated per sample may be compared.
-
Thanks Bart, I am starting to see where you are coming from. My question concerned the following quote:
'Some are suggesting that up to a stop can be gained by not filtering out 2/3rd of the spectrum at a given sampling position, which is not true because the 2/3rds are added through interpolation instead of being sampled directly.'
I framed it in terms of the same sensor with or without CFA because I understand those better and I do not know Foveon at all. Let's see if I can rephrase it to include Foveon.
To make things simple I will assume APS-C size for both. As far as I understand this should mean that (simplifying) 1 Foveon sensel will have about the same area as 3 RGB ones. For a given Exposure, 384 photons of daylight spectrum will fall on the area of the Foveon sensel which (simplifying) will capture them all and sort them producing a raw count of (simplifying) (128,128,128) as you suggested.
For the same exposure a similar set of 384 photons will arrive over the 3 RGB simplified-bayer sensels, or 128 each. However, the sensel under the Red filter will only see (simplifying) 42 photons because the others will have been rejected by the passband of the filter. Similarly for the other two sensels under the respective idealized Blue and Green filters. The result will be a raw count of (simplifying) (42,43,43).
Neighboring sensels in a uniform patch will produce the same results so interpolation, demosaicing or super-resolution will yield the same average values, which are 1/3 those of the Foveon (we know this is not true, but indulge me in this simplified example) with consequently lower IQ because 2/3 of the photons and their information were turned into a puff of (warm) smoke :-)
Is my question clearer now?
-
... As you can see, the interpolation guessed right regardless of the filter color (because a uniform patch is simple to interpolate) regardless of the amount of light that was recorded through the filter and the colors that were absorbed by the filter. The missing channel data and the level were interpolated/reconstructed from the surrounding pixels.
For what you acknowledge is the easiest possible case, where there is no color variation so that all three color channels are providing accurate luminosity information. Can you please at least acknowledge that real world subjects often have some significant color variation, making it somewhat harder for CFA interpolation to retain resolution while avoiding problems like moiré? At least so long as we do not have totally oversampled sensors.
-
Thanks Bart, I am starting to see where you are coming from. My question concerned the following quote:
I framed it in terms of the same sensor with or without CFA because I understand those better and I do not know Foveon at all. Let's see if I can rephrase it to include Foveon.
To make things simple I will assume APS-C size for both. As far as I understand this should mean that (simplifying) 1 Foveon sensel will have about the same area as 3 RGB ones. For a given Exposure, 384 photons of daylight spectrum will fall on the area of the Foveon sensel which (simplifying) will capture them all and sort them producing a raw count of (simplifying) (128,128,128) as you suggested.
For the same exposure a similar set of 384 photons will arrive over the 3 RGB simplified-bayer sensels, or 128 each. However, the sensel under the Red filter will only see (simplifying) 42 photons because the others will have been rejected by the passband of the filter. Similarly for the other two sensels under the respective idealized Blue and Green filters. The result will be a raw count of (simplifying) (42,43,43).
Hi Jack,
But that's not how the Foveon sensor works, so allow me to interrupt the reasoning. Simplifying, the Foveon R+G+B sensels have the same surface area as the R, or G, or B, filtered Bayer CFA sensels. Over that same surface area, they receive the same number of photons. The Bayer CFA filters out 2/3rd of the spectrum (later to be reconstructed from surrounding sensels), the Foveon type filters out the 3 spectral bands (and keeps all 3/3rd) by using the absorption depth of visible light in silicon as detectors, let's say 1/3rd Blue depth, 1/3rd Green depth, and 1/3rd Red depth. So the corresponding spectral bands receive the same number of photons. If the Bayer CFA happens to have a Green filter, it will see the same number of photons as the Green recording depth of silicon, and likewise for the other colors. It's actually a whole lot more uncertain, but that is roughly the concept.
Cheers,
Bart
-
For what you acknowledge is the easiest possible case, where there is no color variation so that all three color channels are providing accurate luminosity information. Can you please at least acknowledge that real world subjects often have some significant color variation, making it somewhat harder for CFA interpolation to retain resolution while avoiding problems like moiré? At least so long as we do not have totally oversampled sensors.
Hi,
I have no problem acknowledging that, in fact I am the one who even demonstrated (http://www.luminous-landscape.com/forum/index.php?topic=73644.msg588079#msg588079) it earlier with an extremely unlikely worst case scenario. Not only does that demonstration show that the Bayer CFA reconstruction starts to gradually fail as the fine detail approaches Nyquist (zoom in and you'll see false color artifacting), but it can even result in a loss of half of the resolution (in that unlikely worst case scenario).
I'm just trying to keep some explanations simple enough to follow. I thought that the concept of interpolation in a uniform area was easier to follow.
Cheers,
Bart
-
Conceptually its like using the spot healing brush to remove dust spots all over the image. Obviously it works well as the output of our cameras shows. For pixel level changing detail it would be inferior to foveon or full sensor full color.
3CCD and 3 CMOS are used in the video camera realm, it's too bad its not brought to cameras. A 3 chip camera would be a legitimate $4,000 camera to me.
-
If the Bayer CFA happens to have a Green filter, it will see the same number of photons as the Green recording depth of silicon, and likewise for the other colors. It's actually a whole lot more uncertain, but that is roughly the concept.
I think that there is some transmission loss in the Bayer CFA. So a "bare" Foveon type device without a color filter in front should possibly collect more of those photons that are otherwise absorbed in the filter layer.
-
Simplifying, the Foveon R+G+B sensels have the same surface area as the R, or G, or B, filtered Bayer CFA sensels. Over that same surface area, they receive the same number of photons. The Bayer CFA filters out 2/3rd of the spectrum (later to be reconstructed from surrounding sensels), the Foveon type filters out the 3 spectral bands (and keeps all 3/3rd) by using the absorption depth of visible light in silicon as detectors, let's say 1/3rd Blue depth, 1/3rd Green depth, and 1/3rd Red depth. So the corresponding spectral bands receive the same number of photons. If the Bayer CFA happens to have a Green filter, it will see the same number of photons as the Green recording depth of silicon, and likewise for the other colors. It's actually a whole lot more uncertain, but that is roughly the concept.
Ah, ok. Therefore with that assumption 1 ideal Foveon 'pixel' is the same size as 1 Bayer 'sensel' (as opposed to the three I had imagined) in current DSLRs. In that case the result of our simplified thought experiment would indeed be what you suggest, raw values (128,128,128) vs (128,0,0) respectively, and with a Bayer pattern in a uniform patch you could indeed take a guess at the missing information.
In this case too, however, the 2/3s of information thrown away would be apparent. Let's say for instance that the pixel under examination was receiving light from an isolated star that produced a circle of confusion on the sensor with a diameter equal to the pixel pitch: for a given exposure 384 photons recorded by the ideal sensor; 128 by the ideal Bayer, 1/3 the SNR for the Bayer. Of course if you took pictures of scenes with a lot less detail, say a foggy day, you could fill-in a lot of the missing information by interpolation. But then you would not need a sensor with such a high resolution. So the issue is still there, simply shifting from noise to resolution and back.
While I was thinking about your comment, I decided to compare for fun the Sigma SD15 (pixel pitch about 8um) to the Nikon D3200 (sensel pitch about 4 um): 1 SD15 contains almost exactly 1 D3200 RGBG quadruplet - now we are back to my example in the post above with SNR the issue instead of resolution. But let's take it a step further: assume comparing the SD15 sensor to a Bayer with a 1um pixel pitch: now 16 Bayer quadruplets fit inside 1 foveon pixel. You use the best demosaicing algorithm in town... You see where this is going?
So at one extreme the missing information results in degraded resolution, at the other noise performance. If we keep the resolution constant, it seems to me that removing the CFA would indeed improve noise performance by a factor equal to the absorption ratio of the filters. Of course we would then lose all color information. This is one reason why manufacturers are using weaker and weaker CFAs, relying ever more on the 'demosaicing' capabilities of their in-camera engines, a sort of horizontal (vs vertical) Foveon :-)
Jack
-
I think that there is some transmission loss in the Bayer CFA. So a "bare" Foveon type device without a color filter in front should possibly collect more of those photons that are otherwise absorbed in the filter layer.
Hi,
Yes, in an ideal world. But don't forget that the Foveon photosites require a significant amount of transfer gates and possibly transistors, that take up a lot of real estate that's no longer available for letting in photons. In addition, I've seen diagrams (a long time ago) of a somewhat circular and ring like doping structure used to sample at different silicon penetration depths.
On the other hand, we also do not know exactly how transparent the CFA filters are and how effective the doping is in transferring energy, and how effective the microlenses, if any, are in condensing light onto the photosensitive areas. The filters are not perfect band-pass filters, and they also have secondary transmission outside their target band.
And all filters are transparent to IR, so how strong is the IR filter layer really (i.e. how pure is the signal after subtracting the residual IR contribution from all color bands)? Silicon is also transparent for IR, but how much is still recorded due to scatter?
Therefore, for a thought experiment to explain the principle, lets assume they are equal. At least it won't complicate the discussion more than necessary.
But for a general discussion, I agree. I've heard quantum efficiency numbers for the current Foveon of some 25-30% and of Bayer CFA designs at 40-50%, so any reputable sources to support that would be welcome for the general discussion.
Cheers,
Bart
-
Ah, ok. Therefore with that assumption 1 ideal Foveon 'pixel' is the same size as 1 Bayer 'sensel' (as opposed to the three I had imagined) in current DSLRs. In that case the result of our simplified thought experiment would indeed be what you suggest, raw values (128,128,128) vs (128,0,0) respectively, and with a Bayer pattern in a uniform patch you could indeed take a guess at the missing information.
In this case too, however, the 2/3s of information thrown away would be apparent. Let's say for instance that the pixel under examination was receiving light from an isolated star that produced a circle of confusion on the sensor with a diameter equal to the pixel pitch: for a given exposure 384 photons recorded by the ideal sensor; 128 by the ideal Bayer, 1/3 the SNR for the Bayer.
Hi Jack,
Yes, there is a difference between the 384 photons recorded, and the 128 photons recorded for a single isolated spike signal (that's why, in addition to lens blur, an OLPF makes sense). However, the photon shot noise is equal to the square root of the number of photons, so the difference would be 384/sqrt(384) = 19.6, versus 128/sqrt(128) = 11.3 . That doesn't account for the fact that the per channel noise adds in quadrature, and that interpolated channels (from a properly low-pass filtered image source) will probably have a lower than actual spatial noise frequency (interpolation usually gives some loss of modulation due to the weighted averaging), and that should be included in the total equation.
For a better simulation of the S/N ratios due to 1 channel versus 3 channel sampling, one could add Poisson noise to a test image, and measure the differences before and after demosaicing. In fact, here is the result for a specific (VNG) demosaicing algorithm, only Poisson (shot) noise was added (no read noise):
A patch of uniform Gray level 128, Poisson Noise added, [R,G,B] Standard Deviation = [11.230, 11.355, 11.314]
(http://bvdwolf.home.xs4all.nl/temp/PixInsight/GrayPoisson128.png)
Here is the ideal CFA version of that patch with zero contribution for 2/3rds of each pixel:
(http://bvdwolf.home.xs4all.nl/temp/PixInsight/GrayPoisson128+CFA.png)
And here is the result after VNG demosaicing, [R,G,B] Standard Deviation = [10.301, 9.485, 10.274]
(http://bvdwolf.home.xs4all.nl/temp/PixInsight/GrayPoisson128_CFA_VNG.png)
As you can see, the noise was blurred by averaging and by undersampling, and overall noise was not increased but slightly reduced. A simpler demosaicing algorithm, e.g. bilinear, would have blurred even more but would also have lost more real detail had it been present.
Of course if you took pictures of scenes with a lot less detail, say a foggy day, you could fill-in a lot of the missing information by interpolation. But then you would not need a sensor with such a high resolution. So the issue is still there, simply shifting from noise to resolution and back. And people appear to like having clean images and/or more resolution these days of cameras that end in 'e' :-)
Correct, and there are several other issues, some of which have not been mentioned yet. One that may partly be related to the relatively small charge capacity wells of the Foveon design (it needs to store 3 channel charges in the same area that the CFA design can allocate for one channel), is how effective is the color filtering by silicon penetration depth really? When one inspects the Raw data of a Foveon capture, the Raw data looks almost like a monochrome image. There is hardly any difference between the color channels, which means that some serious heavylifting needs to be done on that data to boost saturation, and with that comes color noise amplification. It's one of the reasons that Foveon sensors are relatively poor at higher ISO settings.
While I was thinking about your comment, I decided to compare for fun the Sigma SD15 (pixel pitch about 8um) to the Nikon D3200 (sensel pitch about 4 um): 1 SD15 contains almost exactly 1 D3200 RGBG quadruplet - now we are back to my example in the post above with SNR the issue instead of resolution. But let's take it a step further: assume comparing the SD15 sensor to a Bayer with a 1um pixel pitch: now 16 Bayer quadruplets fit inside 1 foveon pixel. You use the best demosaicing algorithm in town... You see where this is going?
Yes, there is no such thing as Bayer quadruplets, unless one does binning which averages noise.
So at one extreme the missing information results in degraded resolution, at the other noise performance. Some believe that truth lies in the middle: perhaps it's because that way six of one can ignore the half dozen of the other ;-)? Correct me if I am wrong.
The technologies do not scale down with equal ease. Remember what I said about the well depth for 3 channels versus 1 channel on the same area of silicon real estate ... Being able and store 3x as many electrons for a color channel will reduce the noise to 58%.
And then there is the fact that Red, Green, and Blue do not contribute equally to Luminance ...
Cheers,
Bart
-
Bart,
I'm not (maybe later) buying your explanation on bayer having better full well capacity based on foveon being layered. The foveon technology is saying nothing from blue will go deeper than they make their layer. The same for green. Being able to make a blue bayer sensor deeper is not going to do anything.
I can accept the value of putting the electronics underneath so as to not interfere with the light path. Of course foveon must put electronics in the side with their layers. Good micro-lenses would even that advantage quite a bit.
-
Bart,
I'm not (maybe later) buying your explanation on bayer having better full well capacity based on foveon being layered. The foveon technology is saying nothing from blue will go deeper than they make their layer. The same for green. Being able to make a blue bayer sensor deeper is not going to do anything.
I can accept the value of putting the electronics underneath so as to not interfere with the light path. Of course foveon must put electronics in the side with their layers. Good micro-lenses would even that advantage quite a bit.
Hi,
There is not that much usable info published to go on. The Foveon .X3F file format cannot be analysed with e.g. Rawdigger, so we'll have to make do with recent summaries like: http://people.rit.edu/hxc1311/ChenDetectorPaper.pdf (http://people.rit.edu/hxc1311/ChenDetectorPaper.pdf)
That document mentions the detection design, but not the actual charge storage model. How it really works, is anyone's guess. It also repeats some charts and formulas from older Foveon papers, such as the penetration depth dependency on angle of incidence (hence my assumption that large format sensors with more oblique incident light are not likely to happen), and the problematic color separation.
Maybe there is some info in the patent applications that's a bit more specific about the design 'improvements'.
Cheers,
Bart
-
Bart,
As I mentioned in the medium format discussion on CCD vs CMOS:
You know a lot more about the technology than me.
What I will say is I have seen 3 chip HD video vs 1 chip HD video. The 3 chip systems look way better. Maybe an order of magnitude better. Go to your local electronics store and compare the cameras for yourself. Panasonic makes a nice 3CMOS camcorder. Compare it to any manufacturer using 1 chip of similar size. Not the sony nex, that is a much bigger chip.
Edit: by compare I mean shoot video in the store with each. Output it to a HDTV.
If the theory of Bayer is so close to delivering as much as non- bayer why do 3 chip HD camcorders seem to have such a huge advantage? I am not talking theory, I am talking the units for sale in the stores. About a year and a half ago I was undecided on a new camera. When I was in the store they had the camcorders that take stills right beside the cameras. I spent some time in several stores comparing the models. There was one camcorder that seemed to have a massive advantage as output was viewed on a HDTV. The Panasonic 3 chip cameras. Anyone can go to test for themselves in their local electronics store. I doubt much as changed in the last year.
Record some video in the store. Why wouldnt 3 chip or non-bayer via some other method not have the same advantage? My understanding is almost all Pro video systems use 3 chips.
-
Hi,
My understanding is that pro video is mostly single chip. I'm pretty sure Arri Alexa is single chip and so is Red One.
I'm not sure that a three chip design could be fit into a DSLR , as they need a beam splitter and three sensors. The three sensors need to be aligned within perhaps 2 microns for all pixels if resolution is not to be diminished.
It seems that the Foveon, works best at low ISO's which probably means that it has a relatively low quantum efficiency at the system level or that signal processing needed to extract color information demands very low noise levels.
Best regards
Erik
Bart,
As I mentioned in the medium format discussion on CCD vs CMOS:
If the theory of Bayer is so close to delivering as much as non- bayer why do 3 chip HD camcorders seem to have such a huge advantage? I am not talking theory, I am talking the units for sale in the stores. About a year and a half ago I was undecided on a new camera. When I was in the store they had the camcorders that take stills right beside the cameras. I spent some time in several stores comparing the models. There was one camcorder that seemed to have a massive advantage as output was viewed on a HDTV. The Panasonic 3 chip cameras. Anyone can go to test for themselves in their local electronics store. I doubt much as changed in the last year.
Record some video in the store. Why wouldnt 3 chip or non-bayer via some other method not have the same advantage? My understanding is almost all Pro video systems use 3 chips.
-
B&H's Pro Video Studio equipment page
JVC, Panasonic, Sony all 3 chip units
http://www.bhphotovideo.com/c/buy/Studio-EFP-Cameras/ci/16764/N/4256818816 (http://www.bhphotovideo.com/c/buy/Studio-EFP-Cameras/ci/16764/N/4256818816)
-
Yes,
Small chip devices, 1/3". I thought you were thinking larger sensors. The equipment I mentioned is higher up the scale.
Best regards
Erik
B&H's Pro Video Studio equipment page
JVC, Panasonic, Sony all 3 chip units
http://www.bhphotovideo.com/c/buy/Studio-EFP-Cameras/ci/16764/N/4256818816 (http://www.bhphotovideo.com/c/buy/Studio-EFP-Cameras/ci/16764/N/4256818816)
-
Perfectly reasonable, I think most pro video is the stuff used for all HDTV broadcast every day.
You are talking hot new movie studio cameras.
-
Hi Bart, I see we are starting to deviate from the initial 'ideal' thought experiment :-)
the photon shot noise is equal to the square root of the number of photons, so the difference would be 384/sqrt(384) = 19.6, versus 128/sqrt(128) = 11.3 . That doesn't account for the fact that the per channel noise adds in quadrature, and that interpolated channels (from a properly low-pass filtered image source) will probably have a lower than actual spatial noise frequency (interpolation usually gives some loss of modulation due to the weighted averaging), and that should be included in the total equation.
Ok, in this case we need to decide whether we are dealing with a uniform patch of tones or the single sensel (star) version. I take it from your example that we are looking at a uniform patch, so the first image works and we are forgetting about the loss of detail. For simplicity, let's assume that the standard daylight light source can be filtered perfectly by three equally sized bands (vertically by the Foveon sensor and horizontally by the Bayer) and that the area of one Foveon Pixel is the same as that of a single Bayer sensel (the Green one in your example) so that the sensors would output (128,128,128) and (0,128,0) to their respective R*G*B* raw data.
The Bayer sensor would therefore produce a matrix of repeating data such as (128,0,0) (0,128,0) (128,0,0)... in a 'Red' row followed by an offset repeating (0,0,128) (0,128,0) (0,0,128)... in the 'Blue' row and so on. On the other hand the Foveon would produce an equal number of raw data points of value (128,128,128). As you say the value 128 above is the mean, while in fact it would vary from raw value to raw value according to Poisson statistics. These variations would be restricted to the recorded values and not spill over from one sensel to the one next to it because they are inherent in the incoming photons which, if filtered, would simply not be there.
If the above is correct, then the second image appears too sparse and too noisy, but I assume you did not use it for demosaicing since the final one looks correct :-) In theory any demosaicing noise improvement in the ideal Bayer will only result from a further reduction in the substantially lower detail available (the real differentiator in the 'uniform patch' case).
On the other hand, in the case where sensor resolution and detail are part of the equation the answer is quite simple if we simplify things a bit: 384 daylight spectrum photons would yield an average SNR of 19.6. From the ideal Foveon's raw data we could write SNR Foveon (in quadrature) = sqrt(128+128+128) = 19.6. From the ideal Bayer's raw data, since we only have 1/3 of the photons the answer would be, still in quadrature, sqrt(128) = 11.3 as you suggested.
Yes, there is no such thing as Bayer quadruplets, unless one does binning which averages noise.
Yes, in a uniform patch where detail is 'binned'. On the other hand in the case of a twinkly little star... :-)
Jack
-
On further thought I may have misunderstood your comment below
Yes, there is no such thing as Bayer quadruplets, unless one does binning which averages noise.
in response to my earlier one
compare for fun the Sigma SD15 (pixel pitch about 8um) to the Nikon D3200 (sensel pitch about 4 um): 1 SD15 contains almost exactly 1 D3200 RGBG quadruplet - now we are back to my example in the post above with SNR the issue instead of resolution
To clarify, there is no need for binning in this thought experiment, just simple demosaicing.
Four ideal Bayer Sensels (one red, one blue and two green) cover exactly the same real estate as one ideal Foveon Pixel in the example above. If, for a given exposure as before, 384 photons of D50 light arrive on such an area of the Foveon sensor, it will record (128,128,128) for this position in its R*G*B* raw file. On the other hand the same 384 photons will arrive on the same area of a Bayer sensor - but each of the sensels, being 1/4 the area of the Pixel, will only see 1/4 of them on its turf, that is 96. And since each sensel is covered by a perfect bassband filter that only lets through 1/3 of the arriving daylight photons, the sensor will record (32,0,0) (0,32,0) (0,0,32) (0,32,0) in its R*G*B*G* raw file.
For simplicity let's assume that no demosaicing is needed for the Foveon and a simple algorithm (say -h) is used to demosaic the Bayer data. The result would be (32,32,32) with the green data cleaner than red and blue because the two greens were averaged. And, ignoring different weights for simplicity, the relative SNRs would be sqrt(128*3)=19.6 in favor of the Foveon vs sqrt(32*4)=11.3 for the Bayer as you mentioned earlier. The difference being entirely due to the CFA which reduces the signal.
Contrary to the others we dreamed up, imo this example is better at comparing apples to apples because the resolution from both ideal sensors should be similar, including the effects of a 4-dot beam splittin' antialiasing filter. Therefore the advantage of not having a CFA is clearer, reflecting a fairer compromise between noise and detail in the two approaches.
Cheers,
Jack
-
Here are 2 100% crops out of a high ISO shot. Notice the tendency to make red and green splotches? We have gotten so used to just turning on a bit of NR that wipes it out. Is it really noise or is it de-bayering? The ISO was to crank the gain making the splotches visible. A low ISO shot does not show them in any visible way. Are they there in a subtle way?
-
Hi,
Shot noise. Natural variation in number of photons collected. That is what I think. May be something else. Increase "nose reduction->color" if you are using Lightroom. If it is shot noise there is little else to do about it.
At low ISO you sample more photons so the problem goes away.
Best regards
Erik
Here are 2 100% crops out of a high ISO shot. Notice the tendency to make red and green splotches? We have gotten so used to just turning on a bit of NR that wipes it out. Is it really noise or is it de-bayering? The ISO was to crank the gain making the splotches visible. A low ISO shot does not show them in any visible way. Are they there in a subtle way?
-
Hi,
Shot noise. Natural variation in number of photons collected. That is what I think. May be something else. Increase "nose reduction->color" if you are using Lightroom. If it is shot noise there is little else to do about it.
At low ISO you sample more photons so the problem goes away.
Best regards
Erik
Maybe people call it shot noise. My point is what should be a brown field is turned into red and green. The same on the darker mountain spots. Why is the color changed? If it is noise it should be fluctuations around brown not red green.
-
Hi,
The sensor doesn't see browns but reds or greens. So you have a variation in reds and greens.
The natural variation is essentially the square root of the number of photons collected. Let's assume that you collect 50000 photons at saturation. Now, let us assume neutral 18% gray, that is about 9000 photons. Let us further assume that you shoot at 800 ISO, with nominal ISO being 100. Than you would collect 1125 photons per pixel. Let assume that you are looking at a part that is in the shadow, say two stop under. Now we are at 280 photons.
So if we have 280 photons, natural variation would be +/- 16 sof photon count would vary between 263 and 296. Actually 65% of the pixels would vary between 263 and 296, if I recall statistics right. So you get a lot of natural variation. Nothing to do about it, except increasing exposure.
If you want to learn about noise, I would recommend this article: http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html
Best regards
Erik
Maybe people call it shot noise. My point is what should be a brown field is turned into red and green. The same on the darker mountain spots. Why is the color changed? If it is noise it should be fluctuations around brown not red green.
-
Hi,
The sensor doesn't see browns but reds or greens. So you have a variation in reds and greens.
The natural variation is essentially the square root of the number of photons collected. Let's assume that you collect 50000 photons at saturation. Now, let us assume neutral 18% gray, that is about 9000 photons. Let us further assume that you shoot at 800 ISO, with nominal ISO being 100. Than you would collect 1125 photons per pixel. Let assume that you are looking at a part that is in the shadow, say two stop under. Now we are at 280 photons.
So if we have 280 photons, natural variation would be +/- 16 sof photon count would vary between 263 and 296. Actually 65% of the pixels would vary between 263 and 296, if I recall statistics right. So you get a lot of natural variation. Nothing to do about it, except increasing exposure.
If you want to learn about noise, I would recommend this article: http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html
Best regards
Erik
I don't think so. Noise is random at the pixel level. These are splotches >10 pixels diameter. Calculate the probability that a pattern of spaced red pixels would be hit with higher levels of photons over 10x10 pixels, then spaced green pixels 10x10 interweaving all over the shot the same. Maybe an 800% view will make it look less random.
Emil's article is exellent btw, I dont think this pattern is shot noise.
-
It's a software issue. I opened the image in Sony's IDC which showed the noise sprinkled at the pixel level instead of the large splotches. Attached is the IDC version, NR off, sharpening off.
The chroma speckles are a fairly smooth random pattern. You can still see some areas where larger splotches of green show up. De-bayering is a software interpretation routine. It can make a mistake.
I had replied to another thread today about using Noise Ninja for film grain. They had a new version on their website. After installation of the new "picture ninja" I saw wow, it does raw conversion! To test the NR I opened a high ISO raw which led to the god awful color splotches above. They should stick to NR. Beware bad de-bayering.
-
Hi Bart, I see we are starting to deviate from the initial 'ideal' thought experiment :-)
Ok, in this case we need to decide whether we are dealing with a uniform patch of tones or the single sensel (star) version.
Hi Jack,
Actually, I'm a bit at a loss as to what it is that you are trying to ask, tell, or suggest.
Are you trying to make some kind of statement about resolution, or noise? Either way, I'm most willing to explain the situation as I see it, but I do have some difficulty with the various totally unrealistic scenarios that are being proposed, hence my simplification to a uniform area in an attempt to focus on a concept that's simple to understand, with relatively simple math, although even that's being used to stack the deck against the Bayer CFA by using unrealistic scenarios.
I take it from your example that we are looking at a uniform patch, so the first image works and we are forgetting about the loss of detail.
Correct, for the reason given above.
For simplicity, let's assume that the standard daylight light source can be filtered perfectly by three equally sized bands (vertically by the Foveon sensor and horizontally by the Bayer) and that the area of one Foveon Pixel is the same as that of a single Bayer sensel (the Green one in your example) so that the sensors would output (128,128,128) and (0,128,0) to their respective R*G*B* raw data.
Okay sofar, when seen from the perspective of what will be a single output pixel, although it can require some 49 adjacent Bayer CFA samples to reconstruct a single central RGB output pixel.
The Bayer sensor would therefore produce a matrix of repeating data such as (128,0,0) (0,128,0) (128,0,0)... in a 'Red' row followed by an offset repeating (0,0,128) (0,128,0) (0,0,128)... in the 'Blue' row and so on.
Yes, although there are no 'Red', and 'Blue' rows, but I understand what you are describing, 2 rows from a Bayer CFA filtered larger array.
On the other hand the Foveon would produce an equal number of raw data points of value (128,128,128). As you say the value 128 above is the mean, while in fact it would vary from raw value to raw value according to Poisson statistics. These variations would be restricted to the recorded values and not spill over from one sensel to the one next to it because they are inherent in the incoming photons which, if filtered, would simply not be there.
The per sensel (shot noise) photon statistics have a Poisson distribution indeed, and they are independent per sensel (in our simplified model).
If the above is correct, then the second image appears too sparse and too noisy, but I assume you did not use it for demosaicing since the final one looks correct :-)
I used the first image (idealized RGB Foveon model) to produce a Bayer CFA version (second image), which in turn was used to demosaic (third image).
In theory any demosaicing noise improvement in the ideal Bayer will only result from a further reduction in the substantially lower detail available (the real differentiator in the 'uniform patch' case).
That's not correct, and I mentioned the reasons. The demosaicing algorithm, when faced with random noise in a uniform area, will reduce the overall noise for two reasons. The first reason is that random noise, when averaged over a region will average out at a rate that's equal to the square root of the noise variances (AKA the standard deviations add in quadrature), and 2/3rds have zero noise because it's zero (in our theoretical model). The second reason is that the Red and Blue sensels are less densely sampled, and thus have a lower spatial (and thus Luminance, and thus amplitude) contribution in the demosaiced result.
Cheers,
Bart
-
To clarify, there is no need for binning in this thought experiment, just simple demosaicing.
Okay, let's stick to that for now, for clarity.
Four ideal Bayer Sensels (one red, one blue and two green) cover exactly the same real estate as one ideal Foveon Pixel in the example above.
When 4 Bayer CFA sensels cover the same real estate as a single Foveon type of sensel, then the Bayer CFA sensels will have a 2x higher sampling density, and thus almost 2x higher luminance resolution, and one quarter of the surface, each. What happened to our simple one R + G + B sample has the same area as one R or G or B sample?
If, for a given exposure as before, 384 photons of D50 light arrive on such an area of the Foveon sensor, it will record (128,128,128) for this position in its R*G*B* raw file. On the other hand the same 384 photons will arrive on the same area of a Bayer sensor - but each of the sensels, being 1/4 the area of the Pixel, will only see 1/4 of them on its turf, that is 96. And since each sensel is covered by a perfect bassband filter that only lets through 1/3 of the arriving daylight photons, the sensor will record (32,0,0) (0,32,0) (0,0,32) (0,32,0) in its R*G*B*G* raw file.
That doesn't make sense when you view the Bayer CFA sensels as individual sensels and want to compare them to the 4x larger Foveon type sensor. It would require taking the 4 Bayer CFA sensels together (as a sort of RGGB binned pixel) to allow any type of comparison, however flawed it would still be from a practical (demosaicing) point of view.
For simplicity let's assume that no demosaicing is needed for the Foveon and a simple algorithm (say -h) is used to demosaic the Bayer data. The result would be (32,32,32) ...
No, the demosaiced result would be [128,128,128] for the 4 theoretically perfect output pixels (the same as the Foveon type of sensor) added together to the same surface area.
Contrary to the others we dreamed up,
I dream with my eyes closed ...
imo this example is better at comparing apples to apples because the resolution from both ideal sensors should be similar,
No it isn't, IMHO. It's not even apples to oranges, but rather kiwis to kangaroos ...
including the effects of a 4-dot beam splittin' antialiasing filter.
Why obfuscate the analogy by adding an OLPF (let's guess, for one and not the other)? The ideal discrete sampling situation would include an OLPF for both types of sensor, and also incorporate the effect of lens blur (which would make it impossible to on average address only a single sensel with a spike like signal, because the Nyquist/Shannon theorem requires more than 2 sensels for a reliable reconstruction of a signal). It would also have to include the less than perfect color separation as a function of penetration depth in silicon, and the less than 100 percent transmission in the pass band of the CFA filters, even when ignoring various noise sources and optical/MTF effects.
Cheers,
Bart
-
Hi Jack, Actually, I'm a bit at a loss as to what it is that you are trying to ask, tell, or suggest.
Hi Bart, that makes two of us :-) Actually I believe that we are in full agreement, with the possible exception of this statement which goes to the root of (my) confusion:
JH Wrote: 'For simplicity let's assume that ... a simple algorithm (say -h) is used to demosaic the Bayer data. The result would be (32,32,32) ...'
No, the demosaiced result would be [128,128,128] for the 4 theoretically perfect output pixels (the same as the Foveon type of sensor) added together to the same surface area.
This makes no sense to me, unless we bring into the discussion the difference between brightness and exposure of my first post, with related consequences on SNR and IQ.
Forget about Foveon for a second and think only about the Bayer in my example. We are looking at a square area A on the ideal Bayer sensor made up of 4 sensels in a 2x2 matrix, each of area A/4: 1 under a red , 1 under a blue and 2 under a green ideal color filter (CFA). If 384 photons reach area A, each A/4 filter area will only see 1/4 of them or 96. And since each filter only lets through 1/3 of them in our idealized example, the sensel underneath each filter will receive 32 photons and that's the value it would record in the raw data for each of the four sensels of our investigation. Simple demosaicing of the raw data (for instance with dcraw -h 'half' switch, which keeps the red and blue values as 'they are' and averages the greens) would produce a single R*G*B* Pixel for the whole of area A of value (32,32,32), with a given SNR, keeping in mind the earlier proviso on the green channel - because demosaicing works off the raw data. More complicated demosaicing would give the same result, but it would be harder to follow. Of course we could in fact express this as any value we desired through digital post processing operations (let's call them brightness/tonal corrections) in-camera or in-computer, but the underlying information and SNR (IQ) would remain unchanged.
Now let's shrink the sensels: If area A contained 64 smaller Bayer sensels instead of 4, once downrez'd to a single ideal Pixel for area A, for the given SNR as before such a pixel would have the exact same value (32,32,32).
Is my confusion with your (128,128,128) statement above clearer now?
Jack
-
Actually I believe that we are in full agreement, with the possible exception of this statement which goes to the root of (my) confusion:
No, the demosaiced result would be [128,128,128] for the 4 theoretically perfect output pixels (the same as the Foveon type of sensor) added together to the same surface area.
This makes no sense to me, unless we bring into the discussion the difference between brightness and exposure of my first post, with related consequences on SNR and IQ.
Maybe you missed the part of my quote I've marked in italic bold here. You insisted on comparing a 4x larger photon collection area with single Bayer CFA sensels, so in order to make a valid comparison, one would have to add the four [32,32,32] interpolated Bayer CFA ones together, which gives [128,128,128].
Forget about Foveon for a second and think only about the Bayer in my example. We are looking at a square area A on the ideal Bayer sensor made up of 4 sensels in a 2x2 matrix, each of area A/4: 1 under a red , 1 under a blue and 2 under a green ideal color filter (CFA). If 384 photons reach area A, each A/4 filter area will only see 1/4 of them or 96. And since each filter only lets through 1/3 of them in our idealized example, the sensel underneath each filter will receive 32 photons and that's the value it would record in the raw data for each of the four sensels of our investigation.
Correct, 384/4 sensels = 96, and with 1/3rd bandpass filtering 96/3 = 32.
Simple demosaicing of the raw data (for instance with dcraw -h 'half' switch, which keeps the red and blue values as 'they are' and averages the greens) would produce a single R*G*B* Pixel for the whole of area A of value (32,32,32), with a given SNR, keeping in mind the earlier proviso on the green channel - because demosaicing works off the raw data.
Correct, see also the above explanation, [32,32,32] is the result after demosaicing.
More complicated demosaicing would give the same result, but it would be harder to follow. Of course we could in fact express this as any value we desired through digital post processing operations (let's call them brightness/tonal corrections) in-camera or in-computer, but the underlying information and SNR (IQ) would remain unchanged.
I'm not sure why you are mentioning the S/N ratio here but, as you can see in my demonstration earlier, the noise amplitude at the pixel level will be reduced and replaced by a lower spatial frequency noise pattern.
Now let's shrink the sensels: If area A contained 64 smaller Bayer sensels instead of 4, once downrez'd to a single ideal Pixel for area A, for the given SNR as before such a pixel would have the exact same value (32,32,32).
Not really, apart from the practical implications which do not scale down perfectly with geometry. Dividing an area that receives 384 photons in 64 will leave 6 photons on average, an after a 1/3rd bandpass filter that would become 2 photons each. But in the theoretical example there are still the same number of photons falling on the same total area, and there is still 2/3rds being filtered out. So the remaining 1/3rd times the original 384 photons for the total area still makes 128 (64 sensels times 2 photons). When you divide the same area up in smaller sample areas, then each sample will detect fewer photons but they still add up to the same number for the area, 1/3rd sampled, 2/3rd interpolated. It will have hardly any effect on the S/N ratio for the total area, none actually in our theoretical example.
Is my confusion with your (128,128,128) statement above clearer now?
I stiil think it's the 4x larger Foveon type of sensor in your original example that's the basis for any possible confusion. One needs to compare equal areas for a meaningful comparison.
I'm not sure whether there is some subconsciously nagging issue (which is understandable) with the fact that while only '1/3rd' of our photons actually are registered, 2/3rd will be supplemented by interpolation to create full RGB output pixels. Those RGB output pixels which have the (approximately) identical brightness as a full RGB sensor would give, as my demosiacing demonstrations earlier in the thread show. The Bayer CFA converted originals look darker, because 2/3rd of their RGB pixel data is zero, but after supplementing the zeros with interpolated/reconstructed data, the original average brightness is restored.
I do not think that another confusion plays a role here, namely a misconception about demosaicing where some people think that it takes 4 Bayer CFA sensels to make 1 RGB output pixel. That would be a completely wrong representation of how demosaicing works (and it would result in half of the resolution that is actually recorded, which proves that that representation is flawed). But for those who believe that's how demosaicing works, it doesn't.
Cheers,
Bart
-
I'm not sure why you are mentioning the S/N ratio here...
If area A contained 64 smaller Bayer sensels instead of 4, once downrez'd to a single ideal Pixel for area A, for the given SNR as before such a pixel would have the exact same value (32,32,32)
Not really, apart from the practical implications which do not scale down perfectly with geometry.
Here I was referring to the fact that my quote above would result in 64 (2,2,2) demosaiced pixels, each with a SNR of 1.4. After downsizing the 64 pixels into one single Pixel for area A, the resulting value for the Pixel would be (2,2,2) with an SNR of 11.3 - which could be brightness corrected in post to be (32,32,32) [or whatever] at that same SNR.
I think we are fully aligned :-)
-
To be fair to Picture Code, their new Picture Ninja opens the file to a fairly good looking raw conversion. It was me turning their NR reduction sliders down all the way that ended up with the splotched de-bayering posted above. Clearly they did not intend for their system to be used that way. Still, having seen it, inspection of their default raw conversion does show very mild red green regions.
This would never be an issue with foveon or another full chip color capture system.