Light loss in a camera using a Bayer CFA

[joofa]

Hi,

Motivated by some discussion on DPReview I did an experiment to visually depict the light loss while using a Bayer CFA on a sensor. The original message is below:

http://forums.dpreview.com/forums/read.asp?forum=1018&message=36488951

And, a follow up message is below:

http://forums.dpreview.com/forums/read.asp?forum=1018&message=36494647

Enjoy the images,

Joofa

[Ray]

One wonders what sort of performance a D3s might be capable of if Nikon were to introduce a modified model without color array filter and AA filter. Perhaps a B&W-only camera would have limited apeal. On the other hand the bulk of the R&D for the camera has aleady been done, so one might expect the modifications for a B&W-only design to be relatively inexpensive, and the retail price of the camera could be lower due to the absence of relatively expensive filters.

The resolution and lack of noise at high ISO should be stunning.

[hjulenissen]

One wonders what sort of performance a D3s might be capable of if Nikon were to introduce a modified model without color array filter and AA filter. Perhaps a B&W-only camera would have limited apeal. On the other hand the bulk of the R&D for the camera has aleady been done, so one might expect the modifications for a B&W-only design to be relatively inexpensive, and the retail price of the camera could be lower due to the absence of relatively expensive filters.

The resolution and lack of noise at high ISO should be stunning.

If you shoot a lot of scenes where the luminance of the scene is constant and all of the information is seen as variations in either red or blue channel, then yes, I would expect a significant increase in resolution and noise.

Most scenes are not like that, though, and for them I think that the Bayer sensor is a good compromise, and one where a lot of research and economy of scale helps it to shine compared to alternatives.

Perhaps the biggest benefit of a mono-chrome camera would be that the AA filter could be weaker, allowing more sharpness, just like Foveon sensors. On the other hand, who wants to have to use actual color filters?

-h

[AJSJones]

Makes me wonder about the parallel experiment one might think of for film, with its multiple filter and dye layers and how much light gets through to which layer.  Having all colors of photons collected in each well, one expects the well to fill sooner, or the brightness to be greater for a given collection time, no?  This is the price we pay for color photography, is it not?   Didn't B/W film do better than color, partly for this reason?  Film and Foveon do better than Bayer because of the sequential filtering at each location, rather than location filtering (and throwing away photons of other wavelengths from that location), but they also have their issues. Three successive exposures though three colored filters onto a B/W array would give the most information but each would be darker than the whole B/W shot. What do we learn from this interesting experiment, I wonder?

[hjulenissen]

...Film and Foveon do better than Bayer because of the sequential filtering at each location, rather than location filtering (and throwing away photons of other wavelengths from that location), but they also have their issues.

I think "better" is a vague word. "Different" is better, and the market seems to agree that the prefer Bayer amon those options commercially available right now.

Three successive exposures though three colored filters onto a B/W array would give the most information but each would be darker than the whole B/W shot. What do we learn from this interesting experiment, I wonder?

"3-chip" video camera tech would allow every photon to be counted, while using simple monochrome sensors instead of the Foveon sensors (with its drawbacks).

-h

[AJSJones]

I think "better" is a vague word.

The word "better" was in the specific context of capturing photons that land on a particular area (i.e. better = captures more) not an assessment of how good the sensor is in totality!

"3-chip" video camera tech would allow every photon to be counted

Yes, the systems that split each area into separate colours and capture photons from those three colours will capture more photons than the ones that only capture one colour's worth (i.e. Bayer types).  Thus the latter will be darker.  That's the initial observation, so I wondered whether we were to be surprised by the result.

[hjulenissen]

The word "better" was in the specific context of capturing photons that land on a particular area (i.e. better = captures more) not an assessment of how good the sensor is in totality!...

Would not the Foveon high iso performance indicate that in practice, it is worse at counting photons than bayer (and that any mental model we may have indicating otherwise is indeed wrong)?

-h

[AJSJones]

I don't know what the limitations of the Foveon sensor/construction are.  Perhaps the detectors at each layer are less efficient or less transparent or have a smaller fill factor?  They are not likely to be deliberately throwing away photons, but they may not get as much signal out of each that they do get.  Maybe the capacities of the wells are smaller?  It's such a neat idea, but clearly the complexity of the fabrication is challenging to put into practice competitively.
I assume you are not surprised by the initial observation (of exposures ±Bayer filter and the number of photons collected).

[RFPhotography]

Perhaps a B&W-only camera would have limited apeal.

There have been a couple b&w only cameras in the past.  They never did well.

The Foveon sensor loses more light than a Bayer sensor because the light has to travel through 3 colour filters on each pixel rather than a single colour filter.  So while tests have shown bare resolution of Foveon sensors to be greater than a Bayer array camera with the same number of pixels, because the signal has to be boosted due to the light loss, DRange and noise aren't as good.

[AJSJones]

In principle, the light loss should be minimal for each wavelength range if the filters are good (absorb B, let R & G pass through, then absorb  G and let R pass through, then absorb R), so the difficulty must be in making the filters work well on the chip? Or the silicon detectors aren't very good?  Here is an interesting essay on other tricolor pixel design possibilities, all looking equally challenging to reduce to practice. Also discussion on a 6 color array!

[Bart_van_der_Wolf]

I don't know what the limitations of the Foveon sensor/construction are.  Perhaps the detectors at each layer are less efficient or less transparent or have a smaller fill factor?

From the schematics I've seen, one can envision the RGB sensing area as 3 somewhat concentric circular areas, due to the required gates to offload the separate resulting charges, and supporting circuitry per pixel. Remember that we're not talking about a back-illuminated chip design, so opaque transistors and gates have to be subtracted (x3) from the light gathering area. Then there is the need to store 3 different charges per pixel, which means that the realestate per RGB pixel position is 1/3rd of the area at its most for each color band.

In addition, there is a significant amount of crosstalk between the bands, IOW the bandpass transitions are not very sharp. That means that there will be a serious amplification of noise when the bands are mathematically separated (signal subtraction = noise amplification).

It's such a neat idea, but clearly the complexity of the fabrication is challenging to put into practice competitively.

Indeed. Very interesting, but a lot of trade-offs are involved. One of the trade-offs is the sensitivity to light striking the sensor at an angle, which results in a shift in the color separation by penetration depth (partly due to reflection, partly due to longer travel distance and the risk of spilling to neighboring sensel positions). Microlenses can solve part of these angle of incidence issues, but it does increase cost.

I assume you are not surprised by the initial observation (of exposures ±Bayer filter and the number of photons collected).

One has to be careful to not draw the wrong conclusions. Sure, filtering roughly 2/3rds of the light out for each Bayer CFA filtered sensel seems lossy, but don't forget that 2/3rds of the missing info will be subplanted by interpolation, and as such will compensate for the lost light. Therefore it will not reduce overall sensitivity as much as some would like us to believe.

Cheers,
Bart

[AJSJones]

One has to be careful to not draw the wrong conclusions. Sure, filtering roughly 2/3rds of the light out for each Bayer CFA filtered sensel seems lossy, but don't forget that 2/3rds of the missing info will be subplanted by interpolation, and as such will compensate for the lost light. Therefore it will not reduce overall sensitivity as much as some would like us to believe.

Cheers,
Bart

Thanks for the info on the Foveon design details - I hadn't found them.

I have no problem with the Bayer losing some intensity in exchange for a pretty good color data set (information = energy!), especially given the progress that's been made in S/N (related to DR) over the years since I bought my D30

[hjulenissen]

In principle, the light loss should be minimal for each wavelength range if the filters are good (absorb B, let R & G pass through, then absorb  G and let R pass through, then absorb R), so the difficulty must be in making the filters work well on the chip? Or the silicon detectors aren't very good?  Here is an interesting essay on other tricolor pixel design possibilities, all looking equally challenging to reduce to practice. Also discussion on a 6 color array!

One person over at dpreview seems to know what he is talking about (though you never know on the web I guess)
http://forums.dpreview.com/forums/read.asp?forum=1000&message=36399797
http://forums.dpreview.com/forums/read.asp?forum=1000&message=36279330
http://www.alt-vision.com/documentation/AeroSense-2003-Oral.pdf

Unprocessed Foveon images look horrible. The Foveon literature always shows a sensor with a red, green, and blue layer. The way things really work out is that the top layer is sensitive to all colors, the middle layer sensitive to red, orange, yellow, and green, and the bottom layer to red and orange.
....
The Foveon had unique advantages (I know, I designed an industrial camera that used two of them) that justify putting up with the color errors, data inefficiency, and noise, but Sigma didn't capitalize on any of those advantages in their SD9, 10, 14, and 15.

I think that this is a really interesting subject, and it is sad that so much web sources are drowned in "noise" from people invested (economically or emotionally) heavily in one or the other technology.

-h

[JohnTodd]

Perhaps this is something Leica might take a look at. They've shown themselves willing to do limited, high-cost runs of their cameras  with very narrowly-targeted appeal - look at the titanium M9. Given that a lot of digital Leica users are doing and enjoying B&W photography anyway, why not a limited run of Bayerless M9s? Mmmm... lovely :-)

[AJSJones]

Thanks h - interesting reads on Foveon. 
Joe seems to know what he is talking about and one of those posts was the clearest description I've read of how Foveon tries to separate the color information - and it's far from analogous to the three or four layer film (filter/dye layers etc).  Thus the thinking that the color information comes from filter layers that resemble - in their spectral responses - the RGB filters in a Bayer array, was my error of assumption.  While I knew that the retina does a whole lot of processing (think mpeg) of the data before it sends it to the brain, I hadn't considered how the distribution of RG and B sensitive cones resembles a Bayer matrix and why the Bayer "limitations" are not easily detected by the eye/brain until taxed : ever seen images of resolution charts printed in red or blue compared to black (or green) but that's kinda how the eye sees it anyway