Sensor DR vs Camera DR

[ejmartin]

The latest generation of high end Canon and Nikon DSLR's have fairly good dynamic range and low noise. But could it be better? Almost certainly so.

Consider the Canon 1D mk3. At ISO 100, its dynamic range is a little over 11.5 stops; it's about the same at ISO 200, and then it gradually declines to about ten stops at ISO 1600. But the lower end of that range is quite noisy for each ISO, and so it's often said that the "usable" dynamic range is a good deal less.

There is good evidence that the dynamic range of these cameras is being limited not by the sensor, whose DR is about 14 stops, but rather by the electronics that implements the ISO amplification and the analog-to-digital conversion (ADC). A plot of the signal-to-noise (vertical axis, in stops) as a function of absolute exposure (horizontal axis, in stops) reveals many of the issues:



Each stop increase in ISO pushes another stop of highlights past the range of the ADC which are then lost; in the figure this is shown by the graph of S/N for a given ISO ending one stop earlier for each successive ISO. At the shadow end, increasing the ISO expands the range at low exposure, with the amount gradually tapering off until the improvement between ISO 800 and 1600 is rather small.

But the sensor doesn't know what ISO is going to be used, it just records whatever photons arrive, leaving it to circuitry off the sensor to amplify the signal and digitize it. That means that the sensor sees the *entire* range of the figure -- the upper bound or "envelope" of all the different curves. The sensor has about 14 stops of DR, but the limitations of the rest of the circuits allow the final raw data to see less than twelve stops of DR, and the user is forced to choose a "window" of EV within that 14 stop range by selecting the ISO gain.

Could it be possible to recover the full DR seen by the sensor?  

It might well be possible.  What one would like is to somehow be able to use ISO 100 to keep all the highlights, while at the same time using ISO 1600 to recover all the shadows.  But how can one have two ISO settings at once?  By having two separate amplifiers fed from the same sensor data, running in parallel.  Suppose that the sensor signal is sent to two separate processing paths, each path an amplifier and an ADC, with one amplifier set to ISO 100 and the other to ISO 1600.  The ISO 100 path keeps all the highlights but has noisy shadows; the ISO 1600 path loses the top four stops of highlights but has much better shadows.  Quantizing each, one can then combine the image data in a manner similar to HDR processing to yield an image with all 14 stops that the sensor is capable of recording.

What would the result look like?  Well of course, no such camera is currently made, but one can get an idea of the possibilities by shooting two successive images, one at ISO 100 and another at ISO 1600, and combining the two.  I did just that with the following image:



The exposure was chosen so that the light bulb was just clipping in the ISO 100 exposure, the ISO 1600 exposure was taken just after with the same exposure settings.  The two images were then combined at the raw stage, keeping the top four stops of the ISO100 data, ramping between the ISO 100 data and the ISO 1600 data in the next stop down, and then using the ISO 1600 data below that.  The raw image was then treated to Bayer interpolation, a rough white balance, and gamma correction (using IRIS, a raw data analysis program).  Areas involving the blending are the top of the lamp and the top of the bear's head:



For comparison, here's the ISO 100 shot alone with the same treatment:



One can really begin to see the noise in shadows at ISO 100 in the front of the lamp, but the blended exposure is rather clean.  In part this is because ISO 1600 exhibits much less banding noise than ISO 100, apparently the banding noise is a property of the electronics downstream of the sensor and is suppressed by the use of high ISO.

A somewhat more powerful statement is made by the colorchecker chart; first the ISO 100 shot alone:



and now the blended exposure (which comes in this EV range entirely from the ISO 1600 component):



The bottom left square of the color chart is a bit over 9 stops down from raw saturation (EDIT: in the green channel; due to the tungsten lighting, the blue channel is another two stops below that).

It would be nice, to say the least, if the implementation of this sort of dual amplification/HDR blend were possible, and used in a production camera.

[BruceHouston]

Great job, Emil!

Development of in-camera processors has just recently reaching the point of coping with the mechanical and electronic functions of sensor read-out and basic processing at 5-10 fps.  As excess processing cycles become available, I believe that the next great wave of DSLR development will be in areas of algorithmic processing, including multi-image capture for noise reduction and increased DR, among others.  Many of the gyrations that we currently go thru manually with a plethora of stand-alone applications and plug-ins, including for example PTGui, Photomatix, PK Sharpener, etc. could best be performed in-camera.

I predict that we will start seeing the first round of these in-camera enhancements beginning next year.

Bruce

[bernie west]

Just a thought... Would that involve twice the battery power, twice the heat and twice the processing and write time?

[marcmccalmont]

Brilliant!
Marc

[Ray]

Could it be possible to recover the full DR seen by the sensor? 

It might well be possible.  What one would like is to somehow be able to use ISO 100 to keep all the highlights, while at the same time using ISO 1600 to recover all the shadows.  But how can one have two ISO settings at once?  By having two separate amplifiers fed from the same sensor data, running in parallel.  Suppose that the sensor signal is sent to two separate processing paths, each path an amplifier and an ADC, with one amplifier set to ISO 100 and the other to ISO 1600.  The ISO 100 path keeps all the highlights but has noisy shadows; the ISO 1600 path loses the top four stops of highlights but has much better shadows.  Quantizing each, one can then combine the image data in a manner similar to HDR processing to yield an image with all 14 stops that the sensor is capable of recording.

[a href=\"index.php?act=findpost&pid=210895\"][{POST_SNAPBACK}][/a]

Emil,
This does seem a clever way of extracting the full DR that the sensor sees.

When I first saw the results from my own testing with the Canon 20D, comparing equal exposures at ISO 100 and ISO 1600, I wondered why it would not be possible to amplify correctly exposed ISO 100 image as though they were underexposed ISO 1600 images, in order to get those significantly cleaner shadows that are so obvious when comparing the same exposure at ISO 100.

I guess the answer was, the amplifiers, A/D converters and output stages would be too massive. Your idea would seem to split the job into more manageable portions.

[dwdallam]

I would expect in the coming years that "ISO" will be replaced with "Best Image Quality" or "Best Dynamic Range" settings, or the like. Both will produce acceptable images. Then, probably you will get no settings for ISO. Then cameras will just give you the best DR they have given the light available (because there will not be any or enough noise difference to warrant a change in ISO, and is there a reason to have an option to limit DR?).

[ejmartin]

I would expect in the coming years that "ISO" will be replaced with "Best Image Quality" or "Best Dynamic Range" settings, or the like. Both will produce acceptable images. Then, probably you will get no settings for ISO. Then cameras will just give you the best DR they have given the light available (because there will not be any or enough noise difference to warrant a change in ISO, and is there a reason to have an option to limit DR?).
[a href=\"index.php?act=findpost&pid=210934\"][{POST_SNAPBACK}][/a]

Actually, if the idea or something like it is feasible, it means the *end* of in-camera ISO.  One needs only two fixed amplifications to extract the full camera dynamic range, and then you have everything the sensor can collect; there is absolutely no reason at that point to have any sort of variable amplification in the camera.  "ISO" becomes a bit of metadata, like white balance -- a suggestion to the raw converter as to what EV compensation to apply when developing the image.

I believe this is why some MFDB's indeed don't have in-camera variable ISO -- they use components downstream of the sensor that capture the full sensor DR, and indeed the ISO the user sets in the camera is simply a metadata tag, with absolutely no effect on the raw data itself, only a change in its appearance on output, the same as would be obtained by applying an equivalent number of EV of exposure compensation.

[ejmartin]

Just a thought... Would that involve twice the battery power, twice the heat and twice the processing and write time?
[a href=\"index.php?act=findpost&pid=210905\"][{POST_SNAPBACK}][/a]

Somewhat more battery power, I don't think that heat is a concern.  The Nikon D3 uses six processors running in parallel, and battery life seems to be adequate.  And besides, the battery on my 1D3 seems to last forever; I'd be willing to give a bit of that up for image quality like the above    

As for processing and write time, the only additional time  needed, since there are two processors running in parallel, is the extra step of the HDR-style blend of the two outputs from the different amplifications; that is a very simple bit of code that should be rather quick to execute.  Much more processing goes on in generating the embedded jpeg from the raw data.  

As for write time, finally 14 bits would actually be justified for these cameras    and actually a bit more for the 1D3 and D3, so it's likely the output would be rounded up to 16-bit; so that's a 15% overhead in write times and storage footprint.  So use a bigger buffer.

[BJL]

Emil,

    thanks for this thread; it seems a better place to continue the discussion that we started at DPR.

It has struck me that Nikon might be making a first pass at this sort if idea approach in its 2.5fps 14-bit mode for the Sony EXMOR sensor in the D300. Given that the column ADUs of that sensor are only 12-bit, the only likely explanations I see are:
1) Sony wired that sensor for the option of outputting the analog signal to off-board ADUs, or
2) Nikon get the 4-bit output through a process involving multiple reads of the same column sense capacitor signal. Maybe jut averaging multiple A/D conversions, maybe applying different charge-to-voltage factors in several conversions.
The latter seems more likely. The very low frame rate of 2.5fps, especially compared to the sensor's stated maximum frame rate of over 10fps, hints at multiple successive reads in some form, since Nikon and Sony have long been capable of far better than 2.5fps and 30MP/sec with the traditional process of moving the analog signal to an off-board ADU (with the D200's 10MP CCD for example.)


The idea of ISO speed becoming simply a choice of exposure index and subsequent digital "brightness" adjustment, not interfering with optimal processing of the analog signal, is very appealing!

[bjanes]

As for write time, finally 14 bits would actually be justified for these cameras    and actually a bit more for the 1D3 and D3, so it's likely the output would be rounded up to 16-bit; so that's a 15% overhead in write times and storage footprint.  So use a bigger buffer.
[{POST_SNAPBACK}][/a]

Yes, it would seem as if a bit depth of 16 would be sufficient to record all the information captured by the sensor of most current cameras and enable the use of an ISO tag. As [a href=\"http://www.clarkvision.com/imagedetail/digital.sensor.performance.summary/#unity_gain]Roger Clark[/url] points out it does not make sense to digitize anything less than one electron (gain = 1, i.e. 1 ADU or DN = 1 electron). Most current sensors have a full well of less than 2^16 -1(65535) electrons. Some large sensor Canons have a larger full well, but are currently limited by noise in the electronics.

Bill

[ejmartin]

Emil,

    thanks for this thread; it seems a better place to continue the discussion that we started at DPR.

It has struck me that Nikon might be making a first pass at this sort if idea approach in its 2.5fps 14-bit mode for the Sony EXMOR sensor in the D300. Given that the column ADUs of that sensor are only 12-bit, the only likely explanations I see are:
1) Sony wired that sensor for the option of outputting the analog signal to off-board ADUs, or
2) Nikon get the 4-bit output through a process involving multiple reads of the same column sense capacitor signal. Maybe jut averaging multiple A/D conversions, maybe applying different charge-to-voltage factors in several conversions.
The latter seems more likely. The very low frame rate of 2.5fps, especially compared to the sensor's stated maximum frame rate of over 10fps, hints at multiple successive reads in some form, since Nikon and Sony have long been capable of far better than 2.5fps and 30MP/sec with the traditional process of moving the analog signal to an off-board ADU (with the D200's 10MP CCD for example.)

[a href=\"index.php?act=findpost&pid=211449\"][{POST_SNAPBACK}][/a]

Yes, I've seen option 2 suggested as an explanation of the frame-rate drop occurring with 14-bit mode on the D300.  As I mentioned in the DPR thread, if they're using multiple reads, it's likely at the same gain otherwise there is no need to do more than two. Multiple reads at the same gain is a rather inefficient way to beat down the read noise, you only lower the noise by the square root of the number of reads.  It's probably a decent way of suppressing the line noises though if they're using four reads for the 14-bit file. Using the inherently less noisy higher ISO read and combining with lower ISO is vastly more economical of resources I would think.

For the D300, my measurements indicate a gain of 1.8 e-/ADU at LO ISO (which is actually around ISO 150-160 when normalized properly to the other ISO's), a raw saturation level of 15300 ADU in the green channel (inferred from saturation of 3830 in 12-bit mode; I should check it explicitly for 14-bit), for an electron count at raw saturation of 27200 e-, and a read noise of 4.4 ADU=8 e-. Thus the DR at the lowest available ISO is 27200/8=3400=11.7 stops.  ISO 1600 read noise is 4.6 electrons.  So it would seem that the column ADC's for the D300 are already achieving a rather low read noise at base ISO, lower than what Canon is currently showing by a factor about 2; and the improvement to be had from the double read at two different gains would be correspondingly less, since the read noise drops by less than a factor of two; the measurements would indicate the sensor DR is about 12.5 stops at the pixel level.  That appears to be somewhat less than the Canons are capable of even after one properly scales by the ratio of pixel pitches to obtain DR per area which is the fair basis of comparison.  The saturation count of ~27K e- seems a bit low.

[ejmartin]

Yes, it would seem as if a bit depth of 16 would be sufficient to record all the information captured by the sensor of most current cameras and enable the use of an ISO tag. As Roger Clark points out it does not make sense to digitize anything less than one electron (gain = 1, i.e. 1 ADU or DN = 1 electron). Most current sensors have a full well of less than 2^16 -1(65535) electrons. Some large sensor Canons have a larger full well, but are currently limited by noise in the electronics.

Bill
[a href=\"index.php?act=findpost&pid=211488\"][{POST_SNAPBACK}][/a]

I have to say I've never understood the utility of unity gain in this regard.  First of all, if we take the definition literally, one should use 14 bit ADU's for 14 bit cameras, and then the unity gain ISO for the 1D3 is 400.  Roger continues to use 12-bit ADU's to maintain a consistent normalization between 12-bit and 14-bit cameras, but then that shows how arbitrary the definition is, doesn't it, since for 14-bit cameras his stated unity gain ISO is not where one ADU in the raw data equals one electron, it's where one ADU is 1/4 electron.

If we take the literal definition, the unity gain ISO for the 1D3 is ISO 400.  The read noise continues to drop from ISO 400 to ISO 1600, by almost a factor of two.  If I had used an ISO 400 image for my demonstration as the high ISO image, under the mistaken impression that it's not worthwhile going beyond the unity gain ISO, I would have thrown away a stop of DR and about two stops of range where the S/N ratio is shot-noise limited.  The point where one ADU equals one electron is rather meaningless in a situation where the system has several electrons of noise which washes out any correlation between the quantization of raw data and the quantization of electron number.  If read noise were under one electron, then there's something to discuss -- then the raw value is strongly correlated to electron number and unity gain carries some meaning.  But  current DSLR's are far from that point.

Unity gain should not be interpreted in this way; going to higher ISO confers benefits so long as the read noise in electrons is dropping, and the exposure constraints don't allow ETTR at lower ISO.

[BJL]

... if they're using multiple reads, it's likely at the same gain otherwise there is no need to do more than two. Multiple reads at the same gain is a rather inefficient way to beat down the read noise, you only lower the noise by the square root of the number of reads.
[a href=\"index.php?act=findpost&pid=211489\"][{POST_SNAPBACK}][/a]

Agreed: averaging multiple readings at equal gain might simply be the best available solution with a sensor chip whose hardware does not support your proposed approach.
But could Nikon instead be making two readings at different gain levels?

For the D300, my measurements indicate a gain of 1.8 e-/ADU at LO ISO ... a read noise of 4.4 ADU=8 e-. ...  ISO 1600 read noise is 4.6 electrons.  So it would seem that the column ADC's for the D300 are already achieving a rather low read noise at base ISO, lower than what Canon is currently showing by a factor about 2; and the improvement to be had from the double read at two different gains would be correspondingly less ...
[a href=\"index.php?act=findpost&pid=211489\"][{POST_SNAPBACK}][/a]

Dare I suggest that this might be an advantage of the Sony EXMOR sensor doing all processing in parallel at column bottom, and thus at far lower frequencies than the 1DsMkIII sensor has to use with its bottle-neck of all signals going through four (or is it eight) ADC's. The data (measured 1DsMkIII noise in ADU only slowly decreasing with reduced ISO speed while D300 noise in ADU decreases more with reduced ISO speed) fit the hypothesis that the 1DsMkIII gets a larger proportion of its noise after application of ISO gain, due to that post-gain processing being done at a far high frequency: Mhz rates vs KHz rates for the EXMOR sensor.

[Guillermo Luijk]

A plot of the signal-to-noise (vertical axis, in stops) as a function of absolute exposure (horizontal axis, in stops) reveals many of the issues:



Each stop increase in ISO pushes another stop of highlights past the range of the ADC which are then lost; in the figure this is shown by the graph of S/N for a given ISO ending one stop earlier for each successive ISO. At the shadow end, increasing the ISO expands the range at low exposure, with the amount gradually tapering off until the improvement between ISO 800 and 1600 is rather small.

Just to make sure I interpret this plot properly Emil: X-axis is exposure *before* the ISO amplification (straight from the sensor), and Y-axis is the SNR we will achieved in that area of the image for the different ISO values. Is that right?

Regarding your multiple ISO amplification approach I think it's a very clever idea.

BR

[ejmartin]

Just to make sure I interpret this plot properly Emil: X-axis is exposure *before* the ISO amplification. Is that right?

[a href=\"index.php?act=findpost&pid=211533\"][{POST_SNAPBACK}][/a]

The curves are plots of

S/N = S/sqrt[R(G)^2 + S]

where S is the signal in photo-electrons, and R(G) is the read noise in electrons at an ISO setting G. The thing that makes the expansion of the shadow end at higher ISO is the fact that the noise goes down dramatically between ISO 100 and 400 when referred to its equivalent in photo-electrons:



So at low signal S, the noise is dominated by the read noise R(G) for that ISO, and lower read noise gives higher S/N (as far as I'm concerned, this is the only reason to raise the ISO, unless your raw converter does a sloppy job of exposure compensation); eventually as the signal rises the second (photon shot noise) term in the sqrt is the more important one, R(G) becomes largely irrelevant and all the curves merge to the same thing.  At the upper end, each one stop increase in ISO means that it takes only half as many electrons to saturate the ADC, so the saturation point in signal is one stop earlier.

All the plots and the demo images are from the 1D3.  Other Canons and the D3 are similar.  The D300 has a bit less to gain from this technique due to its different sensor architecture, as discussed above.

[Guillermo Luijk]

This is a sample of real improvement ISO100 vs ISO1600 (same aperture/shutter) in the Canon 350D supporting this idea:




Emil, do you think the ADC could be shared by the two paths? no idea if this would save much of the cost, just wondering:

[ejmartin]

Emil, do you think the ADC could be shared by the two paths? no idea if this would save much of the cost, just wondering:


[a href=\"index.php?act=findpost&pid=211551\"][{POST_SNAPBACK}][/a]

Well, circuit design is not my specialty, but I suspect such a scheme would risk introducing more noise as there has to be a comparator that switches the input to the ADC between the two gain channels; further the switch is either on/off, whereas I think a smoother blend is achieved by a continuous crossover between the two channels within some window.  Also, doing the blend afterward allows for compensating relative gain that is not exactly 16 -- for instance I blended the two channels with a factor 15.86 in my example since that was the ratio of the two channels where both had high quality data.

[bjanes]

I have to say I've never understood the utility of unity gain in this regard.  First of all, if we take the definition literally, one should use 14 bit ADU's for 14 bit cameras, and then the unity gain ISO for the 1D3 is 400.  Roger continues to use 12-bit ADU's to maintain a consistent normalization between 12-bit and 14-bit cameras, but then that shows how arbitrary the definition is, doesn't it, since for 14-bit cameras his stated unity gain ISO is not where one ADU in the raw data equals one electron, it's where one ADU is 1/4 electron.

If we take the literal definition, the unity gain ISO for the 1D3 is ISO 400.  The read noise continues to drop from ISO 400 to ISO 1600, by almost a factor of two.  If I had used an ISO 400 image for my demonstration as the high ISO image, under the mistaken impression that it's not worthwhile going beyond the unity gain ISO, I would have thrown away a stop of DR and about two stops of range where the S/N ratio is shot-noise limited.  The point where one ADU equals one electron is rather meaningless in a situation where the system has several electrons of noise which washes out any correlation between the quantization of raw data and the quantization of electron number.  If read noise were under one electron, then there's something to discuss -- then the raw value is strongly correlated to electron number and unity gain carries some meaning.  But  current DSLR's are far from that point.

Unity gain should not be interpreted in this way; going to higher ISO confers benefits so long as the read noise in electrons is dropping, and the exposure constraints don't allow ETTR at lower ISO.
[a href=\"index.php?act=findpost&pid=211493\"][{POST_SNAPBACK}][/a]

I have to agree that Roger's terminology is rather confused and he should make it more coherent. By its literal definition unity gain is when one data number represents one electron and normalizing to 12 bit notation confuses the issue. The sensor can not count individual electrons, but if the full well of the D3 is 65,600 electrons, then one 16 bit data number would represent on average one electron under these conditions and it would make no sense to quantize at a higher bit level. In practice, one requires less bit depth because of noise, as you point out.

Bill

[MichaelEzra]

Emil, this is an excellent idea!

If anyone could write a firmware hack to read sensor information twice with different ISO gains from a single capture; 14 stops DR would be spectacular!

Do we have such a person on this forum?

[Cartman]

Maybe someday we'll see active cooling with Peltier coolers and fans on a modular camera as a method of lowering the noise floor.