Link between high ISO noise and DR?

[BernardLanguillier]

Dear all,

Long ago, a theory was often heard stating that high ISO noise and low ISO DR were 2 faces of the same coin, meaning very tightly related. This would have been related to the size of the photosites, large photosites being able to capture more photons.

The additional photons would enable the reading of a meaningful signal when illumination level is low (meaning when high ISO is used), and would also saturate less quickly resulting in less blown highlights, and therefore more DR.

The MFDB have never met these predictions by far, and now the Sony A900 with its measured DR of 12.6 stop with only average high ISO noise is also clearly not playing by this theory.

What gives?

Cheers,
Bernard

[Ray]

The MFDB have never met these predictions by far, and now the Sony A900 with its measured DR of 12.6 stop with only average high ISO noise is also clearly not playing by this theory.

What gives?

Cheers,
Bernard

What I find significant, Bernard, is that dpreview's figure of 12.6 stops for the A900 is a 'best case' scenario which involves extreme adjustments to RAW images in ACR, such as -2.65 EV and a -50 contrast.

I can find no comparable ACR adjustments in relation to other similar cameras that the A900 is compared with. What we have for other cameras are DR figures in relation to the default ACR settings and in relation to 'auto' ACR settings. The figure of 12.6 stops I therefore find misleading. The ACR default settings provide the worst DR figure; the ACR 'auto' settings provide a significantly better figure, and the best DR outcome results from the extreme settings mentioned above which have not been applied to any other camera in the review.

The only DR comparisons I can find in the review relate to in-camera jpegs, and those are quite surprising. Whilst the A900 excels in jpeg mode, having 1.2 stops greater DR than the 5D and 0.8 stops greater DR than the 1Ds3, the 5D is shown as having 0.4 stops greater DR than the D700.

One has to be clear when one is comparing apples with apples. The 12.6 figure for the A900 appears to be an orange.

[jani]

I can find no comparable ACR adjustments in relation to other similar cameras that the A900 is compared with. What we have for other cameras are DR figures in relation to the default ACR settings and in relation to 'auto' ACR settings. The figure of 12.6 stops I therefore find misleading. The ACR default settings provide the worst DR figure; the ACR 'auto' settings provide a significantly better figure, and the best DR outcome results from the extreme settings mentioned above which have not been applied to any other camera in the review.

As usual the default Adobe Camera RAW conversion delivers less dynamic range than JPEG from the camera (the same contrasty tone curve and very little noise reduction in shadows). Simply switching to 'Auto' in the ACR conversion dialog reaps huge rewards, increasing the dynamic range to around 10.5 stops. The very best we could get out of a raw file manually was around 11.3 EV, which is pretty good (though not quite up to the standard set by the Nikon D3 or the Fujifilm S5 Pro).

[Guillermo Luijk]

What I find significant, Bernard, is that dpreview's figure of 12.6 stops for the A900 is a 'best case' scenario which involves extreme adjustments to RAW images in ACR, such as -2.65 EV and a -50 contrast.

I can find no comparable ACR adjustments in relation to other similar cameras that the A900 is compared with. What we have for other cameras are DR figures in relation to the default ACR settings and in relation to 'auto' ACR settings. The figure of 12.6 stops I therefore find misleading. The ACR default settings provide the worst DR figure; the ACR 'auto' settings provide a significantly better figure, and the best DR outcome results from the extreme settings mentioned above which have not been applied to any other camera in the review.

In my opinion, any DR test in which the results depend on the RAW developer tool used, and the way to use it (settings), is nonsense.
The ability to capture a certain DR by a camera is a _hardware feature_, and it is clearly hardware limited by: saturation in the highlights, and a given (and there is not a unique criteria for this, but some criteria must be chosen if DR results are to be compared) value for the minimum signal to noise ratio considered valid in the shadows. And all this can (and perhaps should) be measured for each of the individual RGB channels separately.

So the dynamic range of a camera would be the difference in EV between the maximum recordable value in its RAW file, and the value of signal that keeps a given ratio against the std deviation of the noise. Any test involving a particular RAW developer and/or particular settings on a given RAW developer such as ACR for me is crap.

Problems may arise to measure this if one day manufacturers start to do destructive noise reduction on the RAW file, meaning by destructive any processing that reduces noise but also destroys texture and captured detail (that would be the day when RAW is not RAW anymore). In that case, the camera would achieve a better measured DR but in a somewhat unfair way and DR comparisions should be attached to captured detail tests.

And another very important issue that has to be taken into account in DR is the sensor resolution: DR is measured at a pixel level, by calculating SNR on each pixel. If a camera has more Mpx than another one with the same pixel SNR, the highest resolution camera will provide a higher overall image DR (not pixel DR), since SNR will increase when rescaling the image to match the size of the lower resolution sensor.

In general: DR is a hardware feature, and it can never depend on the software used to process the RAW file.
The maximum recordable DR decreases the higher the ISO set. Maximum recordable DR is reached when exposure of the scene is maximum (i.e. ETTR) and the lowest electronic camera ISO is used.

O course I am always talking about shooting in RAW. JPEG is another story which depends on camera's software, and not always the highest DR camera will provide the highest visible DR JPEG files.

BR

[Tony Beach]

According to DPR:  "Shadow range is more complicated, in our test we stop measuring values below middle gray as soon as the luminance value drops below our defined 'black point' (about 2% luminance) or the signal-to-noise ratio drops below a predefined value (where shadow detail would be swamped by noise), whichever comes first."

In other words, DPR stops measuring DR at 2% regardless of the SNR, and it is notable that in DPR's "best" settings using ACR that the graph ends abruptly at bottom (shadow) end of the spectrum.  Probably DPR could have continued measuring DR using their method well past 13 stops by simply continuing to halve the values (1%, .5%, .25%, etc) until they actually reached an unacceptable SNR; after all, they were still recording hundreds of photons when they stopped measuring any further.  To illustrate, you can see from these DPR graphs of:

the A900


from:  http://www.dpreview.com/reviews/sonydslra900/page24.asp

and the D3


from:  http://www.dpreview.com/reviews/NikonD3/page20.asp

that the bottom of the measurement has been cut off.   Measuring DR using a calibrated Stouffer Step Wedge is also problematic as it merely measures gray, which is not the same as usable DR.  So much for the DR half of the equation, without which, it is impossible to draw any direct parallels between DR and performance at higher ISOs.

[Ray]

QUOTE (DPReview (1Ds MkIII), page 21 @ August 2008)
As usual the default Adobe Camera RAW conversion delivers less dynamic range than JPEG from the camera (the same contrasty tone curve and very little noise reduction in shadows). Simply switching to 'Auto' in the ACR conversion dialog reaps huge rewards, increasing the dynamic range to around 10.5 stops. The very best we could get out of a raw file manually was around 11.3 EV, which is pretty good (though not quite up to the standard set by the Nikon D3 or the Fujifilm S5 Pro).

Jan,
I'm not sure how valid it is to compare test results from the past with current test results. Doesn't dpreview warn against doing this because of slight changes in procedure and methodology?

However, not to nit pick, I would accept that the DR of the A900 looks as though it's the best of any current model of DSLR, at base ISO. Those who complain about increasing pixel count robbing them of increases in DR should be pleased with the performance of the A900.

[Ray]

In general: DR is a hardware feature, and it can never depend on the software used to process the RAW file.

Point taken, Guillermo, however we all have to use some type of converter if we shoot in RAW mode. Most people probably use ACR or Lightroom for a variety of reasons and are unlikely to change converters because of some very slight, pixel-peeping advantage that another converter may have in respect of one particular model of camera.

Whenever I've compared ACR with another converter, whether DPP, C1, Breezebrowser, RSP etc, I've found ACR to be either better or very close with regard to recovery of highlight and shadow detail. DCRAW is probably the exception but is not easy to use, is it?

[Guillermo Luijk]

Point taken, Guillermo, however we all have to use some type of converter if we shoot in RAW mode. Most people probably use ACR or Lightroom for a variety of reasons and are unlikely to change converters because of some very slight, pixel-peeping advantage that another converter may have in respect of one particular model of camera.

If ACR or any other software does not manage to exploit all the information contained in a given RAW file, it is not fair to show a comparision table saying 'Camera X has a dynamic range of Y' using that software. Dpreview should clearly head their tables with 'If you use ACR and you set these adjustments, you will be able to enjoy with Camera X a dynamic range of Y', and they don't do it. It is not a limitation of the camera, but of the software and the ability of the Dpreview team with the particular settings.

I will give you an example: ACR can obtain more dynamic range in the highlights from Fuji RAF files than the Fuji software. Is that because the Fuji guys are stupid or the ACR guys are very clever? no way. Studying with some depth a RAF file is easy to see that the Fuji software voluntarily discards some highlight information to avoid magenta cast problems that arise when heavily reducing exposure in the Fuji R sensor in strongly exposed shots. ACR allows for this stronger exposure correction, and therefore can suffer from the magenta cast.
If I read a comparision about cameras (not about RAW developers), I want to know what _my Fuji_ can capture, and later will decide which tool to use to fit my needs. But I don't want to see a comparision table subject to a particular software since: this software may not be the one I will use, and/or this software may change tomorrow.

BR

[Ray]

If ACR or any other software does not manage to exploit all the information contained in a given RAW file, it is not fair to show a comparision table saying 'Camera X has a dynamic range of Y' using that software. Dpreview should clearly head their tables with 'If you use ACR and you set these adjustments, you will be able to enjoy with Camera X a dynamic range of Y', and they don't do it. It is not a limitation of the camera, but of the software and the ability of the Dpreview team with the particular settings.

To be fair to Dpreview, that's more or less what they do. They produce DR comparisons based on the camera's jpeg output, then they demonstrate what ACR can do with various settings such as 'default', 'auto', and maximum retrieval of highlight and shadow detail after extreme adjustments of EV and contrast.

They also mention that recovery of grayscale detail in wedges is not the same as recovery of full color detail. They haven't extended their testing to include comparisons of ACR's ability to reconstruct the color detail within those extreme DR ranges and how it might vary with different camera models. But there is no doubt that their DR figures of RAW images are a systemic part of the ACR software, just as many lens tests, such as those at Photozone, are tests of both lens and sensor used.

I don't get any sense that Dpreview is trying to hide this fact, nor can I see how another method of determining DR without the use of a specific RAW converter would help the consumer.

[jani]

Jan,
I'm not sure how valid it is to compare test results from the past with current test results. Doesn't dpreview warn against doing this because of slight changes in procedure and methodology?

I was responding to your complaint that they didn't perform similar testing in the past with (somewhat weak) evidence that they did.

But you can't have it both ways.

However, not to nit pick, I would accept that the DR of the A900 looks as though it's the best of any current model of DSLR, at base ISO. Those who complain about increasing pixel count robbing them of increases in DR should be pleased with the performance of the A900.

I don't feel so sure about that.

The methodology seems flawed in various respects, or there is a lack of disclosure regarding the methodology.

I'm with Guillermo on this one.

But there is no doubt that their DR figures of RAW images are a systemic part of the ACR software, just as many lens tests, such as those at Photozone, are tests of both lens and sensor used.

Yes, at least it's fairly consistent, but as you yourself say: you can't compare former tests with current tests.

One of the reasons is of course that ACR may change from version to version...

[Guillermo Luijk]

According to DPR:  "Shadow range is more complicated, in our test we stop measuring values below middle gray as soon as the luminance value drops below our defined 'black point' (about 2% luminance) or the signal-to-noise ratio drops below a predefined value (where shadow detail would be swamped by noise), whichever comes first."

In other words, DPR stops measuring DR at 2% regardless of the SNR, and it is notable that in DPR's "best" settings using ACR that the graph ends abruptly at bottom (shadow) end of the spectrum.  Probably DPR could have continued measuring DR using their method well past 13 stops by simply continuing to halve the values (1%, .5%, .25%, etc) until they actually reached an unacceptable SNR; after all, they were still recording hundreds of photons when they stopped measuring any further.

This shows how robust and convincing is Dpreview's criteria: they consider the low end of the DR when signal drops below a value, a signal that THEY set by chosing a particular tone curve into ACR through the chosen settings, even if there could be a lot of valid information (good enough SNR) below that point that could perfectly be of use to the user.

And even if they chose the SNR criteria, I wonder how they can measure noise in ACR if there is no guarantee whether ACR is applying or not any noise reduction since that software is a black box from the user's point of view.

Doing always the same wrong procedure, can never be considered a valid procedure.

BR

[bjanes]

In my opinion, any DR test in which the results depend on the RAW developer tool used, and the way to use it (settings), is nonsense.
The ability to capture a certain DR by a camera is a _hardware feature_, and it is clearly hardware limited by: saturation in the highlights, and a given (and there is not a unique criteria for this, but some criteria must be chosen if DR results are to be compared) value for the minimum signal to noise ratio considered valid in the shadows. And all this can (and perhaps should) be measured for each of the individual RGB channels separately.

So the dynamic range of a camera would be the difference in EV between the maximum recordable value in its RAW file, and the value of signal that keeps a given ratio against the std deviation of the noise. Any test involving a particular RAW developer and/or particular settings on a given RAW developer such as ACR for me is crap.

I agree entirely with Guillermo. Achieving greater dynamic range through highlight recovery as done by DPReview involves making up data for the clipped channels, blue and red for daylight exposures with most Bayer array cameras. Shown below is a typical daylight exposure with the Nikon D3 along with a histogram of the raw data. The shot is underexposed, since the green channel is about 2/3 stop from clipping. The red channel is 1 1/3 stops from clipping, and the blue channel is one stop from clipping. With increasing exposure such that the green channel becomes clipped, the other 2 channels would initially still have data and highlight recovery would be possible using data from the non-clipped channels. When the red channel becomes clipped, highlight recovery is no longer possible.

The green channel will have the highest dynamic range, since it receives the most exposure. If you determine DR from a demosaiced image you are combining three different DRs, but what weighting should be used? Probably not an arithemetic average, since the eye is most sensitive to green. Also, in a demosaiced image, the green is contaminated by interpolation from adjacent blue and red pixels. For an excelllent technical article on these matters, see Emil Martinec.
 
[attachment=9148:BirdsHistogram.jpg]

And another very important issue that has to be taken into account in DR is the sensor resolution: DR is measured at a pixel level, by calculating SNR on each pixel. If a camera has more Mpx than another one with the same pixel SNR, the highest resolution camera will provide a higher overall image DR (not pixel DR), since SNR will increase when rescaling the image to match the size of the lower resolution sensor.

In general: DR is a hardware feature, and it can never depend on the software used to process the RAW file.
The maximum recordable DR decreases the higher the ISO set. Maximum recordable DR is reached when exposure of the scene is maximum (i.e. ETTR) and the lowest electronic camera ISO is used.

If the sensor size is held constant, and the pixel density varied to give different resolutions, the total DR will not change much, since the amount of light collected remains the same for both sensors. As Emil and Guillermo explain, one is trading off dynamic range for resolution. When the image of the smaller pixel camera (with the same total sensor area) is downsized so as to have the same resolution as the larger pixel camera, noise will be decreased by pixel binning. The per pixel noise standard deviation as used by DPReview in determining dynamic range can be misleading.

Of course, the reason for increasing megapixels is to allow a larger print size with acceptable image detail. When the print size with the higher megapixel camera is increased so that pixel binning is no longer done, the higher megapixel camera would cease to have a DR advantage, but it would still have better image detail and most likely finer gained noise, which might appear less objectionable to an observer.

Of course I am always talking about shooting in RAW. JPEG is another story which depends on camera's software, and not always the highest DR camera will provide the highest visible DR JPEG files.

Quite true. The raw file has linear scene referred data, whereas the rendered JPEG is object referred. The rendering process involves DR compression and application of a tone curve--see the white paper by Karl Lange on the Adobe site. Highlight and shadow DR as discussed by DPReview may make sense for JPEG rendered images which have a shoulder, linear segment, and knee, but this concept makes no sense for a raw file which is linear.

Bill

[Guillermo Luijk]

When the image of the smaller pixel camera (with the same total sensor area) is downsized so as to have the same resolution as the larger pixel camera, noise will be decreased by pixel binning.

In fact a pure binning strategy by averaging 2x2 pixels into 1 final pixel, which according to Emil's statistics doubles the SNR, is the worst improvement we can expect. When reescaling to 50% the original size with other algorithms like Bicubic, I have done tests on noisy images and the SNR improvement is even higher (the noise histogram gets thinner than with a simple binning averaging). And of course the quality of resizing is also better.

BR

[bjanes]

In fact a pure binning strategy by averaging 2x2 pixels into 1 final pixel, which according to Emil's statistics doubles the SNR, is the worst improvement we can expect. When reescaling to 50% the original size with other algorithms like Bicubic, I have done tests on noisy images and the SNR improvement is even higher (the noise histogram gets thinner than with a simple binning averaging). And of course the quality of resizing is also better.

BR

Yes, the whole matter becomes rather complex. Another thing to consider is that pixel binning is best done in hardware, not software, as explained in this article on the Photometrics site. That article discusses 2 by 2 pixel blending. In that case, 4 pixels are binned into one in hardware. The binned superpixel can be read with the same read noise as a single pixel. However, if binning is done after the fact in software, four separate and discrete read noises are involved.

As far as I know, hardware pixel binning is currently done only in scientific grade CCDs. The most advanced 35 mm style digital cameras now use CMOS, but MF digitals still use CCDs. It would be interesting to have a digital 32 MP camera with 2:1 and 4:1 hardware pixel binning.

Bill

[ejmartin]

DPR's tests are often a source of amusement.  Apparently they don't understand that it is impossible for a linear medium like RAW to encode more stops of DR than there are bits in the encoding (12 for the A900).    

One has to be clear when one is comparing apples with apples. The 12.6 figure for the A900 appears to be an orange.

Or a lemon  

[Ray]

I agree entirely with Guillermo. Achieving greater dynamic range through highlight recovery as done by DPReview involves making up data for the clipped channels, blue and red for daylight exposures with most Bayer array cameras.

Bill,
ACR is what I use to process my RAW images, and I believe it's what most photographers use. What Dpreview does is what I do, and many others, except I try to avoid pushing exposure to the exteme right because I'm aware that ACR's attempt to reconstruct color data is sometimes unsatisfactory. An obvious example is the shift towards green in skies where the blue channel is clipped.

I don't see that knowing the DR of the sensor helps the photographer since the RAW image is useless without a converter. However, I can see that those who are in the process of developing their own RAW converter would find such information useful. Also, such information may be of academic interest to some, even though of no immediate practical benefit.

For the purpose of comparing the DR of two different cameras, the main criterion should be that the same procedure is applied to both cameras using the same converter. Dpreview applied 3 different groups of settings in ACR (default, auto and extreme) and mentioned the fact that reconstruction of color data would not necessarily be accurate with the extreme settings of -2.65 EV and -50 contrast. What they didn't do was use the same RAW converter and same settings with the other cameras in the review. All we got for the other cameras were jpeg comparisons. I would criticise them for that.

[ejmartin]

Bill,
ACR is what I use to process my RAW images, and I believe it's what most photographers use. What Dpreview does is what I do, and many others, except I try to avoid pushing exposure to the exteme right because I'm aware that ACR's attempt to reconstruct color data is sometimes unsatisfactory. An obvious example is the shift towards green in skies where the blue channel is clipped.

I don't see that knowing the DR of the sensor helps the photographer since the RAW image is useless without a converter. However, I can see that those who are in the process of developing their own RAW converter would find such information useful. Also, such information may be of academic interest to some, even though of no immediate practical benefit.

For the purpose of comparing the DR of two different cameras, the main criterion should be that the same procedure is applied to both cameras using the same converter. Dpreview applied 3 different groups of settings in ACR (default, auto and extreme) and mentioned the fact that reconstruction of color data would not necessarily be accurate with the extreme settings of -2.65 EV and -50 contrast. What they didn't do was use the same RAW converter and same settings with the other cameras in the review. All we got for the other cameras were jpeg comparisons. I would criticise them for that.

It is important to be clear when providing test results, to understand what is being tested.  If a purported test of dynamic range is contaminated by the raw converter's highlight recovery algorithm, it is not testing dynamic range; it is bringing in the native white balance of the sensor (which determines when different color channels clip) and a host of other factors.  If DPR doesn't know what is going on under the hood in the raw converter, they don't know what combination of camera properties they are measuring.

[ejmartin]

Dear all,

Long ago, a theory was often heard stating that high ISO noise and low ISO DR were 2 faces of the same coin, meaning very tightly related. This would have been related to the size of the photosites, large photosites being able to capture more photons.

The additional photons would enable the reading of a meaningful signal when illumination level is low (meaning when high ISO is used), and would also saturate less quickly resulting in less blown highlights, and therefore more DR.

The MFDB have never met these predictions by far, and now the Sony A900 with its measured DR of 12.6 stop with only average high ISO noise is also clearly not playing by this theory.

What gives?

Cheers,
Bernard

High ISO noise and low ISO DR need not be closely related.  The noise floor at low ISO in CMOS DSLR's is controlled by the noise in the ISO amplifier and A/D converter, while the noise floor at high ISO is controlled by the noise of the sensor array.  Different components, different noise characteristics, and the one that effects high ISO midtone and shadow noise doesn't enter into the determination of low ISO DR.  

For CCD sensor cameras, there tends to be a tighter relationship; data I've seen shows a simple relationship between low and high ISO electronic noise, and so the statement of a coupling between high ISO noise and low ISO DR is more accurate.

While I have singled out electronic read noise for the purpose of illustrating a counterexample, photon noise is independent of the sensor type, and you are right in that the more photons one captures, the less apparent photon noise becomes.  Dynamic range can be increased either by capturing more photons, or by lowering electronic read noise.  The read noise can be decouple between low and high ISO, but the photon capture halves for each doubling of ISO (more precisely the number of photons captured at the saturation point of the raw data is halved), and so that aspect of DR is directly correlated between high and low ISO.

When considering DR, it is important to realize that it is scale dependent; larger photosites tend to have more DR by capturing more photons, but also eat up more sensor real estate to capture those photons.  Photosites of half the pixel pitch (1/4 the area) with the same efficiency per unit area, will have 1/4 the photon capacity per photosite, but when binned will still capture the same number of photons over the whole sensor (while delivering twice the resolution in the bargain).  The proper figure of merit is DR per unit area of sensor (or better, DR of an area of sensor that is a fixed percentage of the area of the sensor).  When this is done, there is a mild advantage to larger photosites, but not as much as the comparison of DR per photosite would mislead one to believe.

[Tony Beach]

For the purpose of comparing the DR of two different cameras, the main criterion should be that the same procedure is applied to both cameras using the same converter.

Nonsense like this is why I ignore your posts, but since Emil responded then I see you have thrown more garbage into the discussion that should be responded to.

As I see it, the issue is that if you really care about increasingly negligible differences in image quality between any two cameras, then you should also care about the entire workflow that results in the final image.  What you are arguing is analogous to shopping for a car that has maximum horsepower, but then disregarding the owner's manual and putting regular gas instead of premium into the tank.

[Guillermo Luijk]

the main criterion should be that the same procedure is applied to both cameras using the same converter.

Dpreview applied 3 different groups of settings in ACR (default, auto and extreme) and mentioned the fact that reconstruction of color data would not necessarily be accurate with the extreme settings of -2.65 EV and -50 contrast. What they didn't do was use the same RAW converter and same settings with the other cameras in the review. All we got for the other cameras were jpeg comparisons. I would criticise them for that.

Ray, DPreview will never be able to compare cameras using the same tool and parameters unless they repeat all the tests for the old cameras with the last version software everytime a new camera appears in the market, and we know they won't do that. This is a flaw in the method, and you agree.

But there is more, there are also flaws in the concept of what they are doing, such as using highlight recovery to decide the right end of the DR, or setting the low end of the DR according to a threshold signal level, specially when they set ACR to apply a tone curve that will affect this threshold and is out of their control.

I am not an academic expert, what I know is from reading some informed sites and practicing with DCRAW in front of my PC, and I could quickly set up a standard procedure much more rigurous than Dpreview's to yield real and practical DR figures for any camera, now and tomorrow. If they don't do this is simply because DPreview is a end-user oriented site, which is not a bad thing on its own. The problem comes when the way they focus on the end-user to provide him with information, is an incorrect way.