Real-World DR Testing

[Jonathan Wienke]

At this point one needs to think about the sense of the excercise.

Do you own a camera and want to measure it's DR? What for? It's a bit too late, isn't it?

Do you want to compare different cameras? If you do it for yourself (i.e. if you can carry out the test with different cameras before making the decision), then it is a useful test.

However, if you are doing this for a "public" review/test/comparison, then it is pretty much useless; what is the sense in doing that with a particular raw converter?

I can think of several practical reasons, not the least of which would be comparing various RAW converters to see which ones perform the best retaining shadow and highlight detail. For me, knowing which of my cameras performs better at a given ISO would be useful. So would knowing how much DR one is giving up by using camera JPEGs vs shooting RAW.

And IMO, a reviewer such as Phil Askey doing such my DR test in a standardized manner would add value to his reviews.

ACR handles highlight particularly badly. Not only, that it's clipping indication is worthless regarding the truth about the exposure (that is the case with all raw converters I know), but the support of some cameras is totally off in this regard.

I'm a bit surprised by this statement; with Canon DSLRs at least, ACR can extract nearly a stop more usable highlight detail than the other RAW converters I tried. The last time I compared was a couple years ago though.

The most objective method what I saw is the statistical analysis of the data. The pixels of an evenly lit uniform surface are supposed to be close to identical; the deviation indicates the noise (above some limit, but that's irrelevant here).

The problem with that reasoning is that doing standard statistical analyses of the RAW data is a very poor predictor of photographically useful DR and the negative visual impact of noise characteristics on an image. One camera may have a better S/N ratio than another, but if its noise contains structures such as horizontal banding, it may be more visually distracting than another camera's noise pattern that, while mathematically greater, is more similar to film grain and less objectionable overall.

[Panopeeper]

For me, knowing which of my cameras performs better at a given ISO would be useful. So would knowing how much DR one is giving up by using camera JPEGs vs shooting RAW

Fair enough. I do that differently, but I acknowledge, that maniac pixel peeping is not everyone's favourite pastime.

a reviewer such as Phil Askey doing such my DR test in a standardized manner would add value to his reviews

DPReview measurement is based on a step wedge, with de-mosaiced data. I favour not de-mosaiced data, but DPReview's method is consistent among cameras; as such, it is useful. Plus, as Alex posted it on another thread, he does not care, what his camera could produce, all he cares for is, what the supplied software is making out of it. This is a valid position for probably the vast majority of photographers (although I am surprized, that the most professionals are those, who care the least for such issues).

I'm a bit surprised by this statement; with Canon DSLRs at least, ACR can extract nearly a stop more usable highlight detail than the other RAW converters I tried

I think it is a general rule, that the manufacturer's own software can make the most out of the image (Phase one may be an exception). DPP knows the Canon cameras' characteristics much better, than ACR does (I still favour ACR because of its features).

It is very surprising and disappointing, that Adobe does not invest a bit more investigation into such cameras, which make out the majority of the DSLR market.

There are two issues (the two extremes) I have with ACR.

1. The noise chacateristics of Canon cameras is not analyzed good enough. Canon cameras deliver many thousands of masked pixels for the evaluation of the black level, and ACR does not make the best out of that.

2. The clipping point of the pixels is an important issue. I found with several cameras, that ACR mistreats the image by assuming incorrect clipping point (already the fact, that all pixels are assumed to have identical clipping point is a conceptual error.

The most noticable error, what I found occurs with the Canon 40D. ACR assumes clipping at 13600, no matter what ISO. The actual clipping points are, depending on the ISO, between 12740 and 16383.

The problem with that reasoning is that doing standard statistical analyses of the RAW data is a very poor predictor of photographically useful DR and the negative visual impact of noise characteristics on an image

This is true for every standardized method. There is no way of properly judging a camera w/o looking at samples of different situations. For example much more noise on cloth is acceptable than on a smooth surface.

[Tim Gray]

Do you own a camera and want to measure it's DR? What for? It's a bit too late, isn't it?

Inquiring minds want to know

[Jonathan Wienke]

I think it is a general rule, that the manufacturer's own software can make the most out of the image (Phase one may be an exception). DPP knows the Canon cameras' characteristics much better, than ACR does (I still favour ACR because of its features).

That's what the manufacturers claim, but IMO that claim is mostly myth and hocus-pocus. ACR (after calibration with the Fors script) has always given me better color than any other RAW converter, including DPP, the Canon RAW converter that preceded DPP, and Olympus' RAW converter. Regarding noise removal and sharpening, I have my own methods that work better than what's built in to any RAW converter I've tried so far, and getting sharp, detailed images with reasonably low noise levels has never been a problem for me.

There are two issues (the two extremes) I have with ACR.

1. The noise chacateristics of Canon cameras is not analyzed good enough. Canon cameras deliver many thousands of masked pixels for the evaluation of the black level, and ACR does not make the best out of that.

2. The clipping point of the pixels is an important issue. I found with several cameras, that ACR mistreats the image by assuming incorrect clipping point (already the fact, that all pixels are assumed to have identical clipping point is a conceptual error.

The most noticable error, what I found occurs with the Canon 40D. ACR assumes clipping at 13600, no matter what ISO. The actual clipping points are, depending on the ISO, between 12740 and 16383.

And that is about 1/3 of a stop at most. If that's the worst-case scenario I wouldn't spend too much time worrying about it. It certainly isn't in the same league as some of the MFDB software that claims RAWs 3 stops under clipping are exposed to the right.

[bjanes]

I expect that some RAW converters will do much better than others. ACR handles highlights particularly well; Capture One and some of the MFDB manufacturer's programs seem to toss out 2-3 stops of highlights, a thing I find curious given that increased DR is one of the big advantages of MFDBs.
[{POST_SNAPBACK}][/a]

No raw converter can invent details that are not present in the image, but they can reconstruct blown channels if there is at least one channel with detail. Because of white balance, the red and blue channels have a multiplier to equalize them with the green channel, which is usually the most sensitive.

For example, the multipliers for some Nikon cameras are given on the [a href=\"http://www.pochtar.com/NikonWhiteBalanceCoeffs.htm]Borg Web site [/url]. For daylight with the D2x the red and green multipliers are 1.52 and 1.69 respectively. If you overexpose by 0.5 EV for the green channels, the red and blue channels will not be clipped and the green channel can be reconstructed to some extent. However, I would not consider this to be true dynamic range. For this reason, DR is usually measured in the greens. DR in the red and blue channel would be less, because these channels are not fully exposed to the right. One can get better overall DR by placing a magenta filter over the lens to hold back some of the green light and equalize the channels. The multipliers are not that different for Canon sensors.

Consideration of these principles show why highlight recovery is usually possible out to no more than 1 stop overexposure.

I find it difficult to believe that any camera would throw away 2-3 stops of data.

Bill

[Jonathan Wienke]

No raw converter can invent details that are not present in the image, but they can reconstruct blown channels if there is at least one channel with detail. Because of white balance, the red and blue channels have a multiplier to equalize them with the green channel, which is usually the most sensitive.

For example, the multipliers for some Nikon cameras are given on the Borg Web site . For daylight with the D2x the red and green multipliers are 1.52 and 1.69 respectively. If you overexpose by 0.5 EV for the green channels, the red and blue channels will not be clipped and the green channel can be reconstructed to some extent. However, I would not consider this to be true dynamic range.

The color balance issue is why my chart has the color quadrants as well as the gray quadrant, so that clipping in a single color channel is more likely to be noticeable than in a neutral gray. In incandescent lighting, the red channel is usually the first to blow since its color temp is so low.

[Panopeeper]

That's what the manufacturers claim, but IMO that claim is mostly myth and hocus-pocus

The manufacturers are suckers in software, but they have the required data.

There is a clear concensus among Canon users, that DPP gives the best colors. The fact, that ACR *needs* to be customized shows its weakness, but the fact, that it *can* be customized shows its strength.

Now DPP too can be customized via Picture Styles.

that is about 1/3 of a stop at most. If that's the worst-case scenario I wouldn't spend too much time worrying about it

Tell it to someone, who just exposed to the right.

Re Phase One and C1: luckily it is not the accepted measurement of raw processing. Btw, ACR too has a big conceptual error: applying the WB can lead to RGB clipping, even though there was no raw clipping.

[Jonathan Wienke]

There is a clear concensus among Canon users, that DPP gives the best colors. The fact, that ACR *needs* to be customized shows its weakness, but the fact, that it *can* be customized shows its strength.

Unit variation can also account for calibration differences it's not necessarily a design flaw in ACR.

Tell it to someone, who just exposed to the right.

1/3-stop maximum non-optimality is much better than a software that tells someone that a 2-stop underexposure is exposed to the right or clipped.

Btw, ACR too has a big conceptual error: applying the WB can lead to RGB clipping, even though there was no raw clipping.

That can be solved by using a large-gamut color space like ProPhoto. It's not a problem with ACR, it's PEBKAC.

[Panopeeper]

That can be solved by using a large-gamut color space like ProPhoto. It's not a problem with ACR, it's PEBKAC.

I'm afraid you don't understand the problem. It is not a color issue; the role of WB hier is not, that it creates undisplayable colors, but that it "multiplies out" a channel. If you apply that WB  and reduce the exposure, then you get back what you "lost" before.

Addendum: I realize, that applying the WB can lead to undisplayable colors, but that's not all; the problem exists in ppRGB as well.

[John Sheehy]

The color balance issue is why my chart has the color quadrants as well as the gray quadrant, so that clipping in a single color channel is more likely to be noticeable than in a neutral gray. In incandescent lighting, the red channel is usually the first to blow since its color temp is so low.
[a href=\"index.php?act=findpost&pid=160743\"][{POST_SNAPBACK}][/a]

Green and red are pretty even, actually, in the RAW data, in incandescent light.  Only the blue is usually compromised.  If the light source is a heat lamp, like the kind they use to keep fried chicken hot, then red might take a big lead.

[Jonathan Wienke]

I'm afraid you don't understand the problem. It is not a color issue; the role of WB hier is not, that it creates undisplayable colors, but that it "multiplies out" a channel. If you apply that WB  and reduce the exposure, then you get back what you "lost" before.

I'm afraid you don't understand how white balancing works. You have to multiply the channel values to set WB. It makes no difference mathematically whether you use RGB multipliers <1 and increase the exposure setting, or use RGB multipliers >1 and reduce the exposure setting; the net result is exactly the same in both cases. If you can unclip the data by setting exposure to a lower value, then you haven't "lost" anything, and claiming ACR's WB has a "conceptual error" is simply wrong.

[Jonathan Wienke]

Green and red are pretty even, actually, in the RAW data, in incandescent light.  Only the blue is usually compromised.

Depends on the camera. With the 1D-MkII, the red channel will clip about 1/2 stop before green shooting in incandescent. With the 1Ds, they're fairly even. YMMV.

[Graeme Nattress]

Yes, any clipping from WB can be reduced or eliminated by reducing exposure. And as noted above, unclipped channels in the raw can be used to produce a pretty convincing facsimile of the clipped one too, for extra semi-faked DR.

However, WB is not just a simple multiplier on R, G and B, but is usually performed by a 3x3 matrix, using a complex mingling of R, G and B.

Graeme

[bjanes]

I'm afraid you don't understand how white balancing works. You have to multiply the channel values to set WB. It makes no difference mathematically whether you use RGB multipliers <1 and increase the exposure setting, or use RGB multipliers >1 and reduce the exposure setting; the net result is exactly the same in both cases. If you can unclip the data by setting exposure to a lower value, then you haven't "lost" anything, and claiming ACR's WB has a "conceptual error" is simply wrong.
[{POST_SNAPBACK}][/a]

One of the best explanations of white balance that I have seen is given by [a href=\"http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm] Guillermo Luijk[/url] on his web site.

The best dynamic range will be when the red, blue, and green multipliers are all equal. Some people use filters to accomplish this balancing (usually magenta for daylight). If you are using the white and black points to determine the dynamic range, then the color space has no effect. With colored targets, saturation clipping may occur, since RGB color spaces have a limited gamut near L = 0 and L = 1 as shown in this 3D gamut plot of sRGB vs ProPhotoRGB. The wire frame is ProPhotoRGB and the solid is sRGB.

[bjanes]

I can think of several practical reasons, not the least of which would be comparing various RAW converters to see which ones perform the best retaining shadow and highlight detail. For me, knowing which of my cameras performs better at a given ISO would be useful. So would knowing how much DR one is giving up by using camera JPEGs vs shooting RAW.
[a href=\"index.php?act=findpost&pid=160682\"][{POST_SNAPBACK}][/a]

The ISO question is very important, since the noise floor rises dramatically with ISO, especially on a camera such as the Nikon D200, which does not have good ISO performance. If you set the tolerable noise flow rather low for the best quality results, the D200 has less than 3 stops of DR as shown by this Imatest plot:

The problem with that reasoning is that doing standard statistical analyses of the RAW data is a very poor predictor of photographically useful DR and the negative visual impact of noise characteristics on an image.

One camera may have a better S/N ratio than another, but if its noise contains structures such as horizontal banding, it may be more visually distracting than another camera's noise pattern that, while mathematically greater, is more similar to film grain and less objectionable overall.
[a href=\"index.php?act=findpost&pid=160682\"][{POST_SNAPBACK}][/a]

Very important considerations. A standard deviation of the noise is objective, but for a given standard deviation, a camera with high resolution will have a finer noise pattern and the noise will be less objectionable. Imatest gives a noise spectrum analysis, but translating this into perceived noise is a bit difficult.

[Panopeeper]

You have to multiply the channel values to set WB. It makes no difference mathematically whether you use RGB multipliers <1 and increase the exposure setting, or use RGB multipliers >1 and reduce the exposure setting; the net result is exactly the same in both cases

You got it (almost).

If the source value range gets transformed doe to the application of the WB, then the white point needs to be transformed appropriately. The lack of this action is, what I called conceptual error. It is plain nonsense, that the user is confronted with an imaginary overexposure.