Real-World DR Testing

[Jonathan Wienke]

After seeing some of the debates over the best way to calculate a camera's photographically useful dynamic range, I've devised the following method using a new test target:

[attachment=4252:attachment]

The target has 4 quadrants; gray, red, green, and blue. Each quadrant has lines of text of progressively decreasing size and contrast. When the target is sufficiently overexposed, channel clipping will render text in one or more quadrants unreadable, and when the target is sufficiently underexposed, noise will render the text in one or more quadrants unreadable.

Print the chart using Absolute Colorimetric rendering intent. Adjust the framing so that the white box in the center is 100 pixels tall. Use continuous lighting, adjusting shutter speed only. Shoot in 1/3-stop increments far enough above and below normal exposure that you have RAWs with unrecoverable clipping and RAWs with enough noise to obliterate the smallest text. The exposure range where all text in all quadrants can be made legible during RAW conversion represents the photographically useful dynamic range of the camera.

[EDIT: Target image updated with instructions, and fixed clipping in blue channel.]

[juicy]

Interesting!

Most of the continuos light sources (besides sun and HMI) are almost devoid of blue frequencies, or in the case of fluorescents have very spiky spctra. Have you tested how much the spectral distribution of the lightsource affects the results?

Cheers,
J

[Jonathan Wienke]

The color temp of the lighting definitely affects the results you'll get, just as it does shooting in real-world conditions. For example, shooting in sunlight at noon, the color channels in the RAW are fairly well-balanced (within a stop or so of each other) in the RAW data with most cameras. But when shooting in incandescent lighting, the red channel can be more than 2 stops above the blue channel, and that will reduce your usable dynamic range with that lighting. There is definitely value in testing under a variety of lighting conditions to see how much lighting affects the DR of your camera.

[Ray]

Jonathan,
Sound like an interesting concept to me. I'll give it a go   . But I predict that the last thing to remain legible in the lowest stop will be the white on black title, 'Dynamic Range Test Target'.

If that's still legible, or still partly legible in the lowest stop, I take it we could include that stop in the dynamic range figure.

[Jonathan Wienke]

But I predict that the last thing to remain legible in the lowest stop will be the white on black title, 'Dynamic Range Test Target'.

If that's still legible, or still partly legible in the lowest stop, I take it we could include that stop in the dynamic range figure.

If you intend to use that criteria, then you should specify that in your results, as that will yield a much more generous figure than otherwise. I was thinking that a more realistic figure would be obtained from the legibility of the text inside the square. Of course, there's no reason you couldn't run both tests concurrently and report the figures separately.

BTW, the other purpose of the white square is a click target for setting white balance. Another use of the target is to test the camera's ability to distinguish subtle tonal gradations at various exposure levels. Comparing the exposure range where all of the smallest text in the square is legible vs the range where the all of the largest text is legible might be interesting as well.

[Kirk Gittings]

Interesting concept, that looks useful. I will try it out.

[Guillermo Luijk]

Very clever idea Jon, I will try it right now. Since your name is not in it, do you mind if I offer it in other forums?

Regards.

[Jonathan Wienke]

do you mind if I offer it in other forums?

As long as you credit me with the idea and the authorship of the target image, feel free. Linking to this thread would probably be the easiest way.

[Guillermo Luijk]

As long as you credit me with the idea and the authorship of the target image, feel free. Linking to this thread would probably be the easiest way.
[a href=\"index.php?act=findpost&pid=160416\"][{POST_SNAPBACK}][/a]

OK.

Can I make a suggestion? since the sequence 0123456789 is well know by anyone, I think it could be a good idea to include also a final sequence of random characters like those found in optician's test charts. In that way the observer will not have help from the sequence pattern to more properly decide whether a particular character is individually distinguishable or not. The numbers would still be necessary, since in real shots some elements show shape correlation that help identify their layouts.

Something like:
[span style=\'font-size:21pt;line-height:100%\']01234ZEDKS[/span]
[span style=\'font-size:14pt;line-height:100%\']01234YGPNA[/span]
[span style=\'font-size:8pt;line-height:100%\']01234WMXUF[/span]

Regards.

[Jonathan Wienke]

The only flaw in that logic is that one will still have the original chart for reference when doing the test, so "cheating" is possible no matter how random the sequences of characters are. And when doing the evaluation in the RAW converter, you have other shots in the sequence to look at anyway, which may be easier to read.

[Graeme Nattress]

I think you'd be better off with a transmissive variant, so you can measure with one exposure, rather than having to take multiple exposures. I always find that quicker / easier.

Graeme

[Jonathan Wienke]

But then you're limited by the DR of the print process and material, and the evenness of the lighting becomes an issue, especially stray background light contaminating the shadows. With this method, you do have to make more than one exposure, but all of the other things become pretty much non-issues.

[Jonathan Wienke]

I updated the target file, I didn't notice that there was some clipping in the blue channel, making the text in the blue quadrant slightly less legible than the other quadrants. This has been fixed.

[allan67]

I updated the target file, I didn't notice that there was some clipping in the blue channel, making the text in the blue quadrant slightly less legible than the other quadrants. This has been fixed.
[a href=\"index.php?act=findpost&pid=160622\"][{POST_SNAPBACK}][/a]

Thanks for a very good idea, Johathan.
I'll try this this weekend with my 30D and S70. Would be interesting to see how dSLR compares to point-n-shoot (S70 supports RAW).

I've noticed that some figures have discolouration in them.
This is especially visble in Red and Green quadrants.
Look at lower parts of biggest type size.
Is this some posterisation effect from changing contrast from one line to the next?

Allan

[Tim Gray]

thanks Jonathan.  we're forecast for a big storm this weekend, so I'm looking for something to do inside

Hopefully folks will publish their results and we can see what the distribution looks like...

[Graeme Nattress]

The limit of the transmissive process is over 13 stops though.... And yes, you need even lighting and be able to avoid stray lighting. No method is perfect, but I do find a controlled transmissive test very repeatable, whereas a manually stepped test has more chance for human error. That said, your concept is good for getting a real world level of indication.

Graeme

[Jonathan Wienke]

I've noticed that some figures have discolouration in them.
This is especially visble in Red and Green quadrants.
Look at lower parts of biggest type size.
Is this some posterisation effect from changing contrast from one line to the next?

Is this in the updated version of the chart? I'm not seeing anything like that in the updated chart (the one with the copyright notice and instructions).

[bjanes]

The limit of the transmissive process is over 13 stops though.... And yes, you need even lighting and be able to avoid stray lighting. No method is perfect, but I do find a controlled transmissive test very repeatable, whereas a manually stepped test has more chance for human error. That said, your concept is good for getting a real world level of indication.

Graeme
[{POST_SNAPBACK}][/a]

In my own work with the Stouffer step-wedge, I have found that it is very important to mask off the target surround to avoid excessive flare light originating in the lens and camera box. Even so, there is some flare from the brighter portions of the target and from the numbers on the step chart. Since the contrast ratio of Jonathan's target is lower, I would think that the effects of flare would be diminished. One could incorporate a [a href=\"http://www.imatest.com/docs/veilingglare.html]black hole[/url] into the target for further evaluation. One should also use a prime lens with multicoating (present in most lenses nowadays) and possibly Nano-Crystal Coating.

Modern electronic shutters are very precise and accurate, but one must also take into account the repeatability of the auto-diaphragm mechanism.

With multiple exposures, it is also important to make certain that the processing is the same for all images. Some raw converters use an adaptive tone curve, which may vary with exposure. It would probably be best to work with the raw data directly with Iris or DCRaw.

With all the above having been said, what noise floor to accept for decent photographic results is rather subjective and all of this precision may not be needed for practical purposes. The results with Imatest Stepchart seem reasonable for normal purposes and are easy to obtain.

Your comments would be appreciated.

Bill

[Jonathan Wienke]

Modern electronic shutters are very precise and accurate, but one must also take into account the repeatability of the auto-diaphragm mechanism.

With multiple exposures, it is also important to make certain that the processing is the same for all images. Some raw converters use an adaptive tone curve, which may vary with exposure. It would probably be best to work with the raw data directly with Iris or DCRaw.

Regarding diaphragm consistency, the easiest way to eliminate that variable is to simply shoot wide open, so that the diaphragm is not moving at all.

As to the tone curve issue, I disagree. Use the same RAW converter you use to process your images.If you can make the text legible by adjusting the exposure control or curve in the RAW converter, then you can do the same thing on a real image to dig detail out of the highlights or shadows.

I expect that some RAW converters will do much better than others. ACR handles highlights particularly well; Capture One and some of the MFDB manufacturer's programs seem to toss out 2-3 stops of highlights, a thing I find curious given that increased DR is one of the big advantages of MFDBs.

With all the above having been said, what noise floor to accept for decent photographic results is rather subjective and all of this precision may not be needed for practical purposes.

That's the main point of my chart: basing the definition of the noise floor on something as close as possible to what photographers do with real life images, the ability to visually distinguish low-contrast image details and textures in the highlights and shadows. A simple S/N ratio works well for predicting high-contrast test chart performance, but that is useless when trying to get some texture and detail from a wedding dress or black velvet. Whether or not one can read a bit of text in background noise is still somewhat subjective, but much more objective than some of the other methods floating around.

[Panopeeper]

Use the same RAW converter you use to process your images.If you can make the text legible by adjusting the exposure control or curve in the RAW converter, then you can do the same thing on a real image to dig detail out of the highlights or shadows

At this point one needs to think about the sense of the excercise.

Do you own a camera and want to measure it's DR? What for? It's a bit too late, isn't it?

Do you want to compare different cameras? If you do it for yourself (i.e. if you can carry out the test with different cameras before making the decision), then it is a useful test.

However, if you are doing this for a "public" review/test/comparison, then it is pretty much useless; what is the sense in doing that with a particular raw converter?

ACR handles highlights particularly well

ACR handles highlight particularly badly. Not only, that it's clipping indication is worthless regarding the truth about the exposure (that is the case with all raw converters I know), but the support of some cameras is totally off in this regard.

Capture One and some of the MFDB manufacturer's programs seem to toss out 2-3 stops of highlights

I suspect this is not so. This is my "belief", because I am not working with C1. I think those highlights can be "recovered" (a very bad term in this context, because they were not lost in the first place) by simply reducing the exposure. This is, what one has to do in ACR as well, when the program misinterprets the raw data.

The fact, that this is necessary with the HW manufacturer's own software is quite troubling in my eyes.

Whether or not one can read a bit of text in background noise is still somewhat subjective, but much more objective than some of the other methods floating around.

The most objective method what I saw is the statistical analysis of the data. The pixels of an evenly lit uniform surface are supposed to be close to identical; the deviation indicates the noise (above some limit, but that's irrelevant here).