Luminous Landscape Forum
Equipment & Techniques => Digital Cameras & Shooting Techniques => Topic started by: Jonathan Wienke on December 13, 2007, 12:29:43 pm
-
After seeing some of the debates over the best way to calculate a camera's photographically useful dynamic range, I've devised the following method using a new test target:
[attachment=4252:attachment]
The target has 4 quadrants; gray, red, green, and blue. Each quadrant has lines of text of progressively decreasing size and contrast. When the target is sufficiently overexposed, channel clipping will render text in one or more quadrants unreadable, and when the target is sufficiently underexposed, noise will render the text in one or more quadrants unreadable.
Print the chart using Absolute Colorimetric rendering intent. Adjust the framing so that the white box in the center is 100 pixels tall. Use continuous lighting, adjusting shutter speed only. Shoot in 1/3-stop increments far enough above and below normal exposure that you have RAWs with unrecoverable clipping and RAWs with enough noise to obliterate the smallest text. The exposure range where all text in all quadrants can be made legible during RAW conversion represents the photographically useful dynamic range of the camera.
[EDIT: Target image updated with instructions, and fixed clipping in blue channel.]
-
Interesting!
Most of the continuos light sources (besides sun and HMI) are almost devoid of blue frequencies, or in the case of fluorescents have very spiky spctra. Have you tested how much the spectral distribution of the lightsource affects the results?
Cheers,
J
-
The color temp of the lighting definitely affects the results you'll get, just as it does shooting in real-world conditions. For example, shooting in sunlight at noon, the color channels in the RAW are fairly well-balanced (within a stop or so of each other) in the RAW data with most cameras. But when shooting in incandescent lighting, the red channel can be more than 2 stops above the blue channel, and that will reduce your usable dynamic range with that lighting. There is definitely value in testing under a variety of lighting conditions to see how much lighting affects the DR of your camera.
-
Jonathan,
Sound like an interesting concept to me. I'll give it a go . But I predict that the last thing to remain legible in the lowest stop will be the white on black title, 'Dynamic Range Test Target'.
If that's still legible, or still partly legible in the lowest stop, I take it we could include that stop in the dynamic range figure.
-
But I predict that the last thing to remain legible in the lowest stop will be the white on black title, 'Dynamic Range Test Target'.
If that's still legible, or still partly legible in the lowest stop, I take it we could include that stop in the dynamic range figure.
If you intend to use that criteria, then you should specify that in your results, as that will yield a much more generous figure than otherwise. I was thinking that a more realistic figure would be obtained from the legibility of the text inside the square. Of course, there's no reason you couldn't run both tests concurrently and report the figures separately.
BTW, the other purpose of the white square is a click target for setting white balance. Another use of the target is to test the camera's ability to distinguish subtle tonal gradations at various exposure levels. Comparing the exposure range where all of the smallest text in the square is legible vs the range where the all of the largest text is legible might be interesting as well.
-
Interesting concept, that looks useful. I will try it out.
-
Very clever idea Jon, I will try it right now. Since your name is not in it, do you mind if I offer it in other forums?
Regards.
-
do you mind if I offer it in other forums?
As long as you credit me with the idea and the authorship of the target image, feel free. Linking to this thread would probably be the easiest way.
-
As long as you credit me with the idea and the authorship of the target image, feel free. Linking to this thread would probably be the easiest way.
[a href=\"index.php?act=findpost&pid=160416\"][{POST_SNAPBACK}][/a]
OK.
Can I make a suggestion? since the sequence 0123456789 is well know by anyone, I think it could be a good idea to include also a final sequence of random characters like those found in optician's test charts. In that way the observer will not have help from the sequence pattern to more properly decide whether a particular character is individually distinguishable or not. The numbers would still be necessary, since in real shots some elements show shape correlation that help identify their layouts.
Something like:
[span style=\'font-size:21pt;line-height:100%\']01234ZEDKS[/span]
[span style=\'font-size:14pt;line-height:100%\']01234YGPNA[/span]
[span style=\'font-size:8pt;line-height:100%\']01234WMXUF[/span]
Regards.
-
The only flaw in that logic is that one will still have the original chart for reference when doing the test, so "cheating" is possible no matter how random the sequences of characters are. And when doing the evaluation in the RAW converter, you have other shots in the sequence to look at anyway, which may be easier to read.
-
I think you'd be better off with a transmissive variant, so you can measure with one exposure, rather than having to take multiple exposures. I always find that quicker / easier.
Graeme
-
But then you're limited by the DR of the print process and material, and the evenness of the lighting becomes an issue, especially stray background light contaminating the shadows. With this method, you do have to make more than one exposure, but all of the other things become pretty much non-issues.
-
I updated the target file, I didn't notice that there was some clipping in the blue channel, making the text in the blue quadrant slightly less legible than the other quadrants. This has been fixed.
-
I updated the target file, I didn't notice that there was some clipping in the blue channel, making the text in the blue quadrant slightly less legible than the other quadrants. This has been fixed.
[a href=\"index.php?act=findpost&pid=160622\"][{POST_SNAPBACK}][/a]
Thanks for a very good idea, Johathan.
I'll try this this weekend with my 30D and S70. Would be interesting to see how dSLR compares to point-n-shoot (S70 supports RAW).
I've noticed that some figures have discolouration in them.
This is especially visble in Red and Green quadrants.
Look at lower parts of biggest type size.
Is this some posterisation effect from changing contrast from one line to the next?
Allan
-
thanks Jonathan. we're forecast for a big storm this weekend, so I'm looking for something to do inside
Hopefully folks will publish their results and we can see what the distribution looks like...
-
The limit of the transmissive process is over 13 stops though.... And yes, you need even lighting and be able to avoid stray lighting. No method is perfect, but I do find a controlled transmissive test very repeatable, whereas a manually stepped test has more chance for human error. That said, your concept is good for getting a real world level of indication.
Graeme
-
I've noticed that some figures have discolouration in them.
This is especially visble in Red and Green quadrants.
Look at lower parts of biggest type size.
Is this some posterisation effect from changing contrast from one line to the next?
Is this in the updated version of the chart? I'm not seeing anything like that in the updated chart (the one with the copyright notice and instructions).
-
The limit of the transmissive process is over 13 stops though.... And yes, you need even lighting and be able to avoid stray lighting. No method is perfect, but I do find a controlled transmissive test very repeatable, whereas a manually stepped test has more chance for human error. That said, your concept is good for getting a real world level of indication.
Graeme
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=160647\")
In my own work with the Stouffer step-wedge, I have found that it is very important to mask off the target surround to avoid excessive flare light originating in the lens and camera box. Even so, there is some flare from the brighter portions of the target and from the numbers on the step chart. Since the contrast ratio of Jonathan's target is lower, I would think that the effects of flare would be diminished. One could incorporate a [a href=\"http://www.imatest.com/docs/veilingglare.html]black hole[/url] into the target for further evaluation. One should also use a prime lens with multicoating (present in most lenses nowadays) and possibly Nano-Crystal Coating.
Modern electronic shutters are very precise and accurate, but one must also take into account the repeatability of the auto-diaphragm mechanism.
With multiple exposures, it is also important to make certain that the processing is the same for all images. Some raw converters use an adaptive tone curve, which may vary with exposure. It would probably be best to work with the raw data directly with Iris or DCRaw.
With all the above having been said, what noise floor to accept for decent photographic results is rather subjective and all of this precision may not be needed for practical purposes. The results with Imatest Stepchart (http://www.imatest.com/docs/tour_q13.html#dynamic) seem reasonable for normal purposes and are easy to obtain.
Your comments would be appreciated.
Bill
-
Modern electronic shutters are very precise and accurate, but one must also take into account the repeatability of the auto-diaphragm mechanism.
With multiple exposures, it is also important to make certain that the processing is the same for all images. Some raw converters use an adaptive tone curve, which may vary with exposure. It would probably be best to work with the raw data directly with Iris or DCRaw.
Regarding diaphragm consistency, the easiest way to eliminate that variable is to simply shoot wide open, so that the diaphragm is not moving at all.
As to the tone curve issue, I disagree. Use the same RAW converter you use to process your images.If you can make the text legible by adjusting the exposure control or curve in the RAW converter, then you can do the same thing on a real image to dig detail out of the highlights or shadows.
I expect that some RAW converters will do much better than others. ACR handles highlights particularly well; Capture One and some of the MFDB manufacturer's programs seem to toss out 2-3 stops of highlights, a thing I find curious given that increased DR is one of the big advantages of MFDBs.
With all the above having been said, what noise floor to accept for decent photographic results is rather subjective and all of this precision may not be needed for practical purposes.
That's the main point of my chart: basing the definition of the noise floor on something as close as possible to what photographers do with real life images, the ability to visually distinguish low-contrast image details and textures in the highlights and shadows. A simple S/N ratio works well for predicting high-contrast test chart performance, but that is useless when trying to get some texture and detail from a wedding dress or black velvet. Whether or not one can read a bit of text in background noise is still somewhat subjective, but much more objective than some of the other methods floating around.
-
Use the same RAW converter you use to process your images.If you can make the text legible by adjusting the exposure control or curve in the RAW converter, then you can do the same thing on a real image to dig detail out of the highlights or shadows
At this point one needs to think about the sense of the excercise.
Do you own a camera and want to measure it's DR? What for? It's a bit too late, isn't it?
Do you want to compare different cameras? If you do it for yourself (i.e. if you can carry out the test with different cameras before making the decision), then it is a useful test.
However, if you are doing this for a "public" review/test/comparison, then it is pretty much useless; what is the sense in doing that with a particular raw converter?
ACR handles highlights particularly well
ACR handles highlight particularly badly. Not only, that it's clipping indication is worthless regarding the truth about the exposure (that is the case with all raw converters I know), but the support of some cameras is totally off in this regard.
Capture One and some of the MFDB manufacturer's programs seem to toss out 2-3 stops of highlights
I suspect this is not so. This is my "belief", because I am not working with C1. I think those highlights can be "recovered" (a very bad term in this context, because they were not lost in the first place) by simply reducing the exposure. This is, what one has to do in ACR as well, when the program misinterprets the raw data.
The fact, that this is necessary with the HW manufacturer's own software is quite troubling in my eyes.
Whether or not one can read a bit of text in background noise is still somewhat subjective, but much more objective than some of the other methods floating around.
The most objective method what I saw is the statistical analysis of the data. The pixels of an evenly lit uniform surface are supposed to be close to identical; the deviation indicates the noise (above some limit, but that's irrelevant here).
-
At this point one needs to think about the sense of the excercise.
Do you own a camera and want to measure it's DR? What for? It's a bit too late, isn't it?
Do you want to compare different cameras? If you do it for yourself (i.e. if you can carry out the test with different cameras before making the decision), then it is a useful test.
However, if you are doing this for a "public" review/test/comparison, then it is pretty much useless; what is the sense in doing that with a particular raw converter?
I can think of several practical reasons, not the least of which would be comparing various RAW converters to see which ones perform the best retaining shadow and highlight detail. For me, knowing which of my cameras performs better at a given ISO would be useful. So would knowing how much DR one is giving up by using camera JPEGs vs shooting RAW.
And IMO, a reviewer such as Phil Askey doing such my DR test in a standardized manner would add value to his reviews.
ACR handles highlight particularly badly. Not only, that it's clipping indication is worthless regarding the truth about the exposure (that is the case with all raw converters I know), but the support of some cameras is totally off in this regard.
I'm a bit surprised by this statement; with Canon DSLRs at least, ACR can extract nearly a stop more usable highlight detail than the other RAW converters I tried. The last time I compared was a couple years ago though.
The most objective method what I saw is the statistical analysis of the data. The pixels of an evenly lit uniform surface are supposed to be close to identical; the deviation indicates the noise (above some limit, but that's irrelevant here).
The problem with that reasoning is that doing standard statistical analyses of the RAW data is a very poor predictor of photographically useful DR and the negative visual impact of noise characteristics on an image. One camera may have a better S/N ratio than another, but if its noise contains structures such as horizontal banding, it may be more visually distracting than another camera's noise pattern that, while mathematically greater, is more similar to film grain and less objectionable overall.
-
For me, knowing which of my cameras performs better at a given ISO would be useful. So would knowing how much DR one is giving up by using camera JPEGs vs shooting RAW
Fair enough. I do that differently, but I acknowledge, that maniac pixel peeping is not everyone's favourite pastime.
a reviewer such as Phil Askey doing such my DR test in a standardized manner would add value to his reviews
DPReview measurement is based on a step wedge, with de-mosaiced data. I favour not de-mosaiced data, but DPReview's method is consistent among cameras; as such, it is useful. Plus, as Alex posted it on another thread, he does not care, what his camera could produce, all he cares for is, what the supplied software is making out of it. This is a valid position for probably the vast majority of photographers (although I am surprized, that the most professionals are those, who care the least for such issues).
I'm a bit surprised by this statement; with Canon DSLRs at least, ACR can extract nearly a stop more usable highlight detail than the other RAW converters I tried
I think it is a general rule, that the manufacturer's own software can make the most out of the image (Phase one may be an exception). DPP knows the Canon cameras' characteristics much better, than ACR does (I still favour ACR because of its features).
It is very surprising and disappointing, that Adobe does not invest a bit more investigation into such cameras, which make out the majority of the DSLR market.
There are two issues (the two extremes) I have with ACR.
1. The noise chacateristics of Canon cameras is not analyzed good enough. Canon cameras deliver many thousands of masked pixels for the evaluation of the black level, and ACR does not make the best out of that.
2. The clipping point of the pixels is an important issue. I found with several cameras, that ACR mistreats the image by assuming incorrect clipping point (already the fact, that all pixels are assumed to have identical clipping point is a conceptual error.
The most noticable error, what I found occurs with the Canon 40D. ACR assumes clipping at 13600, no matter what ISO. The actual clipping points are, depending on the ISO, between 12740 and 16383.
The problem with that reasoning is that doing standard statistical analyses of the RAW data is a very poor predictor of photographically useful DR and the negative visual impact of noise characteristics on an image
This is true for every standardized method. There is no way of properly judging a camera w/o looking at samples of different situations. For example much more noise on cloth is acceptable than on a smooth surface.
-
Do you own a camera and want to measure it's DR? What for? It's a bit too late, isn't it?
Inquiring minds want to know
-
I think it is a general rule, that the manufacturer's own software can make the most out of the image (Phase one may be an exception). DPP knows the Canon cameras' characteristics much better, than ACR does (I still favour ACR because of its features).
That's what the manufacturers claim, but IMO that claim is mostly myth and hocus-pocus. ACR (after calibration with the Fors script) has always given me better color than any other RAW converter, including DPP, the Canon RAW converter that preceded DPP, and Olympus' RAW converter. Regarding noise removal and sharpening, I have my own methods that work better than what's built in to any RAW converter I've tried so far, and getting sharp, detailed images with reasonably low noise levels has never been a problem for me.
There are two issues (the two extremes) I have with ACR.
1. The noise chacateristics of Canon cameras is not analyzed good enough. Canon cameras deliver many thousands of masked pixels for the evaluation of the black level, and ACR does not make the best out of that.
2. The clipping point of the pixels is an important issue. I found with several cameras, that ACR mistreats the image by assuming incorrect clipping point (already the fact, that all pixels are assumed to have identical clipping point is a conceptual error.
The most noticable error, what I found occurs with the Canon 40D. ACR assumes clipping at 13600, no matter what ISO. The actual clipping points are, depending on the ISO, between 12740 and 16383.
And that is about 1/3 of a stop at most. If that's the worst-case scenario I wouldn't spend too much time worrying about it. It certainly isn't in the same league as some of the MFDB software that claims RAWs 3 stops under clipping are exposed to the right.
-
I expect that some RAW converters will do much better than others. ACR handles highlights particularly well; Capture One and some of the MFDB manufacturer's programs seem to toss out 2-3 stops of highlights, a thing I find curious given that increased DR is one of the big advantages of MFDBs.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=160666\")
No raw converter can invent details that are not present in the image, but they can reconstruct blown channels if there is at least one channel with detail. Because of white balance, the red and blue channels have a multiplier to equalize them with the green channel, which is usually the most sensitive.
For example, the multipliers for some Nikon cameras are given on the [a href=\"http://www.pochtar.com/NikonWhiteBalanceCoeffs.htm]Borg Web site [/url]. For daylight with the D2x the red and green multipliers are 1.52 and 1.69 respectively. If you overexpose by 0.5 EV for the green channels, the red and blue channels will not be clipped and the green channel can be reconstructed to some extent. However, I would not consider this to be true dynamic range. For this reason, DR is usually measured in the greens. DR in the red and blue channel would be less, because these channels are not fully exposed to the right. One can get better overall DR by placing a magenta filter over the lens to hold back some of the green light and equalize the channels. The multipliers are not that different for Canon sensors.
Consideration of these principles show why highlight recovery is usually possible out to no more than 1 stop overexposure.
I find it difficult to believe that any camera would throw away 2-3 stops of data.
Bill
-
No raw converter can invent details that are not present in the image, but they can reconstruct blown channels if there is at least one channel with detail. Because of white balance, the red and blue channels have a multiplier to equalize them with the green channel, which is usually the most sensitive.
For example, the multipliers for some Nikon cameras are given on the Borg Web site (http://www.pochtar.com/NikonWhiteBalanceCoeffs.htm). For daylight with the D2x the red and green multipliers are 1.52 and 1.69 respectively. If you overexpose by 0.5 EV for the green channels, the red and blue channels will not be clipped and the green channel can be reconstructed to some extent. However, I would not consider this to be true dynamic range.
The color balance issue is why my chart has the color quadrants as well as the gray quadrant, so that clipping in a single color channel is more likely to be noticeable than in a neutral gray. In incandescent lighting, the red channel is usually the first to blow since its color temp is so low.
-
That's what the manufacturers claim, but IMO that claim is mostly myth and hocus-pocus
The manufacturers are suckers in software, but they have the required data.
There is a clear concensus among Canon users, that DPP gives the best colors. The fact, that ACR *needs* to be customized shows its weakness, but the fact, that it *can* be customized shows its strength.
Now DPP too can be customized via Picture Styles.
that is about 1/3 of a stop at most. If that's the worst-case scenario I wouldn't spend too much time worrying about it
Tell it to someone, who just exposed to the right.
Re Phase One and C1: luckily it is not the accepted measurement of raw processing. Btw, ACR too has a big conceptual error: applying the WB can lead to RGB clipping, even though there was no raw clipping.
-
There is a clear concensus among Canon users, that DPP gives the best colors. The fact, that ACR *needs* to be customized shows its weakness, but the fact, that it *can* be customized shows its strength.
Unit variation can also account for calibration differences it's not necessarily a design flaw in ACR.
Tell it to someone, who just exposed to the right.
1/3-stop maximum non-optimality is much better than a software that tells someone that a 2-stop underexposure is exposed to the right or clipped.
Btw, ACR too has a big conceptual error: applying the WB can lead to RGB clipping, even though there was no raw clipping.
That can be solved by using a large-gamut color space like ProPhoto. It's not a problem with ACR, it's PEBKAC.
-
That can be solved by using a large-gamut color space like ProPhoto. It's not a problem with ACR, it's PEBKAC.
I'm afraid you don't understand the problem. It is not a color issue; the role of WB hier is not, that it creates undisplayable colors, but that it "multiplies out" a channel. If you apply that WB and reduce the exposure, then you get back what you "lost" before.
Addendum: I realize, that applying the WB can lead to undisplayable colors, but that's not all; the problem exists in ppRGB as well.
-
The color balance issue is why my chart has the color quadrants as well as the gray quadrant, so that clipping in a single color channel is more likely to be noticeable than in a neutral gray. In incandescent lighting, the red channel is usually the first to blow since its color temp is so low.
[a href=\"index.php?act=findpost&pid=160743\"][{POST_SNAPBACK}][/a]
Green and red are pretty even, actually, in the RAW data, in incandescent light. Only the blue is usually compromised. If the light source is a heat lamp, like the kind they use to keep fried chicken hot, then red might take a big lead.
-
I'm afraid you don't understand the problem. It is not a color issue; the role of WB hier is not, that it creates undisplayable colors, but that it "multiplies out" a channel. If you apply that WB and reduce the exposure, then you get back what you "lost" before.
I'm afraid you don't understand how white balancing works. You have to multiply the channel values to set WB. It makes no difference mathematically whether you use RGB multipliers <1 and increase the exposure setting, or use RGB multipliers >1 and reduce the exposure setting; the net result is exactly the same in both cases. If you can unclip the data by setting exposure to a lower value, then you haven't "lost" anything, and claiming ACR's WB has a "conceptual error" is simply wrong.
-
Green and red are pretty even, actually, in the RAW data, in incandescent light. Only the blue is usually compromised.
Depends on the camera. With the 1D-MkII, the red channel will clip about 1/2 stop before green shooting in incandescent. With the 1Ds, they're fairly even. YMMV.
-
Yes, any clipping from WB can be reduced or eliminated by reducing exposure. And as noted above, unclipped channels in the raw can be used to produce a pretty convincing facsimile of the clipped one too, for extra semi-faked DR.
However, WB is not just a simple multiplier on R, G and B, but is usually performed by a 3x3 matrix, using a complex mingling of R, G and B.
Graeme
-
I'm afraid you don't understand how white balancing works. You have to multiply the channel values to set WB. It makes no difference mathematically whether you use RGB multipliers <1 and increase the exposure setting, or use RGB multipliers >1 and reduce the exposure setting; the net result is exactly the same in both cases. If you can unclip the data by setting exposure to a lower value, then you haven't "lost" anything, and claiming ACR's WB has a "conceptual error" is simply wrong.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=160830\")
One of the best explanations of white balance that I have seen is given by [a href=\"http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm] Guillermo Luijk[/url] on his web site.
The best dynamic range will be when the red, blue, and green multipliers are all equal. Some people use filters to accomplish this balancing (usually magenta for daylight). If you are using the white and black points to determine the dynamic range, then the color space has no effect. With colored targets, saturation clipping may occur, since RGB color spaces have a limited gamut near L = 0 and L = 1 as shown in this 3D gamut plot of sRGB vs ProPhotoRGB. The wire frame is ProPhotoRGB and the solid is sRGB.
(http://bjanes.smugmug.com/photos/232506179-O.png)
-
I can think of several practical reasons, not the least of which would be comparing various RAW converters to see which ones perform the best retaining shadow and highlight detail. For me, knowing which of my cameras performs better at a given ISO would be useful. So would knowing how much DR one is giving up by using camera JPEGs vs shooting RAW.
[a href=\"index.php?act=findpost&pid=160682\"][{POST_SNAPBACK}][/a]
The ISO question is very important, since the noise floor rises dramatically with ISO, especially on a camera such as the Nikon D200, which does not have good ISO performance. If you set the tolerable noise flow rather low for the best quality results, the D200 has less than 3 stops of DR as shown by this Imatest plot:
(http://bjanes.smugmug.com/photos/174709450-O.gif)
The problem with that reasoning is that doing standard statistical analyses of the RAW data is a very poor predictor of photographically useful DR and the negative visual impact of noise characteristics on an image.
One camera may have a better S/N ratio than another, but if its noise contains structures such as horizontal banding, it may be more visually distracting than another camera's noise pattern that, while mathematically greater, is more similar to film grain and less objectionable overall.
[a href=\"index.php?act=findpost&pid=160682\"][{POST_SNAPBACK}][/a]
Very important considerations. A standard deviation of the noise is objective, but for a given standard deviation, a camera with high resolution will have a finer noise pattern and the noise will be less objectionable. Imatest gives a noise spectrum analysis, but translating this into perceived noise is a bit difficult.
-
You have to multiply the channel values to set WB. It makes no difference mathematically whether you use RGB multipliers <1 and increase the exposure setting, or use RGB multipliers >1 and reduce the exposure setting; the net result is exactly the same in both cases
You got it (almost).
If the source value range gets transformed doe to the application of the WB, then the white point needs to be transformed appropriately. The lack of this action is, what I called conceptual error. It is plain nonsense, that the user is confronted with an imaginary overexposure.