Canon 5D dynamic range tests

[Ray]

As I suspected, using the legibility of large B&W text as a criterion for DR assessment results in a really high figure.

Unfortunately, I made the mistake of not continuing far enough with the underexposures, so it's not clear at what point the B&W text would cease to be legible. At least another 2/3rds of a stop I would think, giving a maximum DR of 12 stops.

Here are two shots 11.33 stops apart. I'm assuming with this method that one should take the interval in f stops between the longest and shortest exposure and not count the actual number of exposures taken at one stop intervals.

The exposures range from 2 secs to 1/1250th.

[attachment=4240:attachment]  [attachment=4241:attachment]

[Jonathan Wienke]

Using the white text legibility is really a not a valid method, especially if you let the white text clip on the high end of the exposure series. If you do that, you're comparing apples and oranges and the results you get will be inflated by the density difference between the white text and the text in the quadrants.

But even if you use the white text for clipping and legibility, the DR figure you've arrived at is only valid when shooting high-contrast white-on-black subjects, which isn't really very useful. Most of the time we photographers are trying to pull low-contrast tonal subtleties from the shadows--woodgrain texture from a piece of furniture, the texture of the subject's hair, etc. and the text in the quadrants is a much better predictor of how well a particular camera + RAW converter will accomplish that. What results do you get using the quadrant text?

I updated the target image, please download the new one. It fixes the slight clipping present in the blue quadrant.

[Jonathan Wienke]

One other thing; it appears that the white square is bigger than 100 pixels across in your test shot. Maintaining a consistent size is important, because the bigger the letters are at 100%, the easier it is to read them. If you shoot the chart too large it will skew the results.

[bjanes]

As I suspected, using the legibility of large B&W text as a criterion for DR assessment results in a really high figure.

Unfortunately, I made the mistake of not continuing far enough with the underexposures, so it's not clear at what point the B&W text would cease to be legible. At least another 2/3rds of a stop I would think, giving a maximum DR of 12 stops.

Here are two shots 11.33 stops apart. I'm assuming with this method that one should take the interval in f stops between the longest and shortest exposure and not count the actual number of exposures taken at one stop intervals.

The exposures range from 2 secs to 1/1250th.
[a href=\"index.php?act=findpost&pid=160579\"][{POST_SNAPBACK}][/a]

Ray,

The legibility test is one way of determining the noise floor, but the question remains: is that noise level acceptable? For high quality work, many would set the noise floor higher, and this is a subjective matter.

Bill

[Ray]

Using the white text legibility is really a not a valid method, especially if you let the white text clip on the high end of the exposure series. If you do that, you're comparing apples and oranges and the results you get will be inflated by the density difference between the white text and the text in the quadrants.

Jonathan,
(1)There's no indication of clipping of the whites in my ETTR at 2secs. I printed the target on matte paper and the texture of the paper is evident in the white square in the centre and the white border of the A4 sheet, ie. the values fluctuate between 214,214,214 and 218,218,218. The fluctuations of the white border are greater, varying from 221 to 228.

(2) Not only is the B&W text clearly readable 11.3 stops down but also the black cross in the centre. Now obviously such broad-brush detail in such degraded shadows is not much use to a fine-art landscaper but it would be to a spy who's life depended on decoding a message in the 12th stop of a Canon 5D image   , or indeed an amateur astronomer searching for a new discovery that could get him a name in history.

I haven't made much attempt to determine a more useful DR for ordinary photography because I'm at a severe disadvantage regarding the status of my printer. Since I'm at present travelling, I'm using Epson's bottom-of-the-range 4-ink printer which I shall probably discard when I move from my current lodgings.

It's really quite primitive. One generic profile for everything, it makes a noise like a steam engine (clickety clack, clickety clack) and has a tendency to stop printing whenever a light is switched on or off, either in my room or the adjoining room, or whenever the air-conditioner is switched on or off. I've wasted a few prints as a result.

I'll try printing a glossy of your new target, but I don't think that Absolute Colorimetric has any meaning for my printer's color management. On the matte paper I can hardly read the smallest, faintest numbers.

By the way, why are you standardising the size of the target in terms of pixels? This is like dpreview's method of comparing noise. You get results like, the D60 has less noise than the 1Ds up to ISO 400. Surely what's important is the dynamic range of the whole camera.

Also, you didn't answer my question regarding the method of counting stops. Is one stop the interval between one exposure and the next exposure at half the value? On that basis, the extreme DR of the 5D is 11.3 stops. However, if I count all the exposures with a one stop interval between them, I get 12.3 stops.

I'll try this again with my best glossy print of your new target   .

[Panopeeper]

Ray,

1. I don't think the glossy paper is a good idea,

2. if you post the raw images, I can verify the clipping and the differences in stops, as well as the standard deviation on the darkest areas.

[Ray]

Ray,

1. I don't think the glossy paper is a good idea,

2. if you post the raw images, I can verify the clipping and the differences in stops, as well as the standard deviation on the darkest areas.
[a href=\"index.php?act=findpost&pid=160799\"][{POST_SNAPBACK}][/a]

Panopeeper,
Nor did I, which is why I initially printed the target on matte. But the fact is, if I can hardly read the smallest and faintest numbers before I've photographed the target, what chance have I of reading them after they've been photographed.

I've just discovered that one can join Yousendit free for 1GB per month of uploads.

I'll see if I can work out how to use it and take up your offer. I tried sending Guillermo a RAW file once, through my ISP in Australia, but failed.

[Jonathan Wienke]

But the fact is, if I can hardly read the smallest and faintest numbers before I've photographed the target, what chance have I of reading them after they've been photographed.

That's the whole point of the exercise, photograph something with subtle detail that isn't glaringly blatant. You've already demonstrated that using the legibility of the large white-on-black letters skews the results unrealistically high. And the smallest letters are distinguishable in the screenshot you posted already, so your camera is capable of capturing them legibly. It wouldn't be cheating to view the test frames at >100% in ACR, you know.

By the way, why are you standardising the size of the target in terms of pixels? This is like dpreview's method of comparing noise. You get results like, the D60 has less noise than the 1Ds up to ISO 400. Surely what's important is the dynamic range of the whole camera.

And it's equally silly to rate a high-pixel-count digicam better than a DSLR simply because it can throw more pixels at the subject. The larger the letters are, the easier it is to pick them out of the shadow noise. If you fill the frame with the chart, a camera with more megapixels will have an unfair advantage on that basis alone, and cameras with greater pixel counts will get a higher DR rating than they are actually entitled to. Keeping things constant across the board will give you a standardized evaluation of pixel quality. What's the point of increasing pixel count if you're not simultaneously increasing your ability to resolve fine details?

Also, you didn't answer my question regarding the method of counting stops. Is one stop the interval between one exposure and the next exposure at half the value? On that basis, the extreme DR of the 5D is 11.3 stops. However, if I count all the exposures with a one stop interval between them, I get 12.3 stops.

Given a 1/3-stop exposure interval, count the number of exposures where the quadrant text is legible and divide by 3. If you shot 21 frames at 1/3-stop intervals where the quadrant text is legible, then photographically useful DR is 7 stops.

[Ray]

That's the whole point of the exercise, photograph something with subtle detail that isn't glaringly blatant.

Jonathan,
In that case the quality of the print in conjunction with the quality of the lens will affect the result significantly.

And it's equally silly to rate a high-pixel-count digicam better than a DSLR simply because it can throw more pixels at the subject.

I doubt whether a high pixel digicam, say a 12mp Canon G9, could record finer detail at say 6 stops underexposure, than a 6mp DSLR, but if it could, it wouldn't be silly, it would be bloody marvelous   .

I don't see why this should be a competition like a horse race where the stronger horse is handicapped. We're concerned about reality here, aren't we? Refer to the title of your thread, 'Real world DR testing'. If my camera exhibits greater DR in practice, in real world shooting, because it has more pixels, then so be it. Maybe that's one of the reasons I bought the camera.

If you shot 21 frames at 1/3-stop intervals where the quadrant text is legible, then photographically useful DR is 7 stops.

I shot 35 frames including the 2 I've shown above plus all in-between exposures at 1/3rd stop intervals. That makes the extreme outer limit of the 5D 11.67 stops of DR then.

I haven't examined closely the quadrant text yet, but I expect that the faintest and smallest text will becom illegible very quickly.

[Jonathan Wienke]

If my camera exhibits greater DR in practice, in real world shooting, because it has more pixels, then so be it. Maybe that's one of the reasons I bought the camera.

But that's the problem; it wouldn't. Consider this "thought experiment" for a moment. Imagine you conduct my test with a prime lens on your camera and frame each shot so that the square is 100 pixels wide. Now re-run the test with everything the same except you move the camera closer, so that the square is 500 pixels wide. Are you going to get exactly the same results? No. Did your camera magically increase its DR just because you moved it closer to the subject? Absolutely not. In order for the test results to be consistent and meaningful, the number of pixels resolving the target must be standardized. Varying the pixel count on-target will skew the test and make it less relevant to real-world results.

In that case the quality of the print in conjunction with the quality of the lens will affect the result significantly.

Lens quality affects everything, including real-world image quality; for best results, use the best lens you can. Regarding the quality of the test target, printing on decent matte paper with a good quality, properly profiled printer using Absolute Colorimetric rendering and the best quality print mode will give you consistency within the accuracy of the profile. I designed the target image to avoid excessively saturated colors so that gamut issues shouldn't be a problem. If you shoot so that the white square is 100 pixels across, the dithering patterns of the printer will be below the resolution of the sensor unless you have a really cheap printer.

[Ray]

Consider this "thought experiment" for a moment. Imagine you conduct my test with a prime lens on your camera and frame each shot so that the square is 100 pixels wide. Now re-run the test with everything the same except you move the camera closer, so that the square is 500 pixels wide. Are you going to get exactly the same results?

Yes, of course not. I understand that. I'm not recommending the square be any specified number of pixels. I'm recommending the width of the target fit the width of the sensor. In other words, test the DR of the camera, not one of its pixels.

Regarding the quality of the test target, printing on decent matte paper with a good quality, properly profiled printer using Absolute Colorimetric rendering and the best quality print mode will give you consistency within the accuracy of the profile. I designed the target image to avoid excessively saturated colors so that gamut issues shouldn't be a problem. If you shoot so that the white square is 100 pixels across, the dithering patterns of the printer will be below the resolution of the sensor unless you have a really cheap printer.

I have the cheapest printer that Epson sell; the 4-ink C90. It's not properly profiled, I'm using non-Epson paper and I'm working from a poorly calibrated laptop. Those are my current circumstances, which is why I haven't spent too much time examining the intermediate shots.

Panopeeper offered to examine my RAW files to see if the 2-second exposure is clipped or not, so courtesy YouSendIt the links are below for anyone who wants to have a look.

http://www.yousendit.com/download/www/YVJZc2ZFMVhVbS9IRGc9PQ

http://www.yousendit.com/download/www/YVJaZFhuT2JFd2ZIRGc9PQ

[Jonathan Wienke]

Yes, of course not. I understand that. I'm not recommending the square be any specified number of pixels. I'm recommending the width of the target fit the width of the sensor. In other words, test the DR of the camera, not one of its pixels.

And that's exactly what you're NOT doing, for the reasons I brought up in my previous post. Changing the number of pixels used to resolve the text changes the legibility threshold and invalidates the consistency of results. Or do you actually believe that chopping a sensor in half (if you could do so without ruining it and still read out the remaining pixels) would reduce the DR of the remaining pixels? Or that adding additional identical sensors to a camera will increase its DR? That's preposterous.

Velvia's DR is the same whether it is loaded into a 35mm camera or an 8x10. By your logic, the 8x10 would have more DR than the 35mm, even if loaded with the same film stock.

[John Sheehy]

Panopeeper offered to examine my RAW files to see if the 2-second exposure is clipped or not, so courtesy YouSendIt the links are below for anyone who wants to have a look.
[a href=\"index.php?act=findpost&pid=160878\"][{POST_SNAPBACK}][/a]

The 2 second:
The white areas are all clipped in the blue and green channels.  This means that they are either barely clipped in true "daylight", or are very clipped in a bluish light source (window from blue sky, or a so-called "daylight" bulb, which is usually bluer than real daylight).  The red only clips in specular highlights in the lower right, just to the right of the letter "g" in "Target".

Mean: 1758.4         Median: 2019
Sigma: 1176.2
Maxi.: 3692.0       Mini.: 186.0

Clipping is at 3692, obviously.

A minimum of 186 says that in the weakest channel (red), the blackest thing in that image is only log((3692-128)/(186-128))/log(2) = 5.9 stops below saturation, and that is where the negative noise goes the deepest, so the mean is actually slightly higher than that.


The 0.001 second:
Nothings clipped (but that goes without saying).

Mean: 128.7         Median: 129
Sigma: 2.0
Maxi.: 158.0       Mini.: 117.0
 
ISO 100 is not where you want to do this thing.  I don't remember the exact figure for the 5D, but banding noise is much higher at ISO 100, in electrons; something like 20x as high as ISO 1600.  (Later Edit: Sorry, that figure is not right.  for the 20D it is about 3x as high for horizontal, and 12x for vertical; the 5D is similar, IIRC). 1D banding noise does not decrease at the rate of 2D banding noise with small viewing or downsampling; it is a very horrible noise to deal with.  With pure shot noise, I think you'd be able to read the larger texts in the triangles.

[Ray]

Changing the number of pixels used to resolve the text changes the legibility threshold and invalidates the consistency of results.  Or do you actually believe that chopping a sensor in half (if you could do so without ruining it and still read out the remaining pixels) would reduce the DR of the remaining pixels?

Chopping a sensor in half would increase the total noise in an image of a given size, if I've understood the significance of dpreview's noise comparisons between the D60 and the 1Ds.

I'm simply using that analogy as it may apply to DR. I know DR is not the same as noise, but I would have thought that any reduction in noise either at the pixel level or at the 'total image' level has the effect of increasing DR.

In other words, if DR is equal at the pixel level in two different cameras, the camera with the greatest number of pixels will be capable of delivering an image of greater dynamic range. Is this not true? I could be wrong   .

Velvia's DR is the same whether it is loaded into a 35mm camera or an 8x10. By your logic, the 8x10 would have more DR than the 35mm, even if loaded with the same film stock.

The same per unit area, yes. But not the same per unit picture of same FoV. Medium and large format have always been known for their smoother tonality (compared to 35mm) which I suspect also translates to a greater dynamic range in the sense that more detail is there in both highlights and shadows (as well as transitions), simply because there are more film grains available to describe such detail.

[Panopeeper]

Ray,

here are the raw histograms. The 2s shot is good for this purpose, but the 1/1250s is far out of the range. The pixel values are between 119 and 140 (apart from a few stray pixels), while the values of masked pixels, which are used to determine the level of "black current" is between 120 and 130. In other words, much of the image must be indistinguishable from one shot with the lens cap on (35% of the reds and 25% of the greens and blues are under 128).

[Ray]

The 2 second:
The white areas are all clipped in the blue and green channels.  This means that they are either barely clipped in true "daylight", or are very clipped in a bluish light source (window from blue sky, or a so-called "daylight" bulb, which is usually bluer than real daylight).  The red only clips in specular highlights in the lower right, just to the right of the letter "g" in "Target".

John,
That's interesting. The target was illuminated by the natural light of a blue sky through the window. I suppose ACR has done a fair job in reconstructing the clipped channels. What would you say, 2/3rds of a stop overexposed? That brings the extreme DR limit of my 5D down to only 11 stops. Damn!  

ISO 100 is not where you want to do this thing.  I don't remember the exact figure for the 5D, but banding noise is much higher at ISO 100, in electrons; something like 20x as high as ISO 1600.  1D banding noise does not decrease at the rate of 2D banding noise with small viewing or downsampling; it is a very horrible noise to deal with.  With pure shot noise, I think you'd be able to read the larger texts in the triangles.

Since this is supposed to be a DR test, it's difficult to understand how I would get a better result at a higher ISO. Is DR likely to be better at say ISO 200?

[Ray]

Ray,

here are the raw histograms. The 2s shot is good for this purpose, but the 1/1250s is far out of the range. The pixel values are between 119 and 140 (apart from a few stray pixels), while the values of masked pixels, which are used to determine the level of "black current" is between 120 and 130. In other words, much of the image must be indistinguishable from one shot with the lens cap on (35% of the reds and 25% of the greens and blues are under 128).
[a href=\"index.php?act=findpost&pid=160904\"][{POST_SNAPBACK}][/a]

The 2s shot is good for this purpose,

Panopeeper,
I notice that both the green and blue channels in your histogram show a cliff edge on the far right. Is this the clipping that John Sheehy is referring to?

In other words, much of the image must be indistinguishable from one shot with the lens cap on (35% of the reds and 25% of the greens and blues are under 128).

But some very relevant detail there, nevertheless, which you definitely wouldn't get with the lens cap on. There's no doubt, for example, that the subject is a Dynamice Range Test Chart because we can clearly read that it is.  

[BJL]

Ray and Jonathan seem to be advocating two different forms of comparison (relevant to both noise levels and DR I think) that might each have their place.

One I call "per pixel"; the other "per print" or "per image".

Jonathan's approach is "per pixel" assessment is looking at an equal number of pixels, relevant for example if one plans to print at equal PPI, so getting a larger image from a sensor with more photo-sites. After all, you might well have paid for a camera with more pixels to get sharper and/or bigger prints, not so that you could down-sample or otherwise reduce resolution in NR processing to get acceptable noise levels and DR.

Ray's approach is "per image", comparisons relevant to making prints of the same size from different cameras and viewing from the same distance (or otherwise viewing at equal apparent image size) so that extra pixels can be used to lower the per pixel noise floor and thus increase DR, perhaps simply by the dithering effect of printing at a higher PPI, or perhaps with some spatial averaging that roughly equalizes total resolution.

[Panopeeper]

I notice that both the green and blue channels in your histogram show a cliff edge on the far right. Is this the clipping that John Sheehy is referring to?

Right; that's in the 2 second shot. The saturation level at ISO 100 is 3692 (i.e. not the entire 12-bit range is used). Note, that the saturation level may change with ISO and even with the individual copy, but the latter is really negligable.

But some very relevant detail there, nevertheless, which you definitely wouldn't get with the lens cap on

Yes, of course, as many of the pixels are over the black level. However, the noise is inevitably very high with so many pixels under the black level. I guess you made other shots with higher exposures, those should be exemined. The point would be, that some of the texts are clean, while others are too noisy.

Anyway, I still favour the step wedge, like in DPReview's tests, but in pure raw format.

[John Sheehy]

Since this is supposed to be a DR test, it's difficult to understand how I would get a better result at a higher ISO. Is DR likely to be better at say ISO 200?
[a href=\"index.php?act=findpost&pid=160905\"][{POST_SNAPBACK}][/a]

No, I was merely pointing out how bad banding (especially vertical) is in ISO 100 under-exposures.  BTW, there was a mistake in my post, look at it again with the correction.