Canon 5D dynamic range tests

[John Sheehy]

And it's equally silly to rate a high-pixel-count digicam better than a DSLR simply because it can throw more pixels at the subject. The larger the letters are, the easier it is to pick them out of the shadow noise. If you fill the frame with the chart, a camera with more megapixels will have an unfair advantage on that basis alone, and cameras with greater pixel counts will get a higher DR rating than they are actually entitled to.
[a href=\"index.php?act=findpost&pid=160829\"][{POST_SNAPBACK}][/a]

Entitled to for what?  Resolution and individual pixel quality work together towards image noise and DR.  Can you show a case where a line of sensors became less able to resolve detail by packing more pixels?

You seem to use the pixel-centric paradigm, popularized by Roger Clark and others, where DR automatically refers to the abilities of a single pixel as a measuring device.  A pixel, just like the cones and rods in our eyes, is an artificial structure designed to bin photon strikes from an area into a single collection of photons.  The real world of photons is one of infinite resolution, and sparse photon strikes near any single point (tremendous shot noise for hypothetic infinitesimal pixels).  It is ultimately pointless to bin twice, once in the medium, and then again by the eye, but technology forces us to, but it should do as little binning as possible.  Ray's ISO 110,000 version of your test chart proves this point.  If you look at just the green channel in the RAW image, you can barely make out the biggest line in the green quadrant, and even more faintly, the second-biggest line.  The next one is more like, "I think I see something there, but maybe not".   Anyway, if you load it into photoshop, at 100% zoom, and stand back 15 feet from the monitor, you can make out the biggest line well enough to see the individual numbers somewhat distinctly.  Come back near the monitor, and it is less distinct, as individual bright pixels obscure what's around them.  Now, simulate bigger pixels by using Photoshop's pixelate|mosaic feature.  Set it to Preview, and start with 2x2, then 3x3, then 4x4, etc.  Instead of getting clearer of noise, but with lower resolution, resolution reduces with no reduction of noise.  The clearest numbers are had at 100% (tiny pixels), viewed from a distance, because your eyes and brain get to do all the work, which they do better with more resolution, despite more noise per pixel.

[John Sheehy]

Panopeeper,
I notice that both the green and blue channels in your histogram show a cliff edge on the far right. Is this the clipping that John Sheehy is referring to?

Yes.  Gabor's histogram program is assuming that the RAW data can go to 4095, but it can only go to 3692 in your camera at ISO 100.

But some very relevant detail there, nevertheless, which you definitely wouldn't get with the lens cap on.

If you make a selection in photoshop, manually, of each area that you know has the same level throughout, and run the "Blur|Average" filter, you will get nice, clean areas distinct from each other in tonal level (unless they're supposed to be the same).  The ability to capture accurate means at extremely low light levels is tremendous, barring the bands (Pun intended).  If done with negative noise footroom, the response of means near black is totally linear from Canons.  Despite the high levels of noise in each pixel, the mean of noise is infinitesimal, and one individual photon over an a large area does affect the average.

There's no doubt, for example, that the subject is a Dynamice Range Test Chart because we can clearly read that it is. 
[a href=\"index.php?act=findpost&pid=160908\"][{POST_SNAPBACK}][/a]

What about mass hallucination?

[Panopeeper]

Yes.  Gabor's histogram program is assuming that the RAW data can go to 4095, but it can only go to 3692 in your camera at ISO 100

The camera specific parameters of the program (compiled, not user-parameters) include the saturation levels per pixelcolor and ISO, as well as the histogram range. If there is a power of 2 very close to the respective saturation level, then I specify that "round" value, otherwise some rounded value close to the top (there are variations between copies, so I rather give a slightly higher value than the top I see).

However, I don't know these values for many cameras. As the matter of fact, my program worked only with DNG until last week, and now I am about implementing the processing of native raw files (messy, messier, messiest, as if there were a competition between manufacturers). Adoba's DNG converter inserted these values, but very unreliably, I use those values only if I don't know anything better.

Anyway, if you know such values, pls tell me, or send me raw files.

Btw, we were discussing the saturation levels in ACR. As far as I see, the Canon 5D is better supported in ACR than the 40D: I saw DNG files (which reflect ACR) with different white level for different shots by 5D, the same ISO.

Thomas Knoll posted once to my related question, that the white level is based on the camera, ISO and image content. This is obviously not always the case, but they make the effort sometimes.

However, there is only a single value for the saturation per image, though the different pixels of the CFA can have different saturation levels. Ok, the difference is usually not large, but principally there should be separate values. As the matter of fact, my solution is not perfect either, because the two greens of a CFA have in some cases different saturation levels, and I can specify only one per color. If and when my program becomes a raw converter, then I will make the effort to evaluate it in runtime.

[Jonathan Wienke]

Ray and Jonathan seem to be advocating two different forms of comparison (relevant to both noise levels and DR I think) that might each have their place.

One I call "per pixel"; the other "per print" or "per image".

I really don't buy that argument. Let's take a look at the behavior of a randomly-selected pixel on a camera sensor over a series of 100 exposures. The number of photons required to saturate that pixel completely at a given ISO setting is going to be fairly consistent, as is the minimum number of photons required to read that pixel and say with confidence that one has a signal above noise level. The minimum and maximum number of photons that pixel can consistently detect is NOT going to change just because the photographer alters the crop of the images AFTER the exposures were made and changes the number of that pixel's brothers that are included in the final image. The decision to crop or not crop has zero effect on the dynamic range or noise level of a given capture.

Stated another way, the number of pixels included in a print has NOTHING to do with the engineering definition of DR (signal-to-noise ratio), and I categorically reject the idea that it should be included when attempting to define the subset of that range that is photographically useful. Otherwise you enter a murky realm where all the definitions change depending on how many of the captured pixels make it into the final print and you can redefine the meaning of "is" however you like.

If you think of image quality as being (pixel quality) * (pixel quantity), then you must first accurately and consistently analyze the behavior of each pixel, and then factor in differences in pixel quantity, or else you cannot meaningfully compare the performance of imaging devices with unequal pixel count.

[Jonathan Wienke]

Entitled to for what?  Resolution and individual pixel quality work together towards image noise and DR.  Can you show a case where a line of sensors became less able to resolve detail by packing more pixels?

I can think of some models of digicam that were replaced by a higher pixel count model and the new model performed worse than the old one overall because noise levels went up so much...

[Ray]

I really don't buy that argument. [a href=\"index.php?act=findpost&pid=160949\"][{POST_SNAPBACK}][/a]

I do, although the advantages of more pixels and greater resolution might be better explained by John Sheehy. At the level of pure physics (ie. excluding the effects of human perception), as far as I can see, any increase in DR would flow from a lower shot noise, not lower in the absolute sense, but lower in terms of S/N.

Lets consider the following thought experiment. (And one that BJL will appreciate, I'm sure   .)

Let's imagine that at the same time the 5mp Olympus E-1 came out, Olympus also introduced a 20mp camera of the same aspect ratio but with a sensor 4x the area (roughly equivalent to the 1Ds3).

Because the technology used was very similar for both cameras, the DR of each individual pixel was identical in each camera. The photodiodes were the same, the on chip processing for each photodiode was the same, the ISO rating was the same and the read noise per pixel was the same.

If these two cameras were tested using your target per your instructions of getting the centre square 100 pixels wide, then both cameras would appear to have the same DR according to you.

Let's examine what would happen in reality when both cameras are used to shoot a scene, artificial target or general view, keeping the FoV the same for both cameras.

I'm sure you would agree, if you were to take any specified crop of that scene, however small, there would be 4x as many pixels in the crop from the larger format camera.

Let's make the crop so small that it is exactly the same size as a single pixel from the 5mp E-1, and let's say that the intensity of light is so great in that pixel-size part of the scene that it fills the well of the E-1 photodiode, producing a full ETTR.

Let's now examine the identical crop from the same scene shot with the larger format 20mp camera. The light that fills the well of one single E-1 photodiode now falls on 4 pixels from the larger camera. Each pixel receives 1/4 the amount of light, but that includes the pixels also that are gathering light from the dark part of the scene.

In order to fill all the wells in both cameras to an equal level, the camera with 4x the number of pixels needs 4x the exposure, 4x the amount of light. Either the scene needs to be 4x brighter or the aperture used in the larger camera needs to be 2 stops wider.

Now, if it were not for shot noise, there might appear to be no dynamic range advantage here, but with 4x the number of photons arriving at the sensor, I would think that shot noise, as a proportion of total signal level (S/N) would be half. I might be wrong here, though.

It might also be the case that other sources of noise completely obscure any improvement in shot noise to the point where such improvement is insignificant.

[Panopeeper]

Let's imagine that at the same time the 5mp Olympus E-1 came out, Olympus also introduced a 20mp camera of the same aspect ratio but with a sensor 4x the area

In other words, the sensor pixel sites (sensels) are the same size.

The light that fills the well of one single E-1 photodiode now falls on 4 pixels from the larger camera. Each pixel receives 1/4 the amount of light

I think you need to rework your thought experiment.

[Ray]

In other words, the sensor pixel sites (sensels) are the same size.
[a href=\"index.php?act=findpost&pid=160958\"][{POST_SNAPBACK}][/a]

Exactly!

I think you need to rework your thought experiment.

Perhaps you can tell me why. The only factor which I see as being dubious is the significance of the photonic shot noise. It's supposed to vary with the square root of the photon count. Increase the amount of light illuminating the entire scene by a factor or 4, then for the entire scene photonic shot noise should be half, although it is the same at the pixel level for both cameras. Is this not so?

[BJL]

The minimum and maximum number of photons that pixel can consistently detect is NOT going to change just because the photographer alters the crop of the images AFTER the exposures were made and changes the number of that pixel's brothers that are included in the final image.[a href=\"index.php?act=findpost&pid=160949\"][{POST_SNAPBACK}][/a]

I was not thinking abut cropping, but processes like down-sampling or "binning" or fancier options like NR processing, so that the final output _pixels_ are produced using data from several _photosites_ The output pixels can end up with higher DR than the photosites signals used to make them.

[Aside on film. The "photosites" of film, silver halide crystals, have only two output states, so individually have very low DR. Dithering the output of billions of these "one-bit photosites" generates the DR and tonal gradations of a print, and no one cares about measuring the DR of those individual photosites.]

A monochrome sensor with 20 million _photosites_ can have output downsampled to 5MP by summing 2x2 blocks of photosite output, and with uncorrelated noise this will give pixels with S/N twice as high as for simple "one pixel per photosite" output.

[Aside on film. The fine tonal gradations of a silver halide print which under a microscope is pure black dots on a pure white background indicate that "visual averaging" of the light reflected off a print can be enough to increase DR in this way. DR also seems to improve with lower degrees of enlargement, such as when making prints of the same size from larger formats. You effectively increase the output DR of prints made with the same emulsion by adding more "photosites" and printing at the same size.]

[Jonathan Wienke]

If these two cameras were tested using your target per your instructions of getting the centre square 100 pixels wide, then both cameras would appear to have the same DR according to you.

Let's examine what would happen in reality when both cameras are used to shoot a scene, artificial target or general view, keeping the FoV the same for both cameras.

I'm sure you would agree, if you were to take any specified crop of that scene, however small, there would be 4x as many pixels in the crop from the larger format camera.

DR also seems to improve with lower degrees of enlargement, such as when making prints of the same size from larger formats. You effectively increase the output DR of prints made with the same emulsion by adding more "photosites" and printing at the same size.

Here's the fallacy in your arguments: any technique you use to give the higher-resolution camera a DR advantage over the other camera (or the larger format of film) will negate the resolution advantage by a corresponding amount, and require additional exposure to boot. Yes, you can bin 4 pixels together to reduce noise levels, but then you lose the resolution gain, and still have to increase exposure by a factor of 4. I could probably get better DR out of a typical digicam than that of any MFDB pixel if I binned the whole sensor output down to a single pixel, but that would be a completely impractical and retarded thing to do. There is no free lunch here. The whole point of the exercise is to find out what kind of DR performance you can get without giving up resolution. You aren't gaining anything if you increase pixel quality by decreasing pixel quantity.

[Ray]

... and still have to increase exposure by a factor of 4. I could probably get better DR out of a typical digicam than that of any MFDB pixel if I binned the whole sensor output down to a single pixel, but that would be completely impractical and retarded thing to do. There is no free lunch here.[a href=\"index.php?act=findpost&pid=160964\"][{POST_SNAPBACK}][/a]

There's no free lunch. The consequence of using a larger format is always a requirement for greater exposure. The consequence of a greater exposure is a better signal-to-shot-noise, with my method, but not with your method.

As I understand it, the binning that takes place in relation to the sensor with the higher pixel count is in the viewing of equal size images, not at the exposure stage.

[Panopeeper]

Perhaps you can tell me why

If the sensels are of equal size, then why would the same amount of light be distributed among four pixels?

[Jonathan Wienke]

There's no free lunch. The consequence of using a larger format is always a requirement for greater exposure. The consequence of a greater exposure is a better signal-to-shot-noise, with my method, but not with your method.

And you only get that increase in DR with your method at the cost of resolution. If you're measuring DR with a test that effectively throws away 75% or more of actual resolution, the results are bullshit. If I have a 16MP camera and want to test DR, I really don't care what the DR is if I reduce resolution to 4MP or 1MP or 1 pixel. Remember that the whole point of the test is to evaluate real-world, photographically meaningful DR, and binning an image down to 1/4 of its original pixel count is not real-world photographic practice.

[Ray]

If the sensels are of equal size, then why would the same amount of light be distributed among four pixels?
[a href=\"index.php?act=findpost&pid=160970\"][{POST_SNAPBACK}][/a]

For images of equal FoV at equal exposure, the same amount of light would be distributed amongst the four pixels of the larger format.

With Jonathan's method, you'd be using the larger format (in my example) at the same exposure and same physical aperture, same focal length lens and same distance in order to get the centre square 100 pixels wide.

That's not how we photographers use cameras in the real world for everyday shooting, is it?  

ps. I just amended the above. Even I sometimes get confused   .

[Panopeeper]

For images of equal FoV at equal exposure, the same amount of light would be distributed amongst the four pixels of the larger format

1. Equal FoV means equal angle of view.

2. Four times the sensor size means twice the width and length.

3. The same angle requires the lens to be twice the distance from the larger sensor, i.e. the focal length has to be twice as long.

4. Equal exposure means equal product of the shutter time and aperture size (the latter relatively to the focal length).

5. The same aperture number with the longer focal length means larger aperture diameter, more light.

That was it.

[Ray]

1. Equal FoV means equal angle of view.

2. Four times the sensor size means twice the width and length.

3. The same angle requires the lens to be twice the distance from the larger sensor, i.e. the focal length has to be twice as long.

4. Equal exposure means equal product of the shutter time and aperture size (the latter relatively to the focal length).

5. The same aperture number with the longer focal length means larger aperture diameter, more light.

That was it.
[a href=\"index.php?act=findpost&pid=160975\"][{POST_SNAPBACK}][/a]

That was what? You've just given the reasons why larger formats require a greater absolute exposure. In my example, you have 4 pixels in the larger format in place of one pixel in the smaller format. You therefore need 4x the amount of light for a proper exposure of the larger format, with equal FoV.

Using Jonathan's method is like using a Canon 20D in exactly the same way as a 1Ds3, same lens, same f stop, same exposure, same distance to subject.

What's the point of that? All you demonstrate is that a 1Ds3 sensor cropped to the same size as a 20D sensor is essentially a 20D camera.

Editing again: 'What's the point of that?' is of course rhetorical. There may be a point. Since the 1Ds3 and 20D have approximately the same pixel density, one might be curious as to whether Canon had manged to increase the DR of the individual pixels in the 1Ds3, compared with invidual pixels in the 20D. Jonathan's method would be suitable for that purpose.

[Jonathan Wienke]

Editing again: 'What's the point of that?' is of course rhetorical. There may be a point. Since the 1Ds3 and 20D have approximately the same pixel density, one might be curious as to whether Canon had manged to increase the DR of the individual pixels in the 1Ds3, compared with invidual pixels in the 20D. Jonathan's method would be suitable for that purpose.

And my method is also suitable for discovering how well the 1Ds-MkIII will perform without dumbing it down to the resolution of the 20D. When you do the test on two cameras with unequal resolution with the test chart full-frame, the results from the higher-resolution are only valid if you reduce the resolution of the high-res camera to match the low-res camera. If I'm going to pay the price difference between a 20D and 1Ds-MkIII to get the extra resolution, then if I test the DR, I want to measure DR at 1Ds-MkIII resolution, not 20D resolution. Otherwise, there's no reason not to bin the resolution down to 1 pixel and claim a DR of 25 stops and lose the whole real-world-ness of the measurement.

[Ray]

And my method is also suitable for discovering how well the 1Ds-MkIII will perform without dumbing it down to the resolution of the 20D. When you do the test on two cameras with unequal resolution with the test chart full-frame, the results from the higher-resolution are only valid if you reduce the resolution of the high-res camera to match the low-res camera. If I'm going to pay the price difference between a 20D and 1Ds-MkIII to get the extra resolution, then if I test the DR, I want to measure DR at 1Ds-MkIII resolution, not 20D resolution. Otherwise, there's no reason not to bin the resolution down to 1 pixel and claim a DR of 25 stops and lose the whole real-world-ness of the measurement.
[a href=\"index.php?act=findpost&pid=160981\"][{POST_SNAPBACK}][/a]

And my method is also suitable for discovering how well the 1Ds-MkIII will perform without dumbing it down to the resolution of the 20D.

Jonathan,
Here's where I have difficulty in following you. When we make prints, or just view images on the monitor, we're usually rezzing up or rezzing down the image to a suitable size for our immediate purpose. If the issue is noise, which affects DR to some extent, then rezzing down the 1Ds3 image to the 20D file size should produce a slightly cleaner result, if we assume that noise at the pixel level is the same for both cameras.

On the other hand, if we rezz up the 20D file to the same size as the 1Ds3 file, then we are also amplifying the noise in the 20D image. The 1Ds3 image will then not only look cleaner, have slightly more DR (perhaps?) but also higher resolution.

Either way, dumming down or smartening up, the 1Ds3 should retain some advantage regarding noise and DR without sacrificing resolution to the 20D.

However, I'm prepared to accept that the differences regarding DR and noise might be very marginal. This would be an interesting experiment for someone who owns both a 1Ds3 and a 20D or 30D to carry out. Perhaps Mark Segal would like to try this, if he has a 20D.  

He could use your method, first to determine if the DR of the 1Ds3 pixel is greater than that of the 20D pixel. Who knows? It might have a 1/4 of a stop greater DR. The photodiodes on the 1Ds3 sensor might be slightly larger even though the pixel pitch is the same as that of the 20D.

Having determined any difference in DR between the two cameras at the pixel level, we could then do a subjective assessment of DR and noise using images of scenes of high brightness range and equal FoV, examining various print sizes, or image sizes on monitor, but always comparing equal sizes whether print or image.

Anyone who's not confident in carrying out this test could send their 1Ds3 to me. I'll do it   .

[Jonathan Wienke]

Jonathan,
Here's where I have difficulty in following you. When we make prints, or just view images on the monitor, we're usually rezzing up or rezzing down the image to a suitable size for our immediate purpose.

That is entirely irrelevant. When you make a 600-pixel web JPEG from a 5D image, do you keep a copy of the original resolution fille? Of course! And if you need to upsize for a really large print, do you downsize to 2MP first? Of course not. Why? Because you want to keep as many original pixels intact as possible, especially when making large prints and actual original pixels are in short supply anyway. If you want to make a 40x60cm print, the fact that downsampling your image file 4:1 will significantly reduce noise is completely irrelevant to you, because doing so will reduce image detail unacceptably. So the DR you might achieve by doing so is also completely irrelevant.

The reason I chose the 100-pixel size for the center white square is because the small text at that size has some fairly fine detail that will get lost in the noise when the noise level starts getting high enough to be visually objectionable, like the points on the asterisks in particular. So framing to that pixel size makes the legibility test fairly sensitive to noise.

[attachment=4278:attachment][attachment=4279:attachment]

What you're looking at is the chart sized so the white square is 100 pixels wide. By adding only 2% Gaussian noise, the smallest text has become much harder to read, and the points on the asterisks are pretty much obliterated.

Now if we re-frame the target so that the white square is 200 pixels wide, we can add 4% noise before the small text becomes similarly unreadable.

[attachment=4281:attachment]

Congratulations, we just magically increased DR by an entire stop! But at what cost? Nothing major really, just tossing out 3/4 of the resolution of the camera, turning 12 megapixels into 3. Here's the 100-pixel version again, with 4% noise added:

[attachment=4282:attachment]

As you can see, the small text is pretty much obliterated. You can't even find some of the asterisks any more, let alone make out their points. Resolution has been severely compromised by the noise. Each stop you inflate the DR measurement comes at the cost of halving linear resolution, or throwing away 3/4 of your pixel count. And that's not an acceptable trade-off in the vast majority of real-world photography.

[Jonathan Wienke]

I guess the short version of all of this is this: how much dynamic range can a camera capture at its full resolution? Below a certain threshold, noise does not significantly affect resolution; the limiting factors are things like AA filter strength, lens quality, and pixel count. But once noise reaches that threshold, it begins to become the primary limiter of resolution, and every stop noise increases above that threshold halves linear resolution. What I'm trying to determine with my testing methodology is usable DR at the threshold where the noise level is just barely starting to limit overall resolution.