DXOMark: difference in SNR and Tonal Range?

[feppe]

I'm considering upgrading my Canon 450D to 550D. The 550D has some possibly useful features, such as much higher res LCD, better (on paper) AF, and video. But it also has half a "stop" higher resolution (12MP vs 18MP), and does ISO up to 12,800 (although poorly).

So I went to DXOMark for comparison*. The description of SNR and Tonal Range is a bit vague and I'd appreciate if someone could explain that in plain terms.

SNR is apparently measured from 18% gray, while Tonal Range is "an integration of the signal-to-noise ratio (SNR) over the dynamic range of the sensor." Does that mean Tonal Range measures SNR, but done at several different gray levels, and averaged to get to a final score?

SNR and Tonal Range are practically identical for the 450D and 550D, but there's roughly 3/4 of a stop improvement on DR. So the differences are minimal, but the other features just might be worth an upgrade.

* I included the Olympus E-PL1 for comparison as I bought one recently. It seems to perform very well against the bigger sensors, and is significantly poorer only in DR.

[ejmartin]

I'm considering upgrading my Canon 450D to 550D. The 550D has some possibly useful features, such as much higher res LCD, better (on paper) AF, and video. But it also has half a "stop" higher resolution (12MP vs 18MP), and does ISO up to 12,800 (although poorly).

So I went to DXOMark for comparison*. The description of SNR and Tonal Range is a bit vague and I'd appreciate if someone could explain that in plain terms.

SNR is apparently measured from 18% gray, while Tonal Range is "an integration of the signal-to-noise ratio (SNR) over the dynamic range of the sensor." Does that mean Tonal Range measures SNR, but done at several different gray levels, and averaged to get to a final score?

SNR and Tonal Range are practically identical for the 450D and 550D, but there's roughly 3/4 of a stop improvement on DR. So the differences are minimal, but the other features just might be worth an upgrade.

* I included the Olympus E-PL1 for comparison as I bought one recently. It seems to perform very well against the bigger sensors, and is significantly poorer only in DR.

The SNR plot is indeed measured at 18% of raw saturation.  If you go to the individual camera's page rather than the camera comparison tool, you can find an extra set of tabs, one of which displays SNR throughout the tonal range, much more information as I explain here.

DxO's Tonal Range is a measure of how many tones can be distinguished within the set of raw levels available.  Two tones cannot be distinguished if their difference is less than the noise std dev.  If the noise were constant, the number of distinguishable tonal levels would be the range of available raw levels divided by the noise std dev; but since the noise varies with tonality due to photon noise (it grows with the illumination level, as the square root) one has to add up (integrate, in math terms) over the range of raw levels the fractional amount of a distinguishable tonal level that each raw level constitutes.  The total is what DxO calls Tonal Range; a higher value means more tones are distinguishable.  They present the result in "bits", ie two to the power of the number of bits distinguishable levels.  Similar to the dynamic range, but accounting for the light-gathering capacity as well as the noise.

[feppe]

The SNR plot is indeed measured at 18% of raw saturation.  If you go to the individual camera's page rather than the camera comparison tool, you can find an extra set of tabs, one of which displays SNR throughout the tonal range, much more information as I explain here.

DxO's Tonal Range is a measure of how many tones can be distinguished within the set of raw levels available.  Two tones cannot be distinguished if their difference is less than the noise std dev.  If the noise were constant, the number of distinguishable tonal levels would be the range of available raw levels divided by the noise std dev; but since the noise varies with tonality due to photon noise (it grows with the illumination level, as the square root) one has to add up (integrate, in math terms) over the range of raw levels the fractional amount of a distinguishable tonal level that each raw level constitutes.  The total is what DxO calls Tonal Range; a higher value means more tones are distinguishable.  They present the result in "bits", ie two to the power of the number of bits distinguishable levels.  Similar to the dynamic range, but accounting for the light-gathering capacity as well as the noise.

Comparing the full SNR charts it seems that the 450D has better SNR at ISO100 (90+% of my shooting), and even the E-PL1 has (marginally) better SNR at base ISO of 200 than the 550D!

Yet another example of chasing MP at the expense of SNR, and only slight improvement in DR?

[bjanes]

The SNR plot is indeed measured at 18% of raw saturation.  If you go to the individual camera's page rather than the camera comparison tool, you can find an extra set of tabs, one of which displays SNR throughout the tonal range, much more information as I explain here.

DxO's Tonal Range is a measure of how many tones can be distinguished within the set of raw levels available.  Two tones cannot be distinguished if their difference is less than the noise std dev.  If the noise were constant, the number of distinguishable tonal levels would be the range of available raw levels divided by the noise std dev; but since the noise varies with tonality due to photon noise (it grows with the illumination level, as the square root) one has to add up (integrate, in math terms) over the range of raw levels the fractional amount of a distinguishable tonal level that each raw level constitutes.  The total is what DxO calls Tonal Range; a higher value means more tones are distinguishable.  They present the result in "bits", ie two to the power of the number of bits distinguishable levels.  Similar to the dynamic range, but accounting for the light-gathering capacity as well as the noise.

To illustrate Emil's explanation, I took a photograph of a Stouffer wedge with the Nikon D3 at ISO 1600 using a 14 bit NEF and split out the green 1 channel with Iris for analysis. The density steps on this target are in increments of 0.1 optical density (0.3 EV). The steps on the left are clearly distinguishable, but those on the right become indistinct due to noise and display limitations.

[attachment=21224:07a.png]

One can accentuate the differences in Iris by use of the threshold command. Steps 24 and 25 are clearly distinguishable, but as the signal:noise decreases towards the shadows, steps 31 and 32 are indistinguishable. One can look at the means and standard deviations with ImageJ. Using the means and standard deviations, step 31 is 27.1 ± 14.8 and step 32 is 21.8 ± 14.2.

[attachment=21225:G1_Stouffer.png]

[Guillermo Luijk]

Comparing the full SNR charts it seems that the 450D has better SNR at ISO100 (90+% of my shooting), and even the E-PL1 has (marginally) better SNR at base ISO of 200 than the 550D!

Yet another example of chasing MP at the expense of SNR, and only slight improvement in DR?

There is a statistical SNR improvement once the image is downsized to a given final size, and this benefits higher resolution cameras. DxO Mark takes this effect into account in their 'Print' plots, contrary to 'Screen' plots that refer to 100% screen comparisions.

The 550D has fairly less noise than the 450D and specially the E-PL1:



Comparing the SNR plots without taking resolution into account isn't a fair noise comparision between cameras because final images are not compared at 100% zoom, but printed or projected to a given final size.

Regards

[feppe]

Comparing the SNR plots without taking resolution into account isn't a fair noise comparision between cameras because final images are not compared at 100% zoom, but printed or projected to a given final size.

I'm aware of this and I was comparing the "print" tests which are essentially identical between 450D and 550D on SNR and Tonal Range. DR seems to be the only DXO metric where 550D is better (in both screen and print modes).

[Guillermo Luijk]

I'm aware of this and I was comparing the "print" tests which are essentially identical between 450D and 550D on SNR and Tonal Range. DR seems to be the only DXO metric where 550D is better (in both screen and print modes).

I think you are calling 'SNR' to 'SNR at 18%', which is just a particular case of SNR and not even the most interesting:
- SNR at 18% is SNR at well exposed areas (basically at about 2.5EV from saturation, which is a high degree of exposure).
- On the other side DR is not a different metric from SNR 18%, DR is about the same thing: noise. But DR measures SNR in the deep shadows instead of in well exposed areas. This is a much more interesting place to compare cameras since it is where cameras really face a challenge.

When going to the deep shadows, the 550D performs quite better than the 450D, and much better than the E-PL1 (this is common to all Panasonic sensors but that of the GH1, that seems to be quite different). If we were to choose, the SNR 18% would probably the least interesting benchmark since all cameras have a fairly good SNR at 2.5EV from saturation (the contrary would be the same as addmitting 'hey, this camera is noisy even at ISO100 and in good light conditions'), and DR or the SNR curves (not SNR 18%, I mean the complete SNR curve found in the last tab) would be a better place for comparisions.

Regards

[feppe]

I think you are calling 'SNR' to 'SNR at 18%', which is just a particular case of SNR and not even the most interesting:
- SNR at 18% is SNR at well exposed areas (basically at about 2.5EV from saturation, which is a high degree of exposure).
- On the other side DR is not a different metric from SNR 18%, DR is about the same thing: noise. But DR measures SNR in the deep shadows instead of in well exposed areas. This is a much more interesting place to compare cameras since it is where cameras really face a challenge.

When going to the deep shadows, the 550D performs quite better than the 450D, and much better than the E-PL1 (this is common to all Panasonic sensors but that of the GH1, that seems to be quite different). If we were to choose, the SNR 18% would probably the least interesting benchmark since all cameras have a fairly good SNR at 2.5EV from saturation (the contrary would be the same as addmitting 'hey, this camera is noisy even at ISO100 and in good light conditions'), and DR or the SNR curves (not SNR 18%, I mean the complete SNR curve found in the last tab) would be a better place for comparisions.

Regards

Thanks for the good explanation, makes perfect sense.

Correct, I meant 18% gray SNR in my previous post, but I was comparing against the full tonal range graphs in the comment before that, and 450D has better SNR at most gray levels.

450D seems to be better at ISO 100 starting from 1.5% gray, while 550D is indeed better at the very deep blacks. But does that matter? Does 1.5% gray equal to 0-4 on a 0-255 scale? If it does, such advantage is of purely academic interest, as the vast majority of grays (1.5% - 100%) have better SNR with the 450D. I can't even distinguish 0-4 gray levels from prints with my printer (might have been 3 or 5 where the difference is visible, it's been a while since I did the test).

While much of my photography is in low light, I'm questioning the benefits of having better DR only in the very deepest blacks, while all brighter tones have better SNR with the 450D. Even E-PL1 has better SNR than 550D at all gray levels above 8% at ISO 200.

[ErikKaffehr]

Hi,

DxO has to options, "screen" and "print". Screen corresponds to actual pixels and print to what you would actually see in prints. The "print" option may be more relevant for most purposes.

BR
Erik

Comparing the full SNR charts it seems that the 450D has better SNR at ISO100 (90+% of my shooting), and even the E-PL1 has (marginally) better SNR at base ISO of 200 than the 550D!

Yet another example of chasing MP at the expense of SNR, and only slight improvement in DR?

[feppe]

Hi,

DxO has to options, "screen" and "print". Screen corresponds to actual pixels and print to what you would actually see in prints. The "print" option may be more relevant for most purposes.

BR
Erik

As I mentioned earlier, I compared the print option. The full tonal range charts don't specify whether they are print or screen.

[Guillermo Luijk]

Correct, I meant 18% gray SNR in my previous post, but I was comparing against the full tonal range graphs in the comment before that, and 450D has better SNR at most gray levels.

450D seems to be better at ISO 100 starting from 1.5% gray, while 550D is indeed better at the very deep blacks. But does that matter? Does 1.5% gray equal to 0-4 on a 0-255 scale? If it does, such advantage is of purely academic interest, as the vast majority of grays (1.5% - 100%) have better SNR with the 450D. I can't even distinguish 0-4 gray levels from prints with my printer (might have been 3 or 5 where the difference is visible, it's been a while since I did the test).

While much of my photography is in low light, I'm questioning the benefits of having better DR only in the very deepest blacks, while all brighter tones have better SNR with the 450D. Even E-PL1 has better SNR than 550D at all gray levels above 8% at ISO 200.

Nope, think that the X-axis scale in the SNR plots is the linear response of the sensor. So to convert the 1,5% to the more friendly 0-255 scale we see in Photoshop, the gamma correction must be applied.

So think you develop your RAW file in ACR with _all_ settings to 0 (a neutral RAW development*) and in the typical Adobe RGB (gamma 2,2). An exposure value of 1.5% under those conditions would fall in:

0,015^(1/2,2) * 255 = 38

That means the 550D beats the 450D in PS levels under 38, i.e. the shadows where noise is usually a problem, not in the well exposed areas.

But we can go on: the truth is that rarely we do neutral RAW developments*. In fact usually the Bright curve increases all levels. So it is even more correct to think that the 1,5% of the sensor is likely to become a level above 38 on your Photoshop daily work.

Looking at the SNR curves, and always assuming they are correct, I find the 550D a more balanced camera: it is a bit worse (or we'd rather say not so good) than the 450D in the well exposed areas and highlights, and it is better than the 450D in the shadows, where noise is really challenging.



I would even say that the intersection point of SNR for both cameras at ISO100 is around -1,7 in the X-axis log 10 scale, that translates into 10^(-1,7)=2%, which corresponds to a 0-255 value of 0,02^(1/2,2) * 255 = 43

Regarding performance in low light conditions, i.e. when ISO needs to be pushed, the same applies: you can expect some extra noise on the 550D in the well exposed areas, but less noise in the underexposed areas, where noise is really a problem.

And _all_ the above refers to the SNR plots, that are calculated on a per-pixel basis (the 'Screen' criteria). This means the improvement in SNR that the 550D should experience over the 450D because of its higher resolution (18 vs 12 Mpx) was not yet taken into account.

Regards

[feppe]

Nope, think that the X-axis scale in the SNR plots is the linear response of the sensor. So to convert the 1,5% to the more friendly 0-255 scale we see in Photoshop, the gamma correction must be applied.

So think you develop your RAW file in ACR with _all_ settings to 0 (a neutral RAW development*) and in the typical Adobe RGB (gamma 2,2). An exposure value of 1.5% under those conditions would fall in:

0,015^(1/2,2) * 255 = 38

That means the 550D beats the 450D in PS levels under 38, i.e. the shadows where noise is usually a problem, not in the well exposed areas.

But we can go on: the truth is that rarely we do neutral RAW developments*. In fact usually the Bright curve increases all levels. So it is even more correct to think that the 1,5% of the sensor is likely to become a level above 38 on your Photoshop daily work.

Looking at the SNR curves, and always assuming they are correct, I find the 550D a more balanced camera: it is a bit worse (or we'd rather say not so good) than the 450D in the well exposed areas and highlights, and it is better than the 450D in the shadows, where noise is really challenging.

 Thanks again, that was very good explanation. Gray levels below 40 or so are challenging indeed for the 450D, so improvement there would be most welcome, especially for low light photography.

Looks like the 550D might be a worthwhile upgrade - but perhaps I'll wait for the follow-up to 7D or 50D to see what Canon has up their sleeve.

[Ray]

You should work for DxO to make sense of their graphs for mortals  Thanks again, that was very good explanation. Gray levels below 40 or so are challenging indeed for the 450D, so improvement there would be most welcome, especially for low light photography.

Looks like the 550D might be a worthwhile upgrade - but perhaps I'll wait for the follow-up to 7D or 50D to see what Canon has up their sleeve.

C'mon! You know very well there's always something better around the corner   .

In my opinion, the major advantage of the 550D is its video capability which seems to be on a par with that of the 7D.

If you really want a worthwhile jump in DR, SNR and tonality I'm afraid you'll have to upgrade to full frame 35mm.

[feppe]

C'mon! You know very well there's always something better around the corner   .

In my opinion, the major advantage of the 550D is its video capability which seems to be on a par with that of the 7D.

If you really want a worthwhile jump in DR, SNR and tonality I'm afraid you'll have to upgrade to full frame 35mm.

Ain't that the truth about new cameras coming out so often. I've managed to hold off my upgrades, 450D being only the second dSLR I've owned after 30D.

The xxx series have all the features I need, and DxO has shown the sensors are only slightly poorer than the xx series. I paid for a 5DII over a year ago, getting a refund now (long story), but I'm not going to buy this generation FF camera if I ever make the move. My main complaint with 30D and 5DII is the size and weight, but since I now have the E-PL1 for light travel, I could use a larger dSLR for other shooting and in studio.