Pages: [1] 2   Go Down

### AuthorTopic: Photographic DR from DXO Data  (Read 10000 times)

#### bjanes

• Sr. Member
• Offline
• Posts: 3215
##### Photographic DR from DXO Data
« on: January 27, 2012, 05:58:09 PM »

The DXO dynamic range tests are for engineering DR, while photographers are more interested in the dynamic range more relevant for photographic use. Emil Martinec has authored a post explaining the two criteria and how one can determine photographic dynamic range from the DXO data. In attempting to apply Emil's method, I found that the main difficulty was reading the log gray scale values with sufficient accuracy to obtain accurate results. Wikipedia has a section on Estimating values in a diagram with logarithmic scale. The method is fairly straightforward.

One can read the distances from the graphs with the ruler in Photoshop, and calculations are facilitated by making a simple spreadsheet in Excel. An example is calculating these values for various SNRs for the P65+ at base ISO. One downloads the DXO graph and measures the x-distance from the next lower decade grid line of the intersection of the SNR curve with the y-value of a given SNR. One stop is 6dB.

One then enters the x-distance into the spreadsheet to determine the interpolated x-value. The screen DR is calculated. Normalization is done as per the DXO protocol: Normalized DR = screen DR + log base 2 (sqrt[m1/m0]), where m1 is the megapixel count of the test camera and m0 is the megapixel count of the reference camera (8MP for the DX0 calculation). Emil uses a slightly different normalization.

The results of the exercise are shown:

One can compare with the Nikon D3x:

According to the DXO method of normalization, the photographic DR of the Nikon is greater at all noise floors, whereas Emil's method of normalization shows a slight advantage for the Phase One. One should note that SNRs at very low illumination are more affected by read noise, while those at higher illumination are more dependent on shot noise. Current MFDBs have relatively high read noise.

Emil's Fig 19 of his treatise contains an illustration of the appearance of various SNRs, which is helpful in allowing the reader to determine what SNR might be acceptable.

Regards,

Bill

Logged

#### BernardLanguillier

• Sr. Member
• Offline
• Posts: 9901
##### Re: Photographic DR from DXO Data
« Reply #1 on: January 27, 2012, 10:44:13 PM »

In short, however we look at them, the D3x and P65+ appear to be in the same ballpark DR wise, correct?

The contrary would have been surprising considering that they are similar in terms of pixel size and sensor generation.

Cheers,
Bernard
Logged
A few images online here!

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #2 on: January 28, 2012, 01:22:58 AM »

Bill,

Good news ... and bad news ...

As it turns out you can data mine the DxoMark charts and get their data directly.
I've done it.

But the data has been heavily processed and extrapolated; the original data is lost.

For example, you can try an approach like www.sensorgen.info

Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
The problem is, the values you get are often garbage, including negative values for PRNU.

In short, that data simply isn't reliable for sensor characterization.

Regards,
Bill
Logged
Visit me at Photons to Photos

#### Guillermo Luijk

• Sr. Member
• Offline
• Posts: 1573
##### Re: Photographic DR from DXO Data
« Reply #3 on: January 28, 2012, 05:26:41 AM »

For example, you can try an approach like www.sensorgen.info

Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.

There is no need to reinvent the wheel, sensorgen data can be used straight to calculate the DR for a given SNR criteria.

Sometime ago I made a spreadsheet right for this, you just enter (first three parameters from Sensorgen, the SNR criteria at user's will, I like 12dB for regular photographic applications):
• Real ISO
• Full well (s)
• SNR dB criteria (typ. 12dB)

And the sheet calculates the standard SNR curves without the PRNU effect (SNRdB = 20 * log10 [ s / 2^-EV / (r^2 + s / 2^-EV)^(1/2) ]), and the DR for every real ISO as the point where SNR=threshold setting.
I have uploaded this Excel here in case anyone is interested: rangodinamicosensorgen.xls

Regards
« Last Edit: January 28, 2012, 05:32:48 AM by Guillermo Luijk »
Logged

#### bjanes

• Sr. Member
• Offline
• Posts: 3215
##### Re: Photographic DR from DXO Data
« Reply #4 on: January 28, 2012, 09:01:40 AM »

Bill,

Good news ... and bad news ...

As it turns out you can data mine the DxoMark charts and get their data directly.
I've done it.

But the data has been heavily processed and extrapolated; the original data is lost.

For example, you can try an approach like www.sensorgen.info

Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
The problem is, the values you get are often garbage, including negative values for PRNU.

In short, that data simply isn't reliable for sensor characterization.

Bill,

Are you saying that the DXO data are unreliable for photographic purposes? For sensor analysis, I understand that it is necessary to subtract duplicate frames to remove PRNU and other sources of pattern noise as Roger Clark does in his sensor evaluations. The subtraction also removes variations in illumination or reflectance of the target and other effects of pattern noise. When this has been done, the signal to noise ratio squared equals the number of photons collected. Without the subtraction, the noise will higher, reducing the signal:noise, and the estimate will be low.

When a new camera comes out, gurus who don't have the camera often obtain images from the internet and compare the highlights to the deep shadows and attempt to derive a DR by comparing the two values. Not only has pattern noise not been accounted for, but subject detail in the shadow may inflate the standard deviation.

I have often wondered exactly how DXO obtains their DR data. From their explanation on the web, they use neutral density filters arranged in a circular array and take exposures. This is analogous to using a Stouffer wedge, but the densities are arranged circularly rather than linearly. No subtraction has been done. From these data, one can not really derive a true engineering DR (full well/read noise). However, in most photography (aside from astro photography) one does not take bias frames and duplicate frames. However, one can compare maximum to minimum signal and derive information useful in day to day photography.

Since you are a respected guru in sensor evaluation, I would appreciate your opinion in these matters.

Regards,

Bill Janes

Logged

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #5 on: January 28, 2012, 11:45:11 AM »

Guillermo,

It's always appreciated when people share their efforts.

I have not looked at your spreadsheet yet and reserve comment on it for now.

I think this is a fine approach if you need an approximation and aren't mislead about the quality of the data.

You understand that Sensorgen processes the DxoMark SNR curve data to obtain their figures, right?
Sensorgen also uses either incomplete data, some unknown adjustment, or drops certain datapoints.
I know this because I have duplicated the work and get different answers.
My email to the person responsible for Sensorgen has gone unanswered.

Here are results for Nikon D700 read noise:
 ISO Sensorgen Bill Measured 200 15.2 15.8 15.3 400 9.1 8.8 8.5 800 5.8 5.3 5.7 1600 5.3 5.2 5.9 3200 6.0 5.5 5.7 6400 6.0 5.5 5.8 12800 6.0 5.3 2.9 25600 6.5 4.3 1.5

Note that Sensorgen and Bill values disagree even though we used the same DxOMark data for our analysis.
Note that the measured values aren't perfect either, since we expect read noise in electrons to drop as ISO increases.
However, the measured values are real and a variation up or down by .1 electron is acceptable to me.

So, use Sensorgen / DxOMark data with the knowledge that it can lead to highly inaccurate results.
Note that I used the D700 as an example because your post cited the D700.
I could have chosen an example with far greater discrepancies.

Regards,
Bill
Logged
Visit me at Photons to Photos

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #6 on: January 28, 2012, 12:30:49 PM »

Bill,

Yes, pairs of images are used when you want to eliminate fixed noise from the analysis.
As you point out, we seldom eliminate fixed noise in our actual photography.
So I think it's totally appropriate that DxOMark doesn't used paired images for the results they present.

As an aside, given their sweet test set up, it would be sad if they didn't at least gather pairs for additional analysis.

From "Testing protocols for noise and dynamic range"

They appear to use a wheel of 15(?) neutral density filters that are rotated one at a time in front of the camera.
They also measure luminance.
They "interpolate these numerical values for all gray levels to calculate and plot signal-to-noise ratio (SNR) curves".
The raw data does not appear to be available, only their interpolated results.

Unfortunately the quality of the resulting data is unacceptable for sensor characterization.

SNR data gathered with single rather than paired images should exhibit well known characteristics.
A quadratic fit to signal versus variance produces three coefficients.
Those coefficients are functions of read noise, gain, and Pixel Response Non-Uniformity (PRNU).

Putting aside questionable read noise and gain results, the DxOMark data frequently produces PRNU values that are negative.
Negative values are impossible and this points to a serious problem with the data as presented.

To summarize.
I think DxOMark is fine for gross characterizations regarding sensor performance at the "photographic" level.
For example, I think their Landscape DR numbers are too high but are valid for comparison purposes.
On the other hand, I think DxOMark is a poor, even misleading, source of data from the actual determination of low level sensor characteristics such as read noise, gain, full well capacity. etc.

Regards,
Bill
Logged
Visit me at Photons to Photos

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #7 on: January 28, 2012, 12:39:58 PM »

Bill,

So far in this thread I have been concentrating on the topic of DxOMark data and it's usefulness.

But regarding "Photographic Dynamic Range" (PDR)

Are you aware of my definition, described on my site?
Are you aware that I have such data at my site?
(For most Nikon cameras)
Do you know that anyone with a Windows PC can use my software to measure the PDR of their camera?
(So long it's a Nikon camera or they can make a linear DNG out of a raw file.)

Also, aside from their criteria being too loose, which results in uniformly high numbers; I think the DxOMark Landscape DR is a reasonable starting point.
In other words, you could figure out how many stops to "penalize" DxOMark and then simply use their numbers.

Regards,
Bill
Logged
Visit me at Photons to Photos

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #8 on: January 28, 2012, 01:20:22 PM »

Guillermo,

It is linked to Libro1. You might want to break those links if you're going to distribute it.

IMO, to be truly useful you must compute the target SNR as a function of sensor size and photosite size.
(Then, if you had good data, you would have what I call Photographic Dynamic Range  )

Regards,
Bill
Logged
Visit me at Photons to Photos

#### bjanes

• Sr. Member
• Offline
• Posts: 3215
##### Re: Photographic DR from DXO Data
« Reply #9 on: January 28, 2012, 01:30:54 PM »

Bill,

So far in this thread I have been concentrating on the topic of DxOMark data and it's usefulness.

But regarding "Photographic Dynamic Range" (PDR)

Are you aware of my definition, described on my site?
Are you aware that I have such data at my site?
(For most Nikon cameras)
Do you know that anyone with a Windows PC can use my software to measure the PDR of their camera?
(So long it's a Nikon camera or they can make a linear DNG out of a raw file.)

Also, aside from their criteria being too loose, which results in uniformly high numbers; I think the DxOMark Landscape DR is a reasonable starting point.
In other words, you could figure out how many stops to "penalize" DxOMark and then simply use their numbers.

Bill,

Yes, I am aware of your web site and did contribute raw data for my Nikon D3 when it was new. The website has a plethora of good information, but it is not well organized. I don't really know where your definition of DR resides. Of course, the DXO DR readings are not useful for practical photography, but when you start setting what noise floor is acceptable for photography, a subjective component is introduced. That is why I made my post with a reference to Emil's SNR image showing the declaration of independence with various SNRs. Let the reader decide. Since the methodology was suggested by Emil, I presumed it must have some merit.

With regard to their DXO target with the filters, I had assumed that they merely photographed all filters at once, much as one exposes a Stouffer wedge. The radial arrangement would at least make the cos^4 falloff less critical. From the image of the woman photographing the chart, I can't really tell what field she is photographing. If she is photographing individual filters, they should be much more widely separated so as to avoid flare light from the brighter areas affecting the dimmer targets. I had assumed that they merely put the target on a light table and photographed the whole thing.

The problem with merely adding a fudge factor to the DXO data is that at the higher level of the noise floor, the relative contribution of read noise to the DR decreases. With a higher noise floor, the sensor with higher read noise from a large sensor would look better.

As you know, veiling flare degrades DR, but it is difficult to avoid. For Imatest, Norman has devised a method involving photographing a Q14 target at various exposures and combining them mathematically.

Regards,

Bill
« Last Edit: January 28, 2012, 01:40:15 PM by bjanes »
Logged

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #10 on: January 28, 2012, 02:04:01 PM »

Bill,

Quote
The website has a plethora of good information, but it is not well organized. I don't really know where your definition of DR resides.

Yes, my site could use a facelift.
Especially since indenting seems broken with newer browsers.

But a simple search for "Photographic Dynamic Range" would have produced:
Engineering and Photographic Dynamic Range (2009-03-02)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Engineering_and_Photographic_Dynamic_Range.htm )
Photographic Dynamic Range Summary (2009-03-04)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Photographic_Dynamic_Range_Summary.htm )

In short, all Dynamic Range (DR) determination are fundamentally an exploration of the low end of the Photon Transfer Curve (PTC).
It's really just a question of what intersection you're looking for.
Since we can model the low end if we know read noise and gain (or Full Well Capacity (FWC) ) we can predict these figures.
I routinely predict the Nikon numbers and the predictions are always slightly above, but close to, the measured figures.

Because of the shape of the PTC, different SNR criteria produce essentially identical curves with different y-axis offsets.
That's why DxOMark Landscape DR and my PDR agree well, with an offset.

DR falls into two categories, engineering and photographic.
Engineering is at the photosite level whereas photographic is normalized using some criteria.
I don't find engineering DR useful.
I'm entirely satisfied with the criteria I have chosen for my PDR.

If you took my approach but used different criteria, your curves would simply be higher or lower.
So I suggest using my charts and simply adjusting your interpretation of the y-axis accordingly.
This is the intended use of the chart.

For example, for my photographic needs I find that a y-axis value of 6.5 is appropriate.
Look across, I can judge the maximum ISO at which I would get that quality for any particular camera.

If fact, using this (arbitrary) 6.5 criteria I have calculated these ISO values for a number of cameras.
See Photographic Dynamic Range ISOs at PDR of 6.5 Chart
( http://home.comcast.net/~NikonD70/Investigations/Photographic_Dynamic_Range_65.jpg )

If you want to make predictions for cameras I don't list or to use a different criteria, then I guess you need to "reinvent the wheel".
Just remember that in doing so you need quality data for read noise, and gain; and that is not DxOMark data.
FWIW, my criteria works out to (20 * 800)/(height of sensor) as the S/N target per photosite.

Regards,
Bill
« Last Edit: January 28, 2012, 02:14:21 PM by bclaff »
Logged
Visit me at Photons to Photos

#### ErikKaffehr

• Sr. Member
• Offline
• Posts: 10191
##### Re: Photographic DR from DXO Data
« Reply #11 on: January 28, 2012, 04:12:38 PM »

Hi,

You can simply use the "Tonal Range" curve in DxO mark.

On the other hand, I'd suggest that using engineering definition of DR may be OK. It's about noise in the darkest darks, where we would have little tonal separation anyway. Also, there is probably a reason engineers use that definition, and that reason may be a good one.

Best regards
Erik

Bill,

Yes, my site could use a facelift.
Especially since indenting seems broken with newer browsers.

But a simple search for "Photographic Dynamic Range" would have produced:
Engineering and Photographic Dynamic Range (2009-03-02)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Engineering_and_Photographic_Dynamic_Range.htm )
Photographic Dynamic Range Summary (2009-03-04)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Photographic_Dynamic_Range_Summary.htm )

In short, all Dynamic Range (DR) determination are fundamentally an exploration of the low end of the Photon Transfer Curve (PTC).
It's really just a question of what intersection you're looking for.
Since we can model the low end if we know read noise and gain (or Full Well Capacity (FWC) ) we can predict these figures.
I routinely predict the Nikon numbers and the predictions are always slightly above, but close to, the measured figures.

Because of the shape of the PTC, different SNR criteria produce essentially identical curves with different y-axis offsets.
That's why DxOMark Landscape DR and my PDR agree well, with an offset.

DR falls into two categories, engineering and photographic.
Engineering is at the photosite level whereas photographic is normalized using some criteria.
I don't find engineering DR useful.
I'm entirely satisfied with the criteria I have chosen for my PDR.

If you took my approach but used different criteria, your curves would simply be higher or lower.
So I suggest using my charts and simply adjusting your interpretation of the y-axis accordingly.
This is the intended use of the chart.

For example, for my photographic needs I find that a y-axis value of 6.5 is appropriate.
Look across, I can judge the maximum ISO at which I would get that quality for any particular camera.

If fact, using this (arbitrary) 6.5 criteria I have calculated these ISO values for a number of cameras.
See Photographic Dynamic Range ISOs at PDR of 6.5 Chart
( http://home.comcast.net/~NikonD70/Investigations/Photographic_Dynamic_Range_65.jpg )

If you want to make predictions for cameras I don't list or to use a different criteria, then I guess you need to "reinvent the wheel".
Just remember that in doing so you need quality data for read noise, and gain; and that is not DxOMark data.
FWIW, my criteria works out to (20 * 800)/(height of sensor) as the S/N target per photosite.

Regards,
Bill

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #12 on: January 28, 2012, 04:52:06 PM »

Erik,

Quote
You can simply use the "Tonal Range" curve in DxO mark.

I admit I don't know what they're measuring there.
But it doesn't fall off quickly enough compared to  "Photographic Dynamic Range".

Quote
On the other hand, I'd suggest that using engineering definition of DR may be OK. It's about noise in the darkest darks, where we would have little tonal separation anyway. Also, there is probably a reason engineers use that definition, and that reason may be a good one.

No. Engineers are not interested in the photographic quality.
About noise in the darks, absolutely.
Any DR measure that doesn't take into account photosite size in relation to the final output is not useful in comparing cameras.

Regards,
Bill
Logged
Visit me at Photons to Photos

#### ErikKaffehr

• Sr. Member
• Offline
• Posts: 10191
##### Re: Photographic DR from DXO Data
« Reply #13 on: January 28, 2012, 05:08:36 PM »

Hi,

Tonal range in DxO is essentially DR with a different threshold on SNR.It may be 16, but I don't know. DxO measurements "in print mode" are normalized to a standard print size.

Best regards
Erik

Erik,

I admit I don't know what they're measuring there.
But it doesn't fall off quickly enough compared to  "Photographic Dynamic Range".

No. Engineers are not interested in the photographic quality.
About noise in the darks, absolutely.
Any DR measure that doesn't take into account photosite size in relation to the final output is not useful in comparing cameras.

Regards,
Bill

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #14 on: January 28, 2012, 05:18:25 PM »

Erik,

Quote
Tonal range in DxO is essentially DR with a different threshold on SNR.

Perhaps, but I doubt it.
In my experience if that were so the curves would drop off more quickly.

In any case, it's of no consequence to me.
I don't use DxOMark except for isolated gross cross-checks.

If you find DxOMark Tonal Range useful, that's great for you; I'm not trying to talk you out of it.
(I do admit that I am trying to talk others out of adopting it blindly as a worthy measure.)

Regards,
Bill
Logged
Visit me at Photons to Photos

#### Guillermo Luijk

• Sr. Member
• Offline
• Posts: 1573
##### Re: Photographic DR from DXO Data
« Reply #15 on: January 28, 2012, 07:05:07 PM »

In any case, it's of no consequence to me.
I don't use DxOMark except for isolated gross cross-checks.

Hi Bill (did you know Guillermo = Bill? there are three of us here already), I also take DxOMark as a trusty reference. I mean, in many aspects it remains a bit as a black box, but at least they rely on RAW data and follow procedures that applied to different cameras should be fine to make comparisions.

Let me ask you something: this is the SNR response of a Pentax K5. Yellow are my measures (based on looking at StdDev in uniform colour patches, always over pure RAW data extracted using DCRAW), and the blue plot is Sensorgen plot based on their read noise and full well measures:

I don't do any RAW data substraction to eliminate the effect of PRNU, and I think that is wht my plots get lower SNR for high exposure values. I really don't know how much PRNU affects DxOMark curves (I think they eliminate this effect and that is what their SNR near saturation improves by 3dB/EV), and hence Sensorgen summarized figures.

Can you think why specially my ISO80 measures could have a much lower SNR than an ideal expected PRNU-free response? ISO1600 on the other side is much more accurate. And both ISO80 and ISO1600 are fine in low exposure areas where the PRNU effect is not dominant.

Regards

Logged

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #16 on: January 28, 2012, 08:43:50 PM »

Guillermo,

Quote
did you know Guillermo = Bill?
Yes

I think that your measurements of the Pentax K5 are probably accurate.
I question the DxOMark values, and by extension the Sensorgen derived values.
For example, the Sensorgen derived Read Noise for ISO 80 is almost certainly low.
A value of 3.8 or 3.9 would be more in line with the other ISOs.

DxOMark does not correct for PRNU, so that would not account for any difference.

Since you capture more light (photons, electrons) at ISO 80 than at ISO 1600, the effect of PRNU would be greater.

I would encourage you to trust yourself!
It is not "why is my data low?", it is "why is their data high?"

If you email the data that you have collected I would be happy to look at it.

Regards,
Bill
Logged
Visit me at Photons to Photos

#### deejjjaaaa

• Sr. Member
• Offline
• Posts: 1170
##### Re: Photographic DR from DXO Data
« Reply #17 on: January 29, 2012, 12:32:40 AM »

Do you know that anyone with a Windows PC can use my software to measure the PDR of their camera?
(So long it's a Nikon camera or they can make a linear DNG out of a raw file.)

Bill, it seems that your program (NefUtil) does not work w/ linear DNG files... out of curiosity I tried to convert a .NEF from D7000 to linear .DNG using Adobe DNG converter v6.5.0.218 - NefUtil says
"Warning: unsupported image format"
Logged

#### bclaff

• Jr. Member
• Offline
• Posts: 77
##### Re: Photographic DR from DXO Data
« Reply #18 on: January 29, 2012, 12:43:42 AM »

Bill, it seems that your program (NefUtil) does not work w/ linear DNG files... out of curiosity I tried to convert a .NEF from D7000 to linear .DNG using Adobe DNG converter v6.5.0.218 - NefUtil says
"Warning: unsupported image format"
Of course you known that NefUtil handles D7000 NEF directly and you're just testing NDG, right?
I think NefUtil is picky about the exact format.
With my DNG Converter I set Compatibility to "Custom", Backward version to DNG 1.3, Linear NOT checked, and Uncompressed CHECKED.
(I now realize I said linear earlier and should have said uncompressed !!! )

It is a pain, I admit. If more people want DNG support then perhaps I'll look into supporting it.

The other problem, with Canon and some others, is that NefUtil is largely unaware of non-zero offsets for black.
People sometimes send me files, and I "hack" this for them.

« Last Edit: January 29, 2012, 12:47:04 AM by bclaff »
Logged
Visit me at Photons to Photos

#### deejjjaaaa

• Sr. Member
• Offline
• Posts: 1170
##### Re: Photographic DR from DXO Data
« Reply #19 on: January 29, 2012, 01:07:54 AM »

(I now realize I said linear earlier and should have said uncompressed !!! )

I see - that was an issue.

Logged
Pages: [1] 2   Go Up