Photographic DR from DXO Data

[deejjjaaaa]

It is a pain, I admit. If more people want DNG support then perhaps I'll look into supporting it.

It seems that your program is a good tool - so (converted or native) DNG support for other cameras (unless something like using libraw C++ library is possible to read the data from the raw files) will be an ideal solution (as you mentioned that there are issues w/ Canon and some other models)...

[Guillermo Luijk]

I think that your measurements of the Pentax K5 are probably accurate.
I question the DxOMark values, and by extension the Sensorgen derived values.
For example, the Sensorgen derived Read Noise for ISO 80 is almost certainly low.
A value of 3.8 or 3.9 would be more in line with the other ISOs.

DxOMark does not correct for PRNU, so that would not account for any difference.

Since you capture more light (photons, electrons) at ISO 80 than at ISO 1600, the effect of PRNU would be greater.

I would encourage you to trust yourself!
It is not "why is my data low?", it is "why is their data high?"

If you email the data that you have collected I would be happy to look at it.

OK, DxOMark don't eliminate PRNU. Sensorgen's 'read noise' and 'full well' capacity are derived from DxOMark plots (which have not been de-PRNU'ed). So what can we expect from those value pairs? it's difficult for me to try to track the consequences.

My plots based on Sensorgen are theoretical plots only assuming the read noise and full well terms in the noise model as given by DxOMark. That is why they end in a perfect 3dB/EV slope near saturation.

My measures are totally independent from DxOMark. I calculated the exact saturation point (for the K5 this happened to be in the end of the 14-bit scale) and took this as reference 0EV. Then compared pairs (EV, SNR), where EV is how far the average signal in a given patch is from saturation, and SNR is noise's StdDev compared to the corresponding EV. All three channels were used equally to produce plot samples, each contributing at its RAW EV value achieved.

In this Excel file I collected all measures obtained with Rawnalyze. This article explains the general procedure.

The point is my DR figure (SNR=12dB criteria) matches well the value derived from Sensorgen SNR curve because in areas with low exposure values both sources provide similar SNR.

Regards

[bclaff]

Guillermo,

I'm having trouble following the raw data you collected for the K-5, lines 94 and 95, Avg and StdDev.

For example, Column C Ave = 75 StdDev = 2.08

This data came from analyzing the same area of two images taken under identical conditions?
Ave is the average of the two areas added together?
StdDev is the square root of the variance of the subtraction of the two areas divided by 2?
Which channel is this?
Did you try to combine channels? If so, how?

(I admit I did not read the article yet. It's not in my first language  )

Regards,
Bill

[Guillermo Luijk]

I'm having trouble following the raw data you collected for the K-5, lines 94 and 95, Avg and StdDev.

For example, Column C Ave = 75 StdDev = 2.08

This data came from analyzing the same area of two images taken under identical conditions?
Ave is the average of the two areas added together?
StdDev is the square root of the variance of the subtraction of the two areas divided by 2?
Which channel is this?
Did you try to combine channels? If so, how?

Hi Bill, I think this image makes it clear:



Every pair (Average, Standard Deviation) comes from a single patch in a single channel. So every patch yields 3 values to plot the curve (one for every RAW channel).

Average is the mean RAW value in the patch, i.e. the Signal, and StdDev is the Noise, so we obtain a pair (RAW exposure, SNR).

I substract the black level and take into account the saturation level when needed.

Sensor linearity is assumed in order to locate each Average RAW value in stops with respect to saturation, so as uniformity between the 3 RAW channels (I don't have any reason to think a channel will produce a different SNR for the same number of accounted photons), that is why the samples come from the three channels without making any distinction nor combination.

Regards!

[bclaff]

Guillermo,

OK.
I misunderstood.
When your said pair I assumed you were subtracting to remove PRNU.
(Even though you said earlier that you were not!)

Of course there are 4 channels, not 3; there are two green and they are not usually identical.

I would stick to one of the green channels.
Since you are not adjusting for PRNU I think the channels might not all behave exactly the same.
If you use all channels then you should at least record the channels separately to verify your assumption by examining the data.

Regards

[Guillermo Luijk]

Since you are not adjusting for PRNU I think the channels might not all behave exactly the same.

Yes, that check was implicit in the procedure. If you look at the plots all points are clearly scattered following the same trend. Should one of the 4 RAW channels behave differently, its points wouldn't produce similar pairs as the others, should be shifted or something.

Now I'm thinking that Rawnalyze must treat G1/G2 assuming they behave equally, since it only outputs a single value for G. In some cameras such as many 4/3 this is not true though, but in general it is.

[bclaff]

Guillermo,

If you look at the plots all points are clearly scattered following the same trend. Should one of the 4 RAW channels behave differently, its points wouldn't produce similar pairs as the others, should be shifted or something.

You don't identify which points are from which channel so it's not clear that there is not a pattern of some sort.

Regards

[Guillermo Luijk]

You don't identify which points are from which channel so it's not clear that there is not a pattern of some sort.

I recall to have done just a quick visual inspection at that time (this was 2 years ago), taking dots from a single channel, and I couldn't see anything different from the other channels. If you look at the dots, they are very aligned with the trend, so I think the error margin is more than acceptable:



Larger errors surely come from not chosing a perfectly accurate black point (needed to consider the data linear in order to locate the dots in the X-axis), effect of the PRNU with good exposure,...

Regards

[bjanes]

Good news ... and bad news ...

As it turns out you can data mine the DxoMark charts and get their data directly.
I've done it.

But the data has been heavily processed and extrapolated; the original data is lost.

For example, you can try an approach like www.sensorgen.info

Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
The problem is, the values you get are often garbage, including negative values for PRNU.

In short, that data simply isn't reliable for sensor characterization.

Bill,

The DXO data may not be good for sensor characterization, but it does have some utility. The engineering DR may not be be useful for photographic purposes, but the photographic DR extrapolated by Emil's method does have some validity (I think).

To evaluate PRNU with my Nikon D3, I took duplicate shots from my monitor according to your method. I used a 300 mm lens at f/8 to minimize light falloff. I then separated out the green 1 and 2 channels using Iris and cropped the resulting files to the 201 x 201 pixel central area of the images and analyzed them with ImageJ, taking the mean and standard deviations of the images and then subtracting the identical images and dividing the resulting standard deviation by sqrt(2).

The histograms of a representative image and difference image are shown:



Subtracting the images removed PRNU, and the resulting standard deviations represent shot noise and read noise, which add in quadrature. I then subtracted in quadrature the the SD of the image without PRNU from SDs of total noise (individual image SDs) to obtain the PRNU and then expressed PRNU as a percentage of the data numbers (DN). PRNU was 0.3% of the DNs. To avoid negative numbers in the subtraction, I added 400 to one image before subtracting the other.

Then I constructed a theoretical noise model using the full well and read noise from your data with and without a PRNU of 0.3% and calculating the SNR.



The resulting plot with PRNU is virtually identical to the DXO SNR plot. The Screen DR reported by DXO was also virtually identical to the engineering DR from your full well and read noise data.



Your comments would be appreciated.

Regards,

Bill

[bclaff]

Bill,

It's good work, but you miss my point.

You took measured D3 values, 63600 and 16.4512 from my site.
You personally measured a PRNU of .3% (I get .45% but this difference is not important for this discussion)

You found that the measured values produced a curve that is highly similar to the DxOMark curve.

However, if you started solely with the DxoMark data you would get D3 values of:
59200 rather than 63600 for FWC; about 7% difference
14.798 rather than 16.4512 for read noise in electrons; about 10% difference

If you took DxOMark data for an arbitrary camera you would get impossible read noise values for some cameras.
(And impossible PRNU as well, FWIW.)

In short, the DxOMark data cannot be relied upon for these normal sensor characteristics.
It's sad because they probably have good underlying data that we can't retrieve because it's been over cooked for presentation.
(Or intentionally to obfuscate it.)

I have attempted to clarify this with DxOMark using two different approaches and have had no response to either inquiry.

Regards,
Bill
P.S. - Have you received my emails?

[bjanes]

Bill,

It's good work, but you miss my point.

You took measured D3 values, 63600 and 16.4512 from my site.
You personally measured a PRNU of .3% (I get .45% but this difference is not important for this discussion)

You found that the measured values produced a curve that is highly similar to the DxOMark curve.

However, if you started solely with the DxoMark data you would get D3 values of:
59200 rather than 63600 for FWC; about 7% difference
14.798 rather than 16.4512 for read noise in electrons; about 10% difference

If you took DxOMark data for an arbitrary camera you would get impossible read noise values for some cameras.
(And impossible PRNU as well, FWIW.)

In short, the DxOMark data cannot be relied upon for these normal sensor characteristics.
It's sad because they probably have good underlying data that we can't retrieve because it's been over cooked for presentation.
(Or intentionally to obfuscate it.)

I have attempted to clarify this with DxOMark using two different approaches and have had no response to either inquiry.

Bill,

I only have a few data points and can't check my data for consistency, so it is not surprising that I obtained a slightly different value for PRNU. The data collection by hand using the tools that I employed, is quite laborious and prone to error if one is not careful. Your automated method is much better.

Now I understand your point. Hitherto, most readers have accepted the DXO data as accurate. If they were truly to follow the scientific method, they would post detailed descriptions of how their measurements were obtained and how the calculations were performed. The raw data would be helpful, too. Otherwise, other observers can not repeat their experiments, as you have found.

Regards,

Bill Janes

[bclaff]

Bill,

they would post detailed descriptions of how their measurements were obtained and how the calculations were performed

Yes, DxoMark makes a show of this, but fall short in execution (IMO).

Regards,
Bill

[Chris Warren]

Bill,

...
However, if you started solely with the DxoMark data you would get D3 values of:
59200 rather than 63600 for FWC; about 7% difference
14.798 rather than 16.4512 for read noise in electrons; about 10% difference

If you took DxOMark data for an arbitrary camera you would get impossible read noise values for some cameras.
(And impossible PRNU as well, FWIW.)

In short, the DxOMark data cannot be relied upon for these normal sensor characteristics.
It's sad because they probably have good underlying data that we can't retrieve because it's been over cooked for presentation.
(Or intentionally to obfuscate it.)...

Yes hi Bill (all Bill's actually),  This is interesting to me, as I have seen some similar differences.  My name is Chris, and have had some disscussions with you on DPR in the past.  My camera is the Nikon D40, and I have been doing some traditional Photon Transfer measurements on it using differenced flat fields.  I varied exposure via exposure time, and found the sensor behaving quite linearly (unfortunately I don't know how to add charts) for DNs out vs light in, and also found the variance vs average DNs to behave linearly as well.  I took several data points, including very low exposure levels in, which allowed me to extrapolate the variance line to the y-axis and get good read noise data.  For evaluating the read noise, I used only flat fields that did not have more than 1% zeroed (or black clipped) data.  Here are my results:

ISO     k (e/DN)    Nr (e)    FW (e)    PRNU %
200      7.10        13.84     28,381    0.57
400      3.53        11.41     14,095    0.564
800      1.83        11.53     7,277      0.59
1600     0.94        12.56     3,735     0.58

For this, I used a 100x100 central region, extracted the G1 channel via Iris, and evaluated statistics with ImageJ.  k = camera gain constant, Nr = Read Noise, and FW = Fullwell Capacity.  I compared my measured results with curve fitting to the Dxomark full SNR curves, which is shown below:

ISO     k (e/DN)     Nr (e)     FW (e)    PRNU %
200      6.53         10.81     26,088     0.404
400      3.00         8.05       11,995     0.177
800      1.54         8.53        6,170     -0.4
1600     0.84        10.06       3,372     -0.56

Some comments:
-  for my measured results, PRNU stays relatively constant wrto ISO, which makes sense to me, however for the Dxomark data, the PRNU #s vary with ISO and sometimes go negative. (It is instructive to do the curve fitting, because Sensorgen doe not report PRNU).
-  I understand that Nikon clips their blacks, but how do they do so?  Do they:
      - take say 100 dark frames at a short exposure time, average them together, and use this averaged dark frame to subtract
      - subtract a constant DN level from each pixel
      - use some other method
     (I'm not against it, just curious, I'd think method 1 would be the cleanest, but I don't know)
-  PRNU can be appreciable at low ISO, i.e. near saturation the shot noise is = sqrt(28,000) = 167 e, PRNU noise = 160 e, so I think camera makers should provide a button or function to do flat fielding if desired for getting some non-destructive NR.

Well, that's what I measured for my camera so far.

Chris

 

[bclaff]

Chris,

Great work. I'm caught up on another project right now but want to comment at length when I have more time.

Some random comments.

Varying exposure time is a fine technique provided you don't accidentally cross one of the unavoidable NR boundaries; keep it faster than 1/4s.

Your k values look pretty good, clearly better than DxoMark, since 1/k should be linear with ISO.

My DxoMark numbers are slightly different. Probably I have more data than you.

My gain (1/k) numbers are similar to yours. Probably no more or less accurate.

You read noise in DN is slightly lower than mine. That's a surprise since mine are from "optical black". Perhaps sample variation?

Yes, PRNU would be independent of ISO.

I would guess that Nikon "clips" their blacks to the average optical black value, perhaps even at the row level.

If you use a PC it might be fun to cross-check against my NefUtil output.

Regards,
Bill

[Chris Warren]

...

Varying exposure time is a fine technique provided you don't accidentally cross one of the unavoidable NR boundaries; keep it faster than 1/4s.

...

Ok thanks Bill,

I kept exposure time faster than 1/15 sec for all ISOs.  I'd like to try using your NefUtil, ok to email you at your site?

Chris

[bclaff]

Chris,

You can always email me for assistance or with questions, but NefUtil itself, and documentation are on my site under 'Downloads'.

Regards,
Bill