Luminous Landscape Forum
Equipment & Techniques => Cameras, Lenses and Shooting gear => Topic started by: bjanes on January 27, 2012, 05:58:09 pm
-
The DXO dynamic range tests are for engineering DR, while photographers are more interested in the dynamic range more relevant for photographic use. Emil Martinec has authored a post (http://www.luminous-landscape.com/forum/index.php?topic=42158.0) explaining the two criteria and how one can determine photographic dynamic range from the DXO data. In attempting to apply Emil's method, I found that the main difficulty was reading the log gray scale values with sufficient accuracy to obtain accurate results. Wikipedia has a section on Estimating values in a diagram with logarithmic scale (http://en.wikipedia.org/wiki/Logarithmic_scale). The method is fairly straightforward.
One can read the distances from the graphs with the ruler in Photoshop, and calculations are facilitated by making a simple spreadsheet in Excel. An example is calculating these values for various SNRs for the P65+ at base ISO. One downloads the DXO graph and measures the x-distance from the next lower decade grid line of the intersection of the SNR curve with the y-value of a given SNR. One stop is 6dB.
(http://bjanes.smugmug.com/Photography/DX2IQ180/i-jzQFJzG/0/O/P65-SNR.png)
One then enters the x-distance into the spreadsheet to determine the interpolated x-value. The screen DR is calculated. Normalization is done as per the DXO protocol: Normalized DR = screen DR + log base 2 (sqrt[m1/m0]), where m1 is the megapixel count of the test camera and m0 is the megapixel count of the reference camera (8MP for the DX0 calculation). Emil uses a slightly different normalization.
The results of the exercise are shown:
(http://bjanes.smugmug.com/Photography/DX2IQ180/i-LSCX67Z/0/O/ExcelDRP65.png)
One can compare with the Nikon D3x:
(http://bjanes.smugmug.com/Photography/DX2IQ180/i-jzbczSN/0/O/D3-xSNR.png)
(http://bjanes.smugmug.com/Photography/DX2IQ180/i-TzSDJJB/0/O/ExcelDRD3xb.png)
According to the DXO method of normalization, the photographic DR of the Nikon is greater at all noise floors, whereas Emil's method of normalization shows a slight advantage for the Phase One. One should note that SNRs at very low illumination are more affected by read noise, while those at higher illumination are more dependent on shot noise. Current MFDBs have relatively high read noise.
Emil's Fig 19 of his treatise contains an illustration of the appearance of various SNRs, which is helpful in allowing the reader to determine what SNR might be acceptable.
Regards,
Bill
-
In short, however we look at them, the D3x and P65+ appear to be in the same ballpark DR wise, correct?
The contrary would have been surprising considering that they are similar in terms of pixel size and sensor generation.
Cheers,
Bernard
-
Bill,
Good news ... and bad news ...
As it turns out you can data mine the DxoMark charts and get their data directly.
I've done it.
But the data has been heavily processed and extrapolated; the original data is lost.
For example, you can try an approach like www.sensorgen.info
Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
The problem is, the values you get are often garbage, including negative values for PRNU.
In short, that data simply isn't reliable for sensor characterization.
Regards,
Bill
-
For example, you can try an approach like www.sensorgen.info
Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
There is no need to reinvent the wheel, sensorgen data can be used straight to calculate the DR for a given SNR criteria.
Sometime ago I made a spreadsheet right for this, you just enter (first three parameters from Sensorgen, the SNR criteria at user's will, I like 12dB for regular photographic applications):
- Real ISO
- Full well (s)
- Read noise (r)
- SNR dB criteria (typ. 12dB)
(http://img20.imageshack.us/img20/253/25930348.gif)
And the sheet calculates the standard SNR curves without the PRNU effect (SNRdB = 20 * log10 [ s / 2^-EV / (r^2 + s / 2^-EV)^(1/2) ]), and the DR for every real ISO as the point where SNR=threshold setting.
I have uploaded this Excel here in case anyone is interested: rangodinamicosensorgen.xls (http://www.guillermoluijk.com/download/rangodinamicosensorgen.xls)
Regards
-
Bill,
Good news ... and bad news ...
As it turns out you can data mine the DxoMark charts and get their data directly.
I've done it.
But the data has been heavily processed and extrapolated; the original data is lost.
For example, you can try an approach like www.sensorgen.info
Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
The problem is, the values you get are often garbage, including negative values for PRNU.
In short, that data simply isn't reliable for sensor characterization.
Bill,
Are you saying that the DXO data are unreliable for photographic purposes? For sensor analysis, I understand that it is necessary to subtract duplicate frames to remove PRNU and other sources of pattern noise as Roger Clark (http://www.clarkvision.com/imagedetail/evaluation-1d2/index.html) does in his sensor evaluations. The subtraction also removes variations in illumination or reflectance of the target and other effects of pattern noise. When this has been done, the signal to noise ratio squared equals the number of photons collected. Without the subtraction, the noise will higher, reducing the signal:noise, and the estimate will be low.
When a new camera comes out, gurus who don't have the camera often obtain images from the internet and compare the highlights to the deep shadows and attempt to derive a DR by comparing the two values. Not only has pattern noise not been accounted for, but subject detail in the shadow may inflate the standard deviation.
I have often wondered exactly how DXO obtains their DR data. From their explanation on the web, they use neutral density filters arranged in a circular array and take exposures. This is analogous to using a Stouffer wedge, but the densities are arranged circularly rather than linearly. No subtraction has been done. From these data, one can not really derive a true engineering DR (full well/read noise). However, in most photography (aside from astro photography) one does not take bias frames and duplicate frames. However, one can compare maximum to minimum signal and derive information useful in day to day photography.
Since you are a respected guru in sensor evaluation, I would appreciate your opinion in these matters.
Regards,
Bill Janes
-
Guillermo,
Thanks for sharing your work.
It's always appreciated when people share their efforts.
I have not looked at your spreadsheet yet and reserve comment on it for now.
I think this is a fine approach if you need an approximation and aren't mislead about the quality of the data.
You understand that Sensorgen processes the DxoMark SNR curve data to obtain their figures, right?
Sensorgen also uses either incomplete data, some unknown adjustment, or drops certain datapoints.
I know this because I have duplicated the work and get different answers.
My email to the person responsible for Sensorgen has gone unanswered.
Here are results for Nikon D700 read noise:
| ISO | Sensorgen | Bill | Measured |
| 200 | 15.2 | 15.8 | 15.3 |
| 400 | 9.1 | 8.8 | 8.5 |
| 800 | 5.8 | 5.3 | 5.7 |
| 1600 | 5.3 | 5.2 | 5.9 |
| 3200 | 6.0 | 5.5 | 5.7 |
| 6400 | 6.0 | 5.5 | 5.8 |
| 12800 | 6.0 | 5.3 | 2.9 |
| 25600 | 6.5 | 4.3 | 1.5 |
Note that Sensorgen and Bill values disagree even though we used the same DxOMark data for our analysis.
Note that the measured values aren't perfect either, since we expect read noise in electrons to drop as ISO increases.
However, the measured values are real and a variation up or down by .1 electron is acceptable to me.
So, use Sensorgen / DxOMark data with the knowledge that it can lead to highly inaccurate results.
Note that I used the D700 as an example because your post cited the D700.
I could have chosen an example with far greater discrepancies.
Regards,
Bill
-
Bill,
Yes, pairs of images are used when you want to eliminate fixed noise from the analysis.
As you point out, we seldom eliminate fixed noise in our actual photography.
So I think it's totally appropriate that DxOMark doesn't used paired images for the results they present.
As an aside, given their sweet test set up, it would be sad if they didn't at least gather pairs for additional analysis.
From "Testing protocols for noise and dynamic range"
( http://www.dxomark.com/index.php/About/In-depth-measurements/DxOMark-testing-protocols/Noise-dynamic-range )
They appear to use a wheel of 15(?) neutral density filters that are rotated one at a time in front of the camera.
They also measure luminance.
They "interpolate these numerical values for all gray levels to calculate and plot signal-to-noise ratio (SNR) curves".
The raw data does not appear to be available, only their interpolated results.
Unfortunately the quality of the resulting data is unacceptable for sensor characterization.
SNR data gathered with single rather than paired images should exhibit well known characteristics.
A quadratic fit to signal versus variance produces three coefficients.
Those coefficients are functions of read noise, gain, and Pixel Response Non-Uniformity (PRNU).
Putting aside questionable read noise and gain results, the DxOMark data frequently produces PRNU values that are negative.
Negative values are impossible and this points to a serious problem with the data as presented.
To summarize.
I think DxOMark is fine for gross characterizations regarding sensor performance at the "photographic" level.
For example, I think their Landscape DR numbers are too high but are valid for comparison purposes.
On the other hand, I think DxOMark is a poor, even misleading, source of data from the actual determination of low level sensor characteristics such as read noise, gain, full well capacity. etc.
Regards,
Bill
-
Bill,
So far in this thread I have been concentrating on the topic of DxOMark data and it's usefulness.
But regarding "Photographic Dynamic Range" (PDR)
Are you aware of my definition, described on my site?
Are you aware that I have such data at my site?
(For most Nikon cameras)
Do you know that anyone with a Windows PC can use my software to measure the PDR of their camera?
(So long it's a Nikon camera or they can make a linear DNG out of a raw file.)
Also, aside from their criteria being too loose, which results in uniformly high numbers; I think the DxOMark Landscape DR is a reasonable starting point.
In other words, you could figure out how many stops to "penalize" DxOMark and then simply use their numbers.
Regards,
Bill
-
Guillermo,
Regarding the spreadsheet.
It is linked to Libro1. You might want to break those links if you're going to distribute it.
IMO, to be truly useful you must compute the target SNR as a function of sensor size and photosite size.
(Then, if you had good data, you would have what I call Photographic Dynamic Range ;) )
Regards,
Bill
-
Bill,
So far in this thread I have been concentrating on the topic of DxOMark data and it's usefulness.
But regarding "Photographic Dynamic Range" (PDR)
Are you aware of my definition, described on my site?
Are you aware that I have such data at my site?
(For most Nikon cameras)
Do you know that anyone with a Windows PC can use my software to measure the PDR of their camera?
(So long it's a Nikon camera or they can make a linear DNG out of a raw file.)
Also, aside from their criteria being too loose, which results in uniformly high numbers; I think the DxOMark Landscape DR is a reasonable starting point.
In other words, you could figure out how many stops to "penalize" DxOMark and then simply use their numbers.
Bill,
Yes, I am aware of your web site and did contribute raw data for my Nikon D3 when it was new. The website has a plethora of good information, but it is not well organized. I don't really know where your definition of DR resides. Of course, the DXO DR readings are not useful for practical photography, but when you start setting what noise floor is acceptable for photography, a subjective component is introduced. That is why I made my post with a reference to Emil's SNR image showing the declaration of independence with various SNRs. Let the reader decide. Since the methodology was suggested by Emil, I presumed it must have some merit.
With regard to their DXO target with the filters, I had assumed that they merely photographed all filters at once, much as one exposes a Stouffer wedge. The radial arrangement would at least make the cos^4 falloff less critical. From the image of the woman photographing the chart, I can't really tell what field she is photographing. If she is photographing individual filters, they should be much more widely separated so as to avoid flare light from the brighter areas affecting the dimmer targets. I had assumed that they merely put the target on a light table and photographed the whole thing.
The problem with merely adding a fudge factor to the DXO data is that at the higher level of the noise floor, the relative contribution of read noise to the DR decreases. With a higher noise floor, the sensor with higher read noise from a large sensor would look better.
As you know, veiling flare degrades DR, but it is difficult to avoid. For Imatest, Norman has devised a method involving photographing a Q14 target at various exposures and combining them mathematically.
Regards,
Bill
-
Bill,
The website has a plethora of good information, but it is not well organized. I don't really know where your definition of DR resides.
Yes, my site could use a facelift. :)
Especially since indenting seems broken with newer browsers.
But a simple search for "Photographic Dynamic Range" would have produced:
Engineering and Photographic Dynamic Range (2009-03-02)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Engineering_and_Photographic_Dynamic_Range.htm )
Photographic Dynamic Range Summary (2009-03-04)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Photographic_Dynamic_Range_Summary.htm )
In short, all Dynamic Range (DR) determination are fundamentally an exploration of the low end of the Photon Transfer Curve (PTC).
It's really just a question of what intersection you're looking for.
Since we can model the low end if we know read noise and gain (or Full Well Capacity (FWC) ) we can predict these figures.
I routinely predict the Nikon numbers and the predictions are always slightly above, but close to, the measured figures.
Because of the shape of the PTC, different SNR criteria produce essentially identical curves with different y-axis offsets.
That's why DxOMark Landscape DR and my PDR agree well, with an offset.
DR falls into two categories, engineering and photographic.
Engineering is at the photosite level whereas photographic is normalized using some criteria.
I don't find engineering DR useful.
I'm entirely satisfied with the criteria I have chosen for my PDR.
If you took my approach but used different criteria, your curves would simply be higher or lower.
So I suggest using my charts and simply adjusting your interpretation of the y-axis accordingly.
This is the intended use of the chart.
For example, for my photographic needs I find that a y-axis value of 6.5 is appropriate.
Look across, I can judge the maximum ISO at which I would get that quality for any particular camera.
If fact, using this (arbitrary) 6.5 criteria I have calculated these ISO values for a number of cameras.
See Photographic Dynamic Range ISOs at PDR of 6.5 Chart
( http://home.comcast.net/~NikonD70/Investigations/Photographic_Dynamic_Range_65.jpg )
If you want to make predictions for cameras I don't list or to use a different criteria, then I guess you need to "reinvent the wheel".
Just remember that in doing so you need quality data for read noise, and gain; and that is not DxOMark data.
FWIW, my criteria works out to (20 * 800)/(height of sensor) as the S/N target per photosite.
Regards,
Bill
-
Hi,
You can simply use the "Tonal Range" curve in DxO mark.
On the other hand, I'd suggest that using engineering definition of DR may be OK. It's about noise in the darkest darks, where we would have little tonal separation anyway. Also, there is probably a reason engineers use that definition, and that reason may be a good one.
Best regards
Erik
Bill,
Yes, my site could use a facelift. :)
Especially since indenting seems broken with newer browsers.
But a simple search for "Photographic Dynamic Range" would have produced:
Engineering and Photographic Dynamic Range (2009-03-02)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Engineering_and_Photographic_Dynamic_Range.htm )
Photographic Dynamic Range Summary (2009-03-04)
( http://home.comcast.net/~NikonD70/GeneralTopics/Sensors_&_Raw/Sensor_Analysis_Primer/Photographic_Dynamic_Range_Summary.htm )
In short, all Dynamic Range (DR) determination are fundamentally an exploration of the low end of the Photon Transfer Curve (PTC).
It's really just a question of what intersection you're looking for.
Since we can model the low end if we know read noise and gain (or Full Well Capacity (FWC) ) we can predict these figures.
I routinely predict the Nikon numbers and the predictions are always slightly above, but close to, the measured figures.
Because of the shape of the PTC, different SNR criteria produce essentially identical curves with different y-axis offsets.
That's why DxOMark Landscape DR and my PDR agree well, with an offset.
DR falls into two categories, engineering and photographic.
Engineering is at the photosite level whereas photographic is normalized using some criteria.
I don't find engineering DR useful.
I'm entirely satisfied with the criteria I have chosen for my PDR.
If you took my approach but used different criteria, your curves would simply be higher or lower.
So I suggest using my charts and simply adjusting your interpretation of the y-axis accordingly.
This is the intended use of the chart.
For example, for my photographic needs I find that a y-axis value of 6.5 is appropriate.
Look across, I can judge the maximum ISO at which I would get that quality for any particular camera.
If fact, using this (arbitrary) 6.5 criteria I have calculated these ISO values for a number of cameras.
See Photographic Dynamic Range ISOs at PDR of 6.5 Chart
( http://home.comcast.net/~NikonD70/Investigations/Photographic_Dynamic_Range_65.jpg )
If you want to make predictions for cameras I don't list or to use a different criteria, then I guess you need to "reinvent the wheel".
Just remember that in doing so you need quality data for read noise, and gain; and that is not DxOMark data.
FWIW, my criteria works out to (20 * 800)/(height of sensor) as the S/N target per photosite.
Regards,
Bill
-
Erik,
You can simply use the "Tonal Range" curve in DxO mark.
I admit I don't know what they're measuring there.
But it doesn't fall off quickly enough compared to "Photographic Dynamic Range".
On the other hand, I'd suggest that using engineering definition of DR may be OK. It's about noise in the darkest darks, where we would have little tonal separation anyway. Also, there is probably a reason engineers use that definition, and that reason may be a good one.
No. Engineers are not interested in the photographic quality.
About noise in the darks, absolutely.
Any DR measure that doesn't take into account photosite size in relation to the final output is not useful in comparing cameras.
Regards,
Bill
-
Hi,
Tonal range in DxO is essentially DR with a different threshold on SNR.It may be 16, but I don't know. DxO measurements "in print mode" are normalized to a standard print size.
Best regards
Erik
Erik,
I admit I don't know what they're measuring there.
But it doesn't fall off quickly enough compared to "Photographic Dynamic Range".
No. Engineers are not interested in the photographic quality.
About noise in the darks, absolutely.
Any DR measure that doesn't take into account photosite size in relation to the final output is not useful in comparing cameras.
Regards,
Bill
-
Erik,
Tonal range in DxO is essentially DR with a different threshold on SNR.
Perhaps, but I doubt it.
In my experience if that were so the curves would drop off more quickly.
In any case, it's of no consequence to me.
I don't use DxOMark except for isolated gross cross-checks.
If you find DxOMark Tonal Range useful, that's great for you; I'm not trying to talk you out of it.
(I do admit that I am trying to talk others out of adopting it blindly as a worthy measure.)
Regards,
Bill
-
In any case, it's of no consequence to me.
I don't use DxOMark except for isolated gross cross-checks.
Hi Bill (did you know Guillermo = Bill? there are three of us here already), I also take DxOMark as a trusty reference. I mean, in many aspects it remains a bit as a black box, but at least they rely on RAW data and follow procedures that applied to different cameras should be fine to make comparisions.
Let me ask you something: this is the SNR response of a Pentax K5. Yellow are my measures (based on looking at StdDev in uniform colour patches, always over pure RAW data extracted using DCRAW), and the blue plot is Sensorgen plot based on their read noise and full well measures:
(http://www.guillermoluijk.com/article/snr/pentaxk5_modvsmed.gif)
I don't do any RAW data substraction to eliminate the effect of PRNU, and I think that is wht my plots get lower SNR for high exposure values. I really don't know how much PRNU affects DxOMark curves (I think they eliminate this effect and that is what their SNR near saturation improves by 3dB/EV), and hence Sensorgen summarized figures.
Can you think why specially my ISO80 measures could have a much lower SNR than an ideal expected PRNU-free response? ISO1600 on the other side is much more accurate. And both ISO80 and ISO1600 are fine in low exposure areas where the PRNU effect is not dominant.
Regards
-
Guillermo,
did you know Guillermo = Bill?
Yes :D
I think that your measurements of the Pentax K5 are probably accurate.
I question the DxOMark values, and by extension the Sensorgen derived values.
For example, the Sensorgen derived Read Noise for ISO 80 is almost certainly low.
A value of 3.8 or 3.9 would be more in line with the other ISOs.
DxOMark does not correct for PRNU, so that would not account for any difference.
Since you capture more light (photons, electrons) at ISO 80 than at ISO 1600, the effect of PRNU would be greater.
I would encourage you to trust yourself!
It is not "why is my data low?", it is "why is their data high?"
If you email the data that you have collected I would be happy to look at it.
Regards,
Bill
-
Do you know that anyone with a Windows PC can use my software to measure the PDR of their camera?
(So long it's a Nikon camera or they can make a linear DNG out of a raw file.)
Bill, it seems that your program (NefUtil) does not work w/ linear DNG files... out of curiosity I tried to convert a .NEF from D7000 to linear .DNG using Adobe DNG converter v6.5.0.218 - NefUtil says
"Warning: unsupported image format"
-
Bill, it seems that your program (NefUtil) does not work w/ linear DNG files... out of curiosity I tried to convert a .NEF from D7000 to linear .DNG using Adobe DNG converter v6.5.0.218 - NefUtil says
"Warning: unsupported image format"
Of course you known that NefUtil handles D7000 NEF directly and you're just testing NDG, right?
I think NefUtil is picky about the exact format.
With my DNG Converter I set Compatibility to "Custom", Backward version to DNG 1.3, Linear NOT checked, and Uncompressed CHECKED.
(I now realize I said linear earlier and should have said uncompressed !!! )
It is a pain, I admit. If more people want DNG support then perhaps I'll look into supporting it.
The other problem, with Canon and some others, is that NefUtil is largely unaware of non-zero offsets for black.
People sometimes send me files, and I "hack" this for them.
:(
-
(I now realize I said linear earlier and should have said uncompressed !!! )
I see - that was an issue.
-
It is a pain, I admit. If more people want DNG support then perhaps I'll look into supporting it.
It seems that your program is a good tool - so (converted or native) DNG support for other cameras (unless something like using libraw C++ library is possible to read the data from the raw files) will be an ideal solution (as you mentioned that there are issues w/ Canon and some other models)...
-
I think that your measurements of the Pentax K5 are probably accurate.
I question the DxOMark values, and by extension the Sensorgen derived values.
For example, the Sensorgen derived Read Noise for ISO 80 is almost certainly low.
A value of 3.8 or 3.9 would be more in line with the other ISOs.
DxOMark does not correct for PRNU, so that would not account for any difference.
Since you capture more light (photons, electrons) at ISO 80 than at ISO 1600, the effect of PRNU would be greater.
I would encourage you to trust yourself!
It is not "why is my data low?", it is "why is their data high?"
If you email the data that you have collected I would be happy to look at it.
OK, DxOMark don't eliminate PRNU. Sensorgen's 'read noise' and 'full well' capacity are derived from DxOMark plots (which have not been de-PRNU'ed). So what can we expect from those value pairs? it's difficult for me to try to track the consequences.
My plots based on Sensorgen are theoretical plots only assuming the read noise and full well terms in the noise model (http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Noise-characterization/Summary) as given by DxOMark. That is why they end in a perfect 3dB/EV slope near saturation.
My measures are totally independent from DxOMark. I calculated the exact saturation point (for the K5 this happened to be in the end of the 14-bit scale) and took this as reference 0EV. Then compared pairs (EV, SNR), where EV is how far the average signal in a given patch is from saturation, and SNR is noise's StdDev compared to the corresponding EV. All three channels were used equally to produce plot samples, each contributing at its RAW EV value achieved.
In this Excel file (http://www.guillermoluijk.com/download/ruidoyrangodinamico.xls) I collected all measures obtained with Rawnalyze. This article (http://www.guillermoluijk.com/tutorial/noisedr/index.htm) explains the general procedure.
The point is my DR figure (SNR=12dB criteria) matches well the value derived from Sensorgen SNR curve because in areas with low exposure values both sources provide similar SNR.
Regards
-
Guillermo,
I'm having trouble following the raw data you collected for the K-5, lines 94 and 95, Avg and StdDev.
For example, Column C Ave = 75 StdDev = 2.08
This data came from analyzing the same area of two images taken under identical conditions?
Ave is the average of the two areas added together?
StdDev is the square root of the variance of the subtraction of the two areas divided by 2?
Which channel is this?
Did you try to combine channels? If so, how?
(I admit I did not read the article yet. It's not in my first language ;D )
Regards,
Bill
-
I'm having trouble following the raw data you collected for the K-5, lines 94 and 95, Avg and StdDev.
For example, Column C Ave = 75 StdDev = 2.08
This data came from analyzing the same area of two images taken under identical conditions?
Ave is the average of the two areas added together?
StdDev is the square root of the variance of the subtraction of the two areas divided by 2?
Which channel is this?
Did you try to combine channels? If so, how?
Hi Bill, I think this image makes it clear:
(http://www.guillermoluijk.com/tutorial/noisedr/rawnalyze.gif)
Every pair (Average, Standard Deviation) comes from a single patch in a single channel. So every patch yields 3 values to plot the curve (one for every RAW channel).
Average is the mean RAW value in the patch, i.e. the Signal, and StdDev is the Noise, so we obtain a pair (RAW exposure, SNR).
I substract the black level and take into account the saturation level when needed.
Sensor linearity is assumed in order to locate each Average RAW value in stops with respect to saturation, so as uniformity between the 3 RAW channels (I don't have any reason to think a channel will produce a different SNR for the same number of accounted photons), that is why the samples come from the three channels without making any distinction nor combination.
Regards!
-
Guillermo,
OK.
I misunderstood.
When your said pair I assumed you were subtracting to remove PRNU.
(Even though you said earlier that you were not!)
Of course there are 4 channels, not 3; there are two green and they are not usually identical.
I would stick to one of the green channels.
Since you are not adjusting for PRNU I think the channels might not all behave exactly the same.
If you use all channels then you should at least record the channels separately to verify your assumption by examining the data.
Regards
-
Since you are not adjusting for PRNU I think the channels might not all behave exactly the same.
Yes, that check was implicit in the procedure. If you look at the plots all points are clearly scattered following the same trend. Should one of the 4 RAW channels behave differently, its points wouldn't produce similar pairs as the others, should be shifted or something.
Now I'm thinking that Rawnalyze must treat G1/G2 assuming they behave equally, since it only outputs a single value for G. In some cameras such as many 4/3 this is not true though, but in general it is.
-
Guillermo,
If you look at the plots all points are clearly scattered following the same trend. Should one of the 4 RAW channels behave differently, its points wouldn't produce similar pairs as the others, should be shifted or something.
You don't identify which points are from which channel so it's not clear that there is not a pattern of some sort.
Regards
-
You don't identify which points are from which channel so it's not clear that there is not a pattern of some sort.
I recall to have done just a quick visual inspection at that time (this was 2 years ago), taking dots from a single channel, and I couldn't see anything different from the other channels. If you look at the dots, they are very aligned with the trend, so I think the error margin is more than acceptable:
(http://www.guillermoluijk.com/tutorial/noisedr/curvassnrnorm.gif)
Larger errors surely come from not chosing a perfectly accurate black point (needed to consider the data linear in order to locate the dots in the X-axis), effect of the PRNU with good exposure,...
Regards
-
Good news ... and bad news ...
As it turns out you can data mine the DxoMark charts and get their data directly.
I've done it.
But the data has been heavily processed and extrapolated; the original data is lost.
For example, you can try an approach like www.sensorgen.info
Take the SNR data. Convert it to signal versus variance and do a 2nd degree fit.
The problem is, the values you get are often garbage, including negative values for PRNU.
In short, that data simply isn't reliable for sensor characterization.
Bill,
The DXO data may not be good for sensor characterization, but it does have some utility. The engineering DR may not be be useful for photographic purposes, but the photographic DR extrapolated by Emil's method does have some validity (I think).
To evaluate PRNU with my Nikon D3, I took duplicate shots from my monitor according to your method. I used a 300 mm lens at f/8 to minimize light falloff. I then separated out the green 1 and 2 channels using Iris and cropped the resulting files to the 201 x 201 pixel central area of the images and analyzed them with ImageJ, taking the mean and standard deviations of the images and then subtracting the identical images and dividing the resulting standard deviation by sqrt(2).
The histograms of a representative image and difference image are shown:
(http://bjanes.smugmug.com/Photography/PRNU/i-qcJDcND/0/O/1718histo.png)
Subtracting the images removed PRNU, and the resulting standard deviations represent shot noise and read noise, which add in quadrature. I then subtracted in quadrature the the SD of the image without PRNU from SDs of total noise (individual image SDs) to obtain the PRNU and then expressed PRNU as a percentage of the data numbers (DN). PRNU was 0.3% of the DNs. To avoid negative numbers in the subtraction, I added 400 to one image before subtracting the other.
Then I constructed a theoretical noise model using the full well and read noise from your data with and without a PRNU of 0.3% and calculating the SNR.
(http://bjanes.smugmug.com/Photography/PRNU/i-hVkHw82/0/O/D3Excel.png)
The resulting plot with PRNU is virtually identical to the DXO SNR plot. The Screen DR reported by DXO was also virtually identical to the engineering DR from your full well and read noise data.
(http://bjanes.smugmug.com/Photography/PRNU/i-BVCgZ53/0/O/D3Calc.png)
Your comments would be appreciated.
Regards,
Bill
-
Bill,
It's good work, but you miss my point.
You took measured D3 values, 63600 and 16.4512 from my site.
You personally measured a PRNU of .3% (I get .45% but this difference is not important for this discussion)
You found that the measured values produced a curve that is highly similar to the DxOMark curve.
However, if you started solely with the DxoMark data you would get D3 values of:
59200 rather than 63600 for FWC; about 7% difference
14.798 rather than 16.4512 for read noise in electrons; about 10% difference
If you took DxOMark data for an arbitrary camera you would get impossible read noise values for some cameras.
(And impossible PRNU as well, FWIW.)
In short, the DxOMark data cannot be relied upon for these normal sensor characteristics.
It's sad because they probably have good underlying data that we can't retrieve because it's been over cooked for presentation.
(Or intentionally to obfuscate it.)
I have attempted to clarify this with DxOMark using two different approaches and have had no response to either inquiry.
Regards,
Bill
P.S. - Have you received my emails?
-
Bill,
It's good work, but you miss my point.
You took measured D3 values, 63600 and 16.4512 from my site.
You personally measured a PRNU of .3% (I get .45% but this difference is not important for this discussion)
You found that the measured values produced a curve that is highly similar to the DxOMark curve.
However, if you started solely with the DxoMark data you would get D3 values of:
59200 rather than 63600 for FWC; about 7% difference
14.798 rather than 16.4512 for read noise in electrons; about 10% difference
If you took DxOMark data for an arbitrary camera you would get impossible read noise values for some cameras.
(And impossible PRNU as well, FWIW.)
In short, the DxOMark data cannot be relied upon for these normal sensor characteristics.
It's sad because they probably have good underlying data that we can't retrieve because it's been over cooked for presentation.
(Or intentionally to obfuscate it.)
I have attempted to clarify this with DxOMark using two different approaches and have had no response to either inquiry.
Bill,
I only have a few data points and can't check my data for consistency, so it is not surprising that I obtained a slightly different value for PRNU. The data collection by hand using the tools that I employed, is quite laborious and prone to error if one is not careful. Your automated method is much better.
Now I understand your point. Hitherto, most readers have accepted the DXO data as accurate. If they were truly to follow the scientific method, they would post detailed descriptions of how their measurements were obtained and how the calculations were performed. The raw data would be helpful, too. Otherwise, other observers can not repeat their experiments, as you have found.
Regards,
Bill Janes
-
Bill,
they would post detailed descriptions of how their measurements were obtained and how the calculations were performed
Yes, DxoMark makes a show of this, but fall short in execution (IMO).
Regards,
Bill
-
Bill,
...
However, if you started solely with the DxoMark data you would get D3 values of:
59200 rather than 63600 for FWC; about 7% difference
14.798 rather than 16.4512 for read noise in electrons; about 10% difference
If you took DxOMark data for an arbitrary camera you would get impossible read noise values for some cameras.
(And impossible PRNU as well, FWIW.)
In short, the DxOMark data cannot be relied upon for these normal sensor characteristics.
It's sad because they probably have good underlying data that we can't retrieve because it's been over cooked for presentation.
(Or intentionally to obfuscate it.)...
Yes hi Bill (all Bill's actually), This is interesting to me, as I have seen some similar differences. My name is Chris, and have had some disscussions with you on DPR in the past. My camera is the Nikon D40, and I have been doing some traditional Photon Transfer measurements on it using differenced flat fields. I varied exposure via exposure time, and found the sensor behaving quite linearly (unfortunately I don't know how to add charts) for DNs out vs light in, and also found the variance vs average DNs to behave linearly as well. I took several data points, including very low exposure levels in, which allowed me to extrapolate the variance line to the y-axis and get good read noise data. For evaluating the read noise, I used only flat fields that did not have more than 1% zeroed (or black clipped) data. Here are my results:
ISO k (e/DN) Nr (e) FW (e) PRNU %
200 7.10 13.84 28,381 0.57
400 3.53 11.41 14,095 0.564
800 1.83 11.53 7,277 0.59
1600 0.94 12.56 3,735 0.58
For this, I used a 100x100 central region, extracted the G1 channel via Iris, and evaluated statistics with ImageJ. k = camera gain constant, Nr = Read Noise, and FW = Fullwell Capacity. I compared my measured results with curve fitting to the Dxomark full SNR curves, which is shown below:
ISO k (e/DN) Nr (e) FW (e) PRNU %
200 6.53 10.81 26,088 0.404
400 3.00 8.05 11,995 0.177
800 1.54 8.53 6,170 -0.4
1600 0.84 10.06 3,372 -0.56
Some comments:
- for my measured results, PRNU stays relatively constant wrto ISO, which makes sense to me, however for the Dxomark data, the PRNU #s vary with ISO and sometimes go negative. (It is instructive to do the curve fitting, because Sensorgen doe not report PRNU).
- I understand that Nikon clips their blacks, but how do they do so? Do they:
- take say 100 dark frames at a short exposure time, average them together, and use this averaged dark frame to subtract
- subtract a constant DN level from each pixel
- use some other method
(I'm not against it, just curious, I'd think method 1 would be the cleanest, but I don't know)
- PRNU can be appreciable at low ISO, i.e. near saturation the shot noise is = sqrt(28,000) = 167 e, PRNU noise = 160 e, so I think camera makers should provide a button or function to do flat fielding if desired for getting some non-destructive NR.
Well, that's what I measured for my camera so far.
Chris
-
Chris,
Great work. I'm caught up on another project right now but want to comment at length when I have more time.
Some random comments.
Varying exposure time is a fine technique provided you don't accidentally cross one of the unavoidable NR boundaries; keep it faster than 1/4s.
Your k values look pretty good, clearly better than DxoMark, since 1/k should be linear with ISO.
My DxoMark numbers are slightly different. Probably I have more data than you.
My gain (1/k) numbers are similar to yours. Probably no more or less accurate.
You read noise in DN is slightly lower than mine. That's a surprise since mine are from "optical black". Perhaps sample variation?
Yes, PRNU would be independent of ISO.
I would guess that Nikon "clips" their blacks to the average optical black value, perhaps even at the row level.
If you use a PC it might be fun to cross-check against my NefUtil output.
Regards,
Bill
-
...
Varying exposure time is a fine technique provided you don't accidentally cross one of the unavoidable NR boundaries; keep it faster than 1/4s.
...
Ok thanks Bill,
I kept exposure time faster than 1/15 sec for all ISOs. I'd like to try using your NefUtil, ok to email you at your site?
Chris
-
Chris,
You can always email me for assistance or with questions, but NefUtil itself, and documentation are on my site under 'Downloads'.
Regards,
Bill