DxO Sensor Mark

[douglasf13]

Hi, Peter. Thanks for the article.

I wanted to mention that, the Canon S90, which is mentioned in footnote [27,] actually uses a Sony Exmor R sensor, so Sony incidentally has the best sensor score in compacts, as well.

[deejjjaaaa]

but DxOMark is benchmarking sensors

they are benchmarking raw files (postfirmware, not prefirmware), not sensors exactly...

[charleski]

There have been some discussion been around GH1 vs. GH2 and DxO data. According to Michael Reichmann the GH2 has better image quality than the GH1, or at least no worse.

DxO says that it would take about five DxO-mark points to tell image quality apart.

I've learnt to take the DxOMark data with a hefty pinch of salt. There are some review sites (eg DPReview) that provide RAW files taken under reasonably well-controlled conditions from the cameras they cover. Download some of them and see if they reflect the differences in noise indicated by the score.

You'd think that a score difference of almost 3 times the threshold they give would be fairly obvious. You'd expect that a camera that scored higher would not appear to have greater noise. Unfortunately, neither have been my experience. DxO's measurements certainly do seem impressive, but the results fail to correlate with what I see, and sometimes the failure is dramatic.

It's clearly a worthy endeavour on their part, but somewhere along the line something has gone wrong. I suspect that the problem really lies in the analysis and that the current models used for this aren't really up to the job. The DxOMark score should certainly not be used to assess a possible purchase (even on just the basis of sensor quality) without substantial cross-referencing with other comparative reviews.

[deejjjaaaa]

Unfortunately, neither have been my experience.

so post couple of raw files and describe your procedure to illustrate... huh ?

[ErikKaffehr]

Hi,

The Sony new Sony sensors have AD converters on chip and it is my understanding that they have an AD-converter on each column. Something like 6000+ converters on full frame (D3X, Alpha 900, 850).

My guess is that the converters are ramp type. Having a lot of parallel converters gives much longer measurement time for each individual converter.

That may also explain while the D3X is so much better than the Alphas. The DX-3 has a 14 bit pipeline while the Sony has only 12 bits. Nikon may use a longer integration time and achieve 14 bits with the same technology that Sony uses. Nikon D3X has a 14-bit mode which is much slower than 12-bit mode. That may be consistent with this theory.

Nikon D3 had external AD converters.

Best regards
Erik

Peter,
the DxO results have widely been dissected to extract three basic underlying sensor properties: quantum efficiency, full well capacity, and read noise. I can understand that you might have wanted to limit the scope of your article and not discuss them there. But maybe you comment here whether you agree with the common approach taken to extract these values.
I find it particularly interesting to see progress over time, sensor size, and sensor designer for these three properties. As I understand them, quantum efficiency (which somehow includes the fill factor) helps with low light noise but does not really affect dynamic range, full well capacity (naturally scaled for sensel size) affects dynamic range but not low light noise and read noise affects both dynamic range and low light performance.
What I am struggling a bit with is the relationship between read noise and noise in the amplification and the A/D converter. It is said that for cameras which do not have decreasing DR when going from minimum to moderate ISO, the DR is limited by the noise in the A/D converter and not the read noise at the sensel. It is also said that the latest Sony sensors have such a great DR performance (and that somewhat includes the D3x sensor) because of the column ADC which have very low noise levels.
If you have any insight on these issues, I would be very glad to here it.

Sekoya

[Peter van den Hamer]

the DxO results have widely been dissected to extract three basic underlying sensor properties: quantum efficiency, full well capacity, and read noise. I can understand that you might have wanted to limit the scope of your article and not discuss them there. But maybe you comment here whether you agree with the common approach taken to extract these values.

I am fine with this. I have some material on my website that looks at sensor noise bottom-up rather than top-down. I used the bottom-up info (from HarvestImaging.com aka Prof. Albert Theuwissen) as a check for the top-down data (DxOMark Sensor). See http://peter.vdhamer.com/2010/12/25/havest-imaging-ptc-serie/. The next major refinement to 'your' 3-param model might be temporal noise versus fixed-pattern noise. The PTC model as described by HavestImaging has about 10 parameters (depending on how you count, see my tables) and has example values for all elements. The reason they have more params is that they try to distinguish between row-noise, column-noise and pixel-noise (especially for fixed-pattern noise).

What I am struggling a bit with is the relationship between read noise and noise in the amplification and the A/D converter. It is said that for cameras which do not have decreasing DR when going from minimum to moderate ISO, the DR is limited by the noise in the A/D converter and not the read noise at the sensel. It is also said that the latest Sony sensors have such a great DR performance (and that somewhat includes the D3x sensor) because of the column ADC which have very low noise levels.

At this level of detail, I would cluster all the noise added to any sensel (regardless of the light level) as what you call read noise or my "noise floor". There is a pipeline of transformations/processing, and on the outside it is hard to distinguish which stage provides how much (temporal) noise. So I would include amplifier noise ("LNA noise"), quantisation noise, rounding errors, A/D noise, etc in read noise. But this kind of discussion about definitions and distinguishability require a model. The Havest Imaging model might help (I summarize it on my website), although it is a bit more detailed than what you might be looking for.

If you want to comment on this bottum-up stuff, you use the referenced posting on my site or even (if you get very detailed, might require background in Engineering ;-) comment on the originals posting series on HarvestImaging.com. Luminious Landscape is more about the end results and how to use them. Half of my DxOMark Sensor "essay" is already pretty far out for Luminous Landscape.

[Peter van den Hamer]

The Sony new Sony sensors have AD converters on chip and it is my understanding that they have an AD-converter on each column. Something like 6000+ converters on full frame (D3X, Alpha 900, 850).

My guess is that the converters are ramp type. Having a lot of parallel converters gives much longer measurement time for each individual converter.

That may also explain while the D3X is so much better than the Alphas. The DX-3 has a 14 bit pipeline while the Sony has only 12 bits. Nikon may use a longer integration time and achieve 14 bits with the same technology that Sony uses. Nikon D3X has a 14-bit mode which is much slower than 12-bit mode. That may be consistent with this theory.

It sounds pretty straight forward that at least 14-bit are needed now that the best cameras supposedly reach 14 stops of Dynamic Range. Off-chip ADCs sounds flimsy to me as it means transporting pretty high frequency analog signals off-chip: 25 Msamples in 0.1 s translating to a few GHz of analog bandwidth to get the resolution. I don't know whether column-level ADC is really the new magic ingredient. It does give a lot more time for conversion. But on my own Canon 5D2 body, it seems that fixed pattern noise is the main source of low-light noise. So my suspicion was that sensors were able to handle 14b ADC at 250 Msamples/s (maybe by having 4 or 8 // ADCs on-chip and 4 or 8 digital channels) and that recent innovations might be about efficient ways to subtract fixed-pattern row and column noise. But that is just a guess: DxOMark doesn't distinguish between temporal and FPN. And I haven't look up what the state-of-the art is in high-speed high-res ADCs.

See also http://harvestimaging.com/blog/?p=604&cpage=1#comment-11175 and my posting on peter.vdhamer.com about that tutorial by Albert Theuwissen.

[qwz]

Peter van den Hamer

I don't quite get your point. My article doesn't provide a reason why medium-format cameras (or digital backs) don't score too well.

Yes but IF science theory have too many  exception - may be we have something wrong with theory AND/OR with experimental data and methods we collect it.

Тevertheless i appreciate much your detailed explanation in this article on LuLa and this forum thread.

The GretagMacbeth color chart is a workable choice for such a "handful of colors". Agree?

Disagree. I don't clearly understand usefulness of this "handul colors" 'cause it suited only for photographer who shoot gretag macbeth target, 'cause all printed colors gamut (even with best printed technologies we have now) is much much tighter than real world. Also we have flat patches and test chart shot says nothing about subtle chromatic definitions between different lengths of light-waves.

Origin of skepticism is located in practical experience - i saw many files from cameras listed in dxo's rating on colour accuracy bad - it has better colors - especially subtle tones of colour than files from cameras with better dxo'x rating in this.

And i see, for practical purposes, dxo's rating says anything meaning only for shooting black and white photojournalist style.
Or for selling in photostocks, where technicians and mad for silky-smooth textures and pop poster colors (but you can always do smooth and pop - simply killing grain, detail and subtones and cannot make vive-versa in post-processing).

(it remembers me still debated topics about fuji vs kodak slide film quality - because lesser grain and better MTF doesn't automatically count as better image and bigger saturation doesn't always mean better colours tones - varieties of subtle chroma variations - things which makes image volume and authentic)

I almost  agree with other your points.

[qwz]

ErikKaffehr
Sony's chips for A700 and A900 (D300(s) and D3x in Nikon realisation) have possibility for multi-sampling reading to achive 14 or even 16bit ouput.  (8fps for 12 bit, 2.5fps for 14bit and about 0.7 fps for 16bit)
(similar to multisampling in scanners)

Sony don't use this but nikon do - because this 300 and 3x cameras slow down significantly in 14bit mode.

(I'll prefer it Sony have this toom but Sony decided to not).
 

[Peter van den Hamer]

Origin of skepticism is located in practical experience - i saw many files from cameras listed in dxo's rating on colour accuracy bad - it has better colors - especially subtle tones of colour than files from cameras with better dxo'x rating in this.

Qwz: Can you first clearly explain what you believe DxOMark Sensor is trying to measure with "Color Sensitivity" (or Color Depth or Portrait Score)? Never mind HOW they do it. WHAT are they trying to measure here?

If that is clear, you can meaningfully argue that that is not what you or others want to know. Or you can argue that it is measured in the wrong way. Peter

[Sekoya]

I am fine with this. I have some material on my website that looks at sensor noise bottom-up rather than top-down. I used the bottom-up info (from HarvestImaging.com aka Prof. Albert Theuwissen) as a check for the top-down data (DxOMark Sensor). See http://peter.vdhamer.com/2010/12/25/havest-imaging-ptc-serie/. The next major refinement to 'your' 3-param model might be temporal noise versus fixed-pattern noise. The PTC model as described by HavestImaging has about 10 parameters (depending on how you count, see my tables) and has example values for all elements. The reason they have more params is that they try to distinguish between row-noise, column-noise and pixel-noise (especially for fixed-pattern noise).

Thanks, I'll have a look at it.

[Sekoya]

Origin of skepticism is located in practical experience - i saw many files from cameras listed in dxo's rating on colour accuracy bad - it has better colors - especially subtle tones of colour than files from cameras with better dxo'x rating in this.

DxO is not measuring the colour accuracy, they are measuring how much colour variation you get on a flat field of exactly the same colour. A sensor (+ the raw conversion) can make a brown out of a red but as long as uniform red is rendered as a uniform brown the sensor is not adding noise in the form of colour variations.
Colour accuracy can really only be tested by combining the sensor and the raw converter. It is the raw converter which (tries to) ensure(s) that red stays red. And Capture One, the raw converter by and for Phase One, is known to be a rather accurate converter (supposedly the use more different colour sample for the calibration than other raw converters).

[deejjjaaaa]

DxO is not measuring the colour accuracy

DxO also makes a raw converter

[PierreVandevenne]

Interesting article indeed. But your summary http://peter.vdhamer.com/2010/12/25/havest-imaging-ptc-serie/ (and obviously its source material) is really great! Thanks.

[Peter van den Hamer]

DxO is not measuring the colour accuracy, they are measuring how much colour variation you get on a flat field of exactly the same colour. A sensor (+ the raw conversion) can make a brown out of a red but as long as uniform red is rendered as a uniform brown the sensor is not adding noise in the form of colour variations.
Colour accuracy can really only be tested by combining the sensor and the raw converter. [..]

DxO also makes a raw converter

True and true. Unfortunately raw converters are nowadays complex pieces of software (e.g. Adobe Camera Raw) integrated into even fancier programs (Lightroom, DxO Optics Pro). It is, in my opinion, best to avoid all that in order to decode the raw file in a well-defined way.
I tried guessing (earlier in this thread) what a bare-bones raw converter would do - especially if it should have minimal impact on noise, sharpness, contrast, and color:
" It would be fair to use a familiar demosaicing algorithm, no additional noise reduction, and the manufacturer's color matrix to convert from the raw color space to a standard (e.g. sRGB) color space. And then to analyze the measured noise values. "

That would just be a linear (3x3 matrix) transformation on the Raw files RGB values to handle the transmission spectra of the color filters. Serious color management software could try a lot harder (say XRite's Color Checker Passport, or Lightroom/ACR's built-in default camera profiles) based on measured color response and non-linear corrections. These are similar to what ICC profiles for scanners/monitors/printers do. Hints about the color matrix story can be found in the DxOMark "Insight" article where they compare the color sensitivity of the Nikon D5000 to the Canon EOS 500D.

[Peter van den Hamer]

FYI: I just updated the diagrams.

The differences are minor:

• On request I added 3 extra labels for cameras mentioned in the text (Fig 1b/c)
• It now covers 128 cameras (was 126).

In the process I messed up Fig 3 (timeline) a bit because my script automatically rescaled the axis due to a new model. Will fix this later.

Peter

[Peter van den Hamer]

Interesting article indeed. But your summary http://peter.vdhamer.com/2010/12/25/havest-imaging-ptc-serie/ (and obviously its source material) is really great! Thanks.

Thanks Pierre. Most of the credit indeed goes to Albert Theuwissen for the source material. I just deserve an aluminum medal for reading through all of it.

I changed the URL to http://peter.vdhamer.com/2010/12/25/harvest-imaging-ptc-series/. But the old URL still works (due to some automagic WordPress redirection). Note that in January 2011 Albert has started a follow-up on his series. I don't know yet what I will do with that: extend the original posting, start a new posting, or just leave things as they are. Will ask what Albert prefers - my posting does get some traffic (despite that my own DxOMark posting doesn't link to it yet, one was written before the other).

[ejmartin]

Can you first clearly explain what you believe DxOMark Sensor is trying to measure with "Color Sensitivity" (or Color Depth or Portrait Score)? Never mind HOW they do it. WHAT are they trying to measure here?

It is the number of distinguishable colors in the raw data output:

http://dxomark.com/index.php/en/Learn-more/DxOMark-database/Measurements/Color-sensitivity

basically the number of ellipsoids the size of a noise std dev deltaR*deltaG*deltaB that fit within the 'gamut' of the camera (the overall range of R,G,B it can record).

So it is a measure of color richness.   As a practical matter, it should be combined with information from the metamerism index (the degree to which the subspace spanned by the CFA spectral responses overlaps with that of the CIE standard observer) as well as the information that DxO measures on the map from color primaries of the camera to sRGB primaries.  For instance if there are large coefficients in the latter one will have larger chroma noise when mapped to standard output color spaces; see

http://dxomark.com/index.php/en/Our-publications/DxOMark-Insights/Canon-500D-T1i-vs.-Nikon-D5000/Color-blindness-sensor-quality

Basically large coefficients in the color matrix lead to amplification of chroma noise. 

[Peter van den Hamer]

[Color-sensitivity] is basically the number of ellipsoids the size of a noise std dev deltaR*deltaG*deltaB that fit within the 'gamut' of the camera (the overall range of R,G,B it can record).

Thanks. Actually the question was intended to get user <qwg> onto the same page. He was confusing color profile accuracy with DxO's color sensitivity because both happen to involve GretagMacbeth-like color charts.

As you sound more knowledgeable on this than I am, I have a real question (it is mentioned towards the end of the article): Why would DxO measure color sensitivity at low ISO rather than at high ISO? Wouldn't chroma noise normally be imperceptible at low ISO and thus irrelevant? Obviously the results at high ISO will tend to scale with the results at low ISO, but DxO explicitly states that they measure "maximum" color sensitivity.

[ejmartin]

Why would DxO measure color sensitivity at low ISO rather than at high ISO? Wouldn't chroma noise normally be imperceptible at low ISO and thus irrelevant? Obviously the results at high ISO will tend to scale with the results at low ISO, but DxO explicitly states that they measure "maximum" color sensitivity.

I think it is more intended to measure the number of distinguishable colors the camera can record, and so not per se a measure of chroma noise.  More like a chroma dynamic range, if you will, but not a range because it's a volume measure -- how many physically distinguishable bins of color there are in the 3D domain of (R,G,B) values.  So one would want to measure that at the lowest ISO where the number of bins is the largest.

I don't think there is really a useful quantitative measure of chroma noise for raw data.  One doesn't really have chroma data until after demosaic and transform to a color space, and that depends on a number of other factors (demosaic algorithm, transform method and input profile used, for example). For that matter, the noise in the raw data is not really luminance noise either.  It's just noise in the raw data.