DxO marks

[Peter van den Hamer]

Very detailed explanation, thanks.
But article says practically nothing about colour.

Here is a response that I wrote to a similar remark.
If you do a text search on the article, please use "color".

Peter

[Peter van den Hamer]

Here are some graph updates to cover new DxOMark data for

• Sony Alpha 99 : 89 ()
• Pentax K5 IIs : 82
• Canon 6D : 82
• Nikon 1 V2 : 50 (hmmm)

You may need to log in to see the figure.

[Peter van den Hamer]

Here are some graph updates to cover new DxOMark data for

• Sony Alpha 99 : 89 ()
• Pentax K5 IIs : 82
• Canon 6D : 82
• Nikon 1 V2 : 50 (hmmm)

You may need to log in to see the figure.

Last one (can attach max 4 files per posting).

[Ray]

On the other hand, when comparing noise level and DR at elevated exposure index, only SOS makes sense, because that compares at the same shutter speed when using the same aperture ratio in the same lighting. Pushing the DR and SNR 18% curves down to the left and thus down due to RAW files having more than 1/2 stop of highlight headroom (which DxO does) makes no sense when comparing low-light performance, so those graphs are best read by pushing the dots at ISO 200, 400, etc. back into alignment. Fortunately it seems that the "full SNR" curves as DxO are labelled with the camera's "ISO speed" settings or 200, 400, etc., so those can be compared without adjustment.

Not necessarily, BJL. Not all lenses used at the same aperture let pass the same amount of light. There's almost always some degree of light transmission loss which can vary by as much as the variance between the manufacturer's nominated ISO and the DXO-tested ISO.

For the purpose of testing ISO, DXO do not use a lens, otherwise the results would be all over the place. This is why too much nit-picking attention directed to small differences in ISO serve little purpose for the practical photographer who has to use a lens.

For example, Camera A has 1/3rd of a stop less actual sensitivity than camera B, but the lens used with Camera A may have 1/3rd of a stop lower transmission loss than the lens used with Camera B, which effectively cancels out any ISO differences.

On the other hand, if the lens used with Camera A has 1/3rd of a stop greater transmission loss than the lens used with Camera B, then that difference in ISO sensitivity is effectively doubled, and becomes 2/3rds of a stop, which is significant.

This problem of differences between F/stop and T/stop first came to my attention when a I bought the Nikkor 14-24/F2.8 with Canon adapter. The wide-angle zoom I'd previously been using, mainly with my 5D, was the Sigma 15-30.

Naturally, the first thing I did when receiving the Nikkor lens was compare it with my Sigma 15-30, at equal FoV shooting equal scenes with equal lighting, and using the same camera body, the Canon 5D.

I was of course gratified to see that this copy of my Nikkor lens was indeed quite noticeably sharper than the Sigma 15-30. But what astounded me was the discovery that the same exposure at F2.8 with the Nikkor resulted in the same ETTR using my Sigma 15-30 at its maximum aperture of F3.5. There appeared to be as much as a whole stop of difference between the T/stop values of these two lenses.

Whether or not the magnitude of this difference is really so great, I'd like to find out, just out of curiosity. After all, I was using an adapter with the Nikkor lens which may have influenced the results. I've searched the internet for T/stop values for this lens, but cannot find any reference whatsoever. Nor can I find any test of this lens on the DXOMark website.

Maybe the T/stop is so bad there's a conspiracy to keep it a trade secret. 

Of course, if I really wanted to waste my time exploring such issues, I could do more comparisons between the Nikkor zoom on my D700 and the Sigma zoom on my 5D, making allowances for the less-than-0.25 stop of difference in ISO sensitivity between the two cameras.

However, I see no practical purpose in such comparisons. The D700 has a base ISO of 200, at which ISO performance is at least equal to that of the 5D at ISO 100, and mostly better, especially DR which is a whole stop better. Problem solved! A difference of one whole stop of transmission loss is accommodated, and there's no way I would have gone back to using the Sigma zoom on my 5D in preference to the Nikkor zoom on my D700.

[Peter van den Hamer]

For the purpose of testing ISO, DXO do not use a lens, otherwise the results would be all over the place. This is why too much nit-picking attention directed to small differences in ISO serve little purpose for the practical photographer who has to use a lens.

Yes, both nominal sensor ISO and lens f-stop values are routinely too optimistic if you care about say 0.5 stop differences. But you can test a sensor's sensitivity with a lens with known T-stop and vignetting characteristics. DxO measures both values in their lens tests. So I guess that DxO uses a lens in some form or other (commercial lens, collimator) to measure ISO sensitivity. This reduces the amount of stray light that bounces around the room and even bounces around inside the camera.

Although I am perfectly willing to accept that definition ISO metrands can be a fine art (with a rather select audience), I would simply use the measured values from DxO rather than the nominal values specified by the manufacturers (as suggested by BJL). Firstly because it takes care of any weird behavior (like 50 ISO and 100 ISO having the same sensitivity on the 1Dx, or manufacturers cheating with high ISO values). Secondly because typical usage involves comparing measurements from the same lab rather than comparing data from different labs. The latter is too hard because it essentially involves assessing the actual data measurment protocols themselves. Simply comparing DxO results for product A and B should be more than accurate enough for end users. The same applies to results from other serious labs.

[Ray]

So I guess that DxO uses a lens in some form or other (commercial lens, collimator) to measure ISO sensitivity. This reduces the amount of stray light that bounces around the room and even bounces around inside the camera.

Peter,
I just checked the DXOMark protocols again to ensure my memory was correct. This is what they say about ISO testing.

Testing protocol for ISO sensitivity

The purpose of (saturation-based) ISO sensitivity measurement is to measure the exposure necessary to reach a given sensor's saturation point.

To measure the camera sensor’s ISO sensitivity, we set up the camera body alone (without a lens) on a stand to receive light from a controlled source.

The source is positioned far enough away from the camera sensor to ensure good light uniformity on the sensor plane. We then precisely measure the illuminance received by the sensor with a certified lux-meter.

[Peter van den Hamer]

DxO says "To measure the camera sensor’s ISO sensitivity, we set up the camera body alone (without a lens) on a stand to receive light from a controlled source."

You are right. And my guess was only partly right: when the lens cannot be removed, they are forced to test through the lens (and include some compensation for T-stop and vignetting) as I had expected. Unfortunately if you cannot separate lens from body, there doesn't seem to be a way to distinguish between the efficiency of the sensor and the light losses within the lens.

Thanks.

[Ray]

Unfortunately if you cannot separate lens from body, there doesn't seem to be a way to distinguish between the efficiency of the sensor and the light losses within the lens.

Hi Peter,
I'd wondered about that and also assumed that there'd be no way to distinguish between the efficiency of the sensor and the transmission efficiency of the lens if one couldn't separate lens from body.

However, what seems remarkable is that certain models of P&S cameras with fixed lenses seem to test exactly spot on with regard to ISO sensitivity according to DXO standards, which seems a bit of a coincidence.

For example, the nominated ISO values for the Canon Powershot S95 and the Panasonic FZ150 are exactly the same as the DXO tested values, with the occasional variance of just 1.

The Canon Powershot G15 goes against the trend and actually underestimates ISO sensitivity, according to DXO standards. It's nominated ISO of 100 is actually ISO 114, according to DXO tests.

Another issue I find puzzling is DXOMark's lack of any lens test results for the widely acclaimed Nikkor 14-24/F2.8. DXO have clearly tested this lens a long time ago, otherwise they wouldn't be able to provide the excellent 'volume anamorphosis' corrections for this lens in their RAW converter, and support for the lens with a number of camera bodies.

Why aren't they showing the test results? It's clearly a very popular and excellent lens.

[Bart_van_der_Wolf]

Hi Peter,
I'd wondered about that and also assumed that there'd be no way to distinguish between the efficiency of the sensor and the transmission efficiency of the lens if one couldn't separate lens from body.

However, what seems remarkable is that certain models of P&S cameras with fixed lenses seem to test exactly spot on with regard to ISO sensitivity according to DXO standards, which seems a bit of a coincidence.

Hi Ray,

Perhaps it is because with a fixed lens, the ISO is already compensated for known lens transmission losses. We've also seen such under-the-hood gain adjustment behavior for DSLRs when the Aperture value gets wider than f/2, which is when a small compensating amplification boost of the signal is detectable.

Cheers,
Bart

[Ray]

Hi Ray,

Perhaps it is because with a fixed lens, the ISO is already compensated for known lens transmission losses. We've also seen such under-the-hood gain adjustment behavior for DSLRs when the Aperture value gets wider than f/2, which is when a small compensating amplification boost of the signal is detectable.

Cheers,
Bart

Hi Bart,
If this is the case, then one wonders why the manufacturers of more serious cameras with detachable lenses, knowing that all their lenses have T/stop values which are numerically greater to some degree than the F/stop value, do not at least attempt to compensate for such transmission loss by understating the nominated ISO values instead of overstating them.

If one wishes to argue that such manufacturers of DSLRs are not overstating their nominated ISO values but correctly conforming to the ISO standard, perhaps in order to allow headroom for in-camera jpeg conversion, then why would the same manufacturer produce a camera like the S95, which is more likely to be used in jpeg mode, that conforms exactly to the DXO sensor-saturation method of measuring ISO?

Merry Christmas   

[Bart_van_der_Wolf]

Hi Bart,
If this is the case, then one wonders why the manufacturers of more serious cameras with detachable lenses, knowing that all their lenses have T/stop values which are numerically greater to some degree than the F/stop value, do not at least attempt to compensate for such transmission loss by understating the nominated ISO values instead of overstating them.

Hi Ray,

So to avoid confusion, they just leave them as they are, and DxO states them as they find them, no alterations to hide what has or has not been done. DxO do not need to normalize the readings at this stage, it would only remove information.

Cheers, and a merry Christmas to all,
Bart

[qwz]

Peter van den Hamer

Here is a response that I wrote to a similar remark.
If you do a text search on the article, please use "color".

Yes, thank you.
There i found this:

The ability to differentiate colors is called “Color sensitivity” in the article (see Figure . The full DxOMark measurements include much more detailed measurements of how the camera handles color. This includes the color gamut which the sensor can handle, the metamerism index, and the spectral behavior of the color filters.

It doesn't explain much, except simple counting of terms. Of course i want to believe that DXO uses most complicated and scientific bunch of methods, but how it exactly affect this strange so-called bits rating???
Moreover, now its only meaning that this MARKS says nothing practical about color except some relative things about unknown difference between cameras and sensors.
For example, previous generation full-frame cameras has drastically different colour gamut and fidelity (i mean D700, 5DII, A900) but it has less than half of so-called "Portrait (color depth) bits" difference. IF DXO measures only per-channel colour noise, why it is so close (cause in reality this three cameras handles noise completely different)??
Or DXO use in this calculations other things too. But this Rating says nothing about very perceivable differences between this cams (and others too).
And so on, and so on.

Maybe more honest to call article on such reputable resource as LuLa "DxOMark Camera Sensor Noise Explained" and tell readers, that third line of this Rating is meaningless?

[Peter van den Hamer]

Maybe more honest to call article on such reputable resource as LuLa "DxOMark Camera Sensor Noise Explained" and tell readers, that third line of this Rating is meaningless?

Actually, the article says

DxOMark Camera Sensor ratings essentially measure image noise and dynamic range.

and

Another reason why the term “Sensor” in the name of the benchmark can be a bit confusing is that the benchmark only covers the noise performance of the camera sensor.

The original article indeed had a subtitle "A noise benchmark of 183 digital cameras", but I dropped because LuLa templates don't really support subtitles.

Note that DxOMark contains extensive measurement information of the type you are interested in: select a camera -> Measurements -> Color Response (see also "Full CS" if you want to see gamut-related info). So DxOMark provides the data you are probably looking for - but not as part of their benchmark scores. I expect this is because color response or color accuracy is something that can fixed by software (e.g. Lightroom, by loading a "Camera Calibration profile" for your camera type or even your personal camera). So you can make colors as accurate or inaccurate as you wish (although it admit that good color control is tough).

Similarly, I doubt that characterizing the camera gamut more precisely than the  ISO standard 17321 color matrix and metamerism index is helpful. The topics are much more complicated than the stuff that I covered. And the impact on actual photography is again too dependent on software settings.

In other words, statements like "camera A has much more accurate or better-looking colors than camera B" simply may not be meaningful: it is fine if somebody prefers a particular set of color deviations (e.g. the look of Kodachrome or Velvia) but this can be achieved with appropriate software settings. If, on the other hand, somebody wants natural colors, this can be achieved by calibrating with e.g. XRite Color Checker Passport or searching for camera ICC profiles that you prefer.

tell readers, that third line of this Rating is meaningless?

DxOMark defines what it means. And it means something (chroma noise related) that is less fixable than color accuracy. Color accuracy can be fixed pretty well by calibration. Serious noise converters contain calibrations for major cameras. You can provide your own calibration if you want. Chroma noise, on the other hand, harms a (high ISO) image in ways that cannot be fixed without creating other problems.

[qwz]

Peter van der Hamer
OK

[ErikKaffehr]

Hi,

Just commenting a bit on the issue:

1) DxO mark is really about noise.

2) There is something called metamerism index in the DxO test, which describes how well the sensor can match the 16 color fields of the X-rite color checker.

If this is meaningful info or not can be discussed. Tim Parkin found the index coherent with his own observatiosn: http://www.onlandscape.co.uk/2012/02/the-myth-of-universal-colour/

Best regards
Erik

Very detailed explanation, thanks.
But article says practically nothing about colour.
DXO gives some points with decimals (strangely called 'bits' - but term bit means binary digit!) to colour but it appear to be a simple measure of noise of three colour channels, nothing more.
But, in practical terms, its only relevant Blue channel, cause it less sensitive in all modern cameras.
Moreover, DXO calls this strange points meaningful for portraits! But no one shoots portraits for per-channel looking for RAW data.

DXO simply fails to say something useful about colour. And i wonder why. Cause people buy megapixels and noise levels?

Turning back to colour, its more interesting and much more critical to know, firstly the spectral transmission of the lens (because DXO is only company testing both lens and sensors), and secondly actual spectral sensitivity of sensels (and colour gamut of cameras).
It differs much between vendors and models (but not samples of this expensive stuff:-).

Software correction fundamentally cannot helps with this problem (until we convert photographer to painter;-) because theres many (infinite number, actually) of different stimuluses in real world which appears on three-chromatic cameras sensor like the same R-G-B values.
Of course, improving tonal resolution can help but not much (and in modern cameras with at least 12bit processing). It happens because spectral sensitivity of colour filter arrays on sensels is not ideal and each channel has some small (or big) differences with spectral sensitivities of human eye (and other sensors too).

Does DXO points says something about it? Nope. Pity.
Does we have many other ways to measure noise of digital cameras with enough practical accuracy? Yes we have it on many websites (and can shoot themselves or ask people to make samples and share RAW files).
And, finally, what so "exciting" in DXO points - simple website database interface giving us illusion of knowledge? Not much.

[Peter van den Hamer]

..metamerism index in the DxO test, which describes how well the sensor can match the 16 color fields of the X-rite color checker.

Actually, metamerism represents to what degree colors with different spectra may map to the same RGB (etc) value. Once such "metameric failure" happens, no amount of calibration can distinguish the original two colors.

There are other metrics for how well colors represent say the 16 color patches in the Gretag-Macbeth chart (see delta-E in http://en.wikipedia.org/wiki/Color_difference).

This is roughly the level where my understanding of color management stops.

[ErikKaffehr]

Update: formula for SMI enclosed.

Hi,

This is what DxO-marks says about SMI: http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/Color-sensitivity

SMI: Sensitivity Metamerism Index
The sensitivity metamerism index (SMI) is defined in the ISO standard 17321 and describes the ability of a camera to reproduce accurate colors. Digital processing permits changing color rendering at will, but whether the camera can or cannot exactly and accurately reproduce the scene colors is intrinsic to the sensor response and independent of the raw converter.
The underlying physics is that a sensor can distinguish exactly the same colors as the average human eye, if and only if the spectral responses of the sensor can be obtained by a linear combination of the eye cone responses. These conditions are called Luther-Ives conditions, and in practice, these never occur. There are objects that a sensor sees as having certain colors, while the eye sees the same objects differently, and the reverse is also true.
SMI is an index quantifying this property, and is represented by a number lower than 100 (negative values are possible). A value equal to 100 is perfect color accuracy, and is only attained when Luther-Ives conditions hold (which, as previously stated, never happens in practice). A value of 50 is the difference in color between a daylight illuminant and an illuminant generated by fluorescent tubes, which is considered a moderate error.
More precisely, SMI is defined as
 ,
where  is the average CIELAB error observed on a set of various colors. In our experiments, we used the 18 colored patches of a Gretag McBeth color checker, as ISO 17321 recommends. The SMI varies depending on the illuminant.
In practice, the SMI for DSLRs ranges between 75 and 85, and is not very discriminating. It is different for low-end cameras (such as camera phones), which typically have a SMI of about 40. For this reason, we give this measurement as an indication but do not integrate it in DxO Mark.


Best regards
Erik

[qwz]

ErikKaffehr
Yes, Erik, thank you. Its exactly about i worried.
But two-digit index about conformity of only 24 color patches.
(Test target is printed and has much more narrower gamut).

[BJL]

Not necessarily, BJL. Not all lenses used at the same aperture let pass the same amount of light.

An issue that is irrelevant when using TTL light-metering. The only worry that I would have about the overstatement of a camera's sensitivity settings on a camera is if with comparable lenses and the same aperture ratios and apertures and lighting conditions, one camera uses a significantly longer exposure time than others when the cameras are at the same "ISO" sensitivity setting. The DxO measurements of raw level placement tell me nothing about that problem.

However, if you know the T-stops of your lenses and use an external non-TTL lightmeter and wish to use that information to place the midtones a certain number of stops below the maximum raw level, then I can see how the DxO saturation based sensitivity measurements could be useful to you.

Meanwhile, I am happy to agree with the CPIA and ISO that the measurements based on the Standard Output Sensitivity as defined by ISO standard 12232, and as used by most or all cameras makers, are valid and useful, especially for those of us who use TTL light metering most or all the time.

[bjanes]

Meanwhile, I am happy to agree with the CPIA and ISO that the measurements based on the Standard Output Sensitivity as defined by ISO standard 12232, and as used by most or all cameras makers, are valid and useful, especially for those of us who use TTL light metering most or all the time.

One should not assume that the SOS is used with his camera. The Nikon D800e uses REI (recommended exposure index). My 800e places the metered tone at 14% saturation, 2.82 stops below clipping. This is close to the 12.7% saturation of the old saturation standard and is in line with the DXO measured ISO of 73 for the nominal camera setting of 100. This value is 2.82 stops below saturation allows about 0.34 EV of headroom for the highlights. The saturation standard gives a saturation of 12.7%, allowing 0.5 EV of headroom for the highlights.

Regards,

Bill