Are the DXO Mark ratings reliable?

[sunshine1234]

Are the DXO Mark ratings a reliable, authority for objective information about the quality of photo gear?

[BernardLanguillier]

Are the DXO Mark ratings a reliable, authority for objective information about the quality of photo gear?

The first thing to be aware of is that DxO only focuses on sensor performance and completely ignores key aspects of camera performance such as AF and usability.

Their sensor measurements are accurate and reflect IMHO the real world performance of cameras.

This being said, their first page summary numbers are just that, summaries, and can lead to endless discussions. They typically retain only the best performing data point of a sensor and do therefore over emphasise low ISO performance over high ISO one. This tends to display some cameras in extremely good light (typically cameras using Sony/Toshiba sensors) and is probably unfair for Canon sensors that perform well at higher ISOs.

On the other hand, digging a little bit more gives access to detailed curves of each metric for each ISO that can be very useful to identify the strong/weak points of a given sensor in terms of dynamic range for example.

As a summary, DxO provides useful information for those with a bit of understanding of how camera sensors work.

Cheers,
Bernard

[allegretto]

DxO is ... great... if you don't think your eyes are a good guide to what you want to see

Vision by graph and robot

You decide I guess...

[ErikKaffehr]

Hi,

I would agree with Bernard.

The rating is based on three factors essentially:

1) Dynamic range - this is a signal processing term -
2) Colour depth
3) High iso capability

Cameras with great dynamic range are normally performing well on all tree factors. But it is possible to make a sensor that doesn't deliver great dynamic range but works well with high ISO.

The way I see it, almost all cameras are essentially good enough. So the DxO-mark is probable OK but may be no so relevant. It is interesting to study the measurements and it has generally been found that they are accurate.

I was on a workshop with Hans Kruse who is shooting both Canon and Nikon, what he says is that he doesn't need to expose optimally with the Nikon and still get good shadow detail, while Canon is more depending on ETTR. This is essentially exactly what DxO says. This applies at base ISO. With higher ISO-s the Canon catches up. It is visible in measurements but not in the rating.

Best regards
Erik

The first thing to be aware of is that DxO only focuses on sensor performance and completely ignores key aspects of camera performance such as AF and usability.

Their sensor measurements are accurate and reflect IMHO the real world performance of cameras.

This being said, their first page summary numbers are just that, summaries, and can lead to endless discussions. They typically retain only the best performing data point of a sensor and do therefore over emphasise low ISO performance over high ISO one. This tends to display some cameras in extremely good light (typically cameras using Sony/Toshiba sensors) and is probably unfair for Canon sensors that perform well at higher ISOs.

On the other hand, digging a little bit more gives access to detailed curves of each metric for each ISO that can be very useful to identify the strong/weak points of a given sensor in terms of dynamic range for example.

As a summary, DxO provides useful information for those with a bit of understanding of how camera sensors work.

Cheers,
Bernard

[Slobodan Blagojevic]

Of course they are. Numbers don't lie. Unless they rate OUR camera poorly, then we become Hegel-istas "If facts (numbers) contradict theory, so much worse for the ... facts."

They've been very fair to Canon: they said the latest sensor (in 7Dm2) is the same as Nikon's... from five years ago.

[synn]

DxO 's sensor ratings are very relevant to real world and are easily reproducible in real world scenarios. For example, canon shadows will lift nowhere as clean as sony sensors.

That said, there are many other factors such as handling of the camera, AF, lenses etc and even subjective matters such as color rendering, which cannot be deduced from their tests.

Look at them as one of the sources of information, not the only one.

[armand]

They dont rate my Fuji, does it mean it's no good ?! 

[Some Guy]

Wish they would rate the AF systems too.  They do need some work there.

They spend a lot of time testing lenses, but if the AF isn't up to par, drifts, has issues with colors of spectrum, minimal light standards, changes with aperture, focal lengths, etc., it's sort of pointless in the field.

SG

[ErikKaffehr]

Hi,

Fuji has a non bayer sensor which is not supported by the DxO raw converter. Here DxO-s explanation:

http://support.dxo.com/entries/22223617-Will-the-camera-Fuji-X-Pro-1-X-E1-X10-XF1-X-S1-X-Pro1-X100s-and-X20-be-supported-

So the reason is not it is not good but the reason is it is not supported by DxO-s raw conversion. The Sigma cameras are not supported and reported either, for similar reasons.

Another point is that DxO doesn't evaluate all cameras. What they evaluate is mainstream cameras and some MFD cameras. But only just a few MFD-s of recent designs have been tested. But, would they test any of the Pentax 645Z, Phase One IQ-250 or the CMOS based sensors from Leaf or Hasselblad, they all would go the top. Doubling the sensor size would give 10-15 points in the DxO mark rating.

Best regards
Erik

They dont rate my Fuji, does it mean it's no good ?!  

[Isaac]

Look at them as one of the sources of information, not the only one.

✓

[armand]

Hi,

Fuji has a non bayer sensor which is not supported by the DxO raw converter. Here DxO-s explanation:

http://support.dxo.com/entries/22223617-Will-the-camera-Fuji-X-Pro-1-X-E1-X10-XF1-X-S1-X-Pro1-X100s-and-X20-be-supported-

So the reason is not it is not good but the reason is it is not supported by DxO-s raw conversion. The Sigma cameras are not supported and reported either, for similar reasons.

Another point is that DxO doesn't evaluate all cameras. What they evaluate is mainstream cameras and some MFD cameras. But only just a few MFD-s of recent designs have been tested. But, would they test any of the Pentax 645Z, Phase One IQ-250 or the CMOS based sensors from Leaf or Hasselblad, they all would go the top. Doubling the sensor size would give 10-15 points in the DxO mark rating.

Best regards
Erik

Sorry, I should have put more than one smiley. I know why they don't do it, it was more like tongue-in-cheek.

[JV]

According to DxOMark the Leica M9 sensor was the worst full frame sensor they had ever tested.

The M Typ 240 was rated a lot higher and actually quite well received.

Does that mean that everybody prefers the M 240 files over the M9 files?  I didn't think so...

DxOMark gives an idea of the potential of the sensor but there's lot more, often very subjective, like rendering of lenses and colors...

[Abe R. Ration]

According to DxOMark the Leica M9 sensor was the worst full frame sensor they had ever tested.

The M Typ 240 was rated a lot higher and actually quite well received.

Does that mean that everybody prefers the M 240 files over the M9 files?  I didn't think so...

DxOMark gives an idea of the potential of the sensor but there's lot more, often very subjective, like rendering of lenses and colors...

If we were to perform a double blind test with M9 and M240, testing just the sensor/camera with the same lens and subject, it is hard for me to see more than a small minority prefering M9 over M240. The M9 sensor was not too good in the scope of full frame sensors when the camera came out, and it's even less good now (in that scope).

However, both sensors are just fine and capable of delivering very nice images in the right hands.

The DxOMark Overall Score for sensors is something one should not put too much weigth on - after all it's a subjectively decided arbitrary weighting of parameters. The measurement charts instead are very useful. If instead of the score we look at them, we'll notice that the peak performance of M9 sensor is not the worst of all full frame sensors they've tests - Canon 5D fares worse. Also The sensor resolution is ignored in these measurements - another area M9 beats 5D 

[sunshine1234]

Thanks Bernard and everyone else - very helpful to get all your views on the topic. I'm amazed at how unfavorable the results are for Canon...supposedly one of the worlds better brands. What the heck......!

[ErikKaffehr]

Hi,

As a small reflection, I used to have a Sony Alpha 900 and now I also have a Sony Alpha 99. The Sony Alpha 99 has significantly higher rating in DxO mark than the Alpha 900, and that difference is in all areas.

DR is better on the Alpha 99, but it took me three months and duping a Velvia slide to see the difference. The main reason is, I guess, that lens flare rather than sensor limits DR in real world shots.

High ISO capability is better on Sony Alpha 99, precisely as stated by DxO.

I can also mention that I was on a workshop in the Dolomites with Hans Kruse (recommended). Hans was shooting both Canon 5DIII and Nikon D800 (with or without E), as he wanted be able to support customers shooting both systems. I felt that he liked the Canon better but with Nikon he could push the shadows quite a bit. The consequence was that he needed to expose to the right or use HDR on Canon, while on Nikon ETTR was less critical.

So DxO data is relevant, but it says little about the cameras, just measures the image quality of the sensor.

Best regards
Erik

[Some Guy]

Probably not by any Leica owners.   

Leica has been taking a bad hit on their CCD sensors over the latest CMOS ones according to DxO Mark.  Mechanically they are fine cameras, but imaging maybe not so much compared to say a Nikon D810 etc.

http://www.dxomark.com/Reviews/Leica-M9-M9-P-and-M-E-Type-220-review-Ahead-of-the-new-Leica-M-we-round-up-the-DxOMark-Scores-of-its-predecessors/Conclusion

Their M 240 series scored better, maybe in the mid 80's now, but still waning.

SG

[Jim Kasson]

The DxOMark Overall Score for sensors is something one should not put too much weight on - after all it's a subjectively decided arbitrary weighting of parameters. The measurement charts instead are very useful. If instead of the score we look at them, we'll notice that the peak performance of M9 sensor is not the worst of all full frame sensors they've tests - Canon 5D fares worse. Also The sensor resolution is ignored in these measurements - another area M9 beats 5D  

I think you've put your finger on the most problematical part of the DxO testing: boiling down everything to a single number. As you say, whenever you convert a vector to a scalar, your weighting is open to question, as is the orthogonality of the axes. I know people like things simple, but taking a complex device like a camera, and one that can be used for many different purposes, like a camera, and attempting to assign a single measure of quality is a fool's errand.

If you look at all the tests, you have a better chance of finding out if any particular camera will meet your needs. There are some questionable results from time to time, but that's inevitable with this kind of testing, unless you have a dedicated laboratory, many samples of each camera, and a big staff.

I wish DxO would make their raw data available, and not just curve-fitted results for some of the tests.

Jim

[David Eichler]

It is my understanding that Dxo only use targets for the lens tests.  However, targets can only tell part of the story. It is my understanding that a high quality optical bench can provide important additional
information regarding lens performance.

I have no idea if their results are accurate or not, though I have seen no authority that
contradicts them. I will say that the results I seen on other websites with lens tests,
such as Photozone, seem to generally be consistent with the DxO findings.

[ErikKaffehr]

Hi,

There is nothing wrong with targets, except that they need to be large. A shooting distance like f x 50 is needed for correct testing of normal lenses, and that makes for a big room and large targets. Lenses intended for short ranges like portrait or macro should of course be tested a shorter ranges.

DxO tests are system tests, so they tell how a lens performs combined with a sensor and it's cover glass.

MTF tests on lenses alone give a lot of excellent information about the lens, but may ignore the cover glass that has significant effect on optical performance.

I would guess that the DxO tests are rather well designed, as they are basis for optical corrections in DxO raw converter.

Best regards
Erik

It is my understanding that Dxo only use targets for the lens tests.  However, targets can only tell part of the story. It is my understanding that a high quality optical bench can provide important additional
information regarding lens performance.

I have no idea if their results are accurate or not, though I have seen no authority that
contradicts them. I will say that the results I seen on other websites with lens tests,
such as Photozone, seem to generally be consistent with the DxO findings.

[Abe R. Ration]

I guess, that lens flare rather than sensor limits DR in real world shots.

I'm not sure I agree with that, though I don't have any hard evidence at this time for one way or the other. Under normal shooting conditions the modern lens coatings do a terrific job to prevent almost all reflections. Also this would mean that details would be washed out long before read noise had an impact (for most cameras), and I'm not sure that's the case. Now that I write, I think I should test it someday - thank you for giving me an idea!