DxO bashing - a bit of perspective

[ErikKaffehr]

Hi,

DxO mark takes a lot of bashing, which is seldom justified. DxO makes a commercial raw converter and they essentially share some of the data they acquire for each sensor. Sharing information is normally regarded a good thing, except in China and some other countries.

There are a few things to keep in mind regarding DxO-mark for sensors:

- It is based on raw images
- It is mostly about image noise
- DxO mark essentially ignores pixel count (data are normally normalised to 8 MP, but this is largely irrelevant)
- DxO mark measures colour depth but not colour fidelity (well, they measure SMI but it is not a part of DxO-mark)

The DxO-mark is based on three factors:

- Dynamic range, which weakly favors large pixels
- Colour depth, which is the number of colours the sensor can reproduce
- Low light ISO, the highest ISO setting that gives a good image, this also weakly favors large pixels

A key difference of the ISO rating is that it favors cameras with variable pre amps, as it ignores DR. This is the reasons Canons shine in this area.

Sensor area influences DxO-mark heavily, doubling the sensor area probably adds something like 10-12 points to the total DxO-mark.

None of the top eight cameras are "high ISO cameras", one of the top eight is the Phase One IQ 180. The others have all Sony Exmoor sensors and deserve their position mostly due to DR (dynamic range).

In the colour depth category the Phase One cameras dominate, all first three place go to Phase One. This is more due to the sensor size than CMOS vs. CCD or CFA design.

Once DxO-mark tests the IQ-250 it will very probably go to the top, by a wide margin. It is using a larger version of the Sony Exmoor sensor. The new RED Dragon sensor may be higher in rating, though, but it is probably not regarded to be a still camera.

Best regards
Erik

[mjrichardson]

Erik

You make a huge amount of posts on the technicalities of this great user of our time and money, but there are as many different types of photographer as their are manufacturers, resources and styles. Bashing a resource is as absurd as defending it in my opinion, it's an organisation that runs a very specific set of tests to produce a very specific set of results that are important to a specific set of people.

My opinion, for what it's worth, is that photography in the main is a creative process, the look, message, style of any image produced is down to the vision of the photographer using the equipment at his or her disposal. The results from DXO testing give a very very small amount of important data to help photographers who value that information make decisions on what they want to buy, to others it means nothing. A lens is not great because of a high DXO mark, a camera is not great because it has a high DXO mark, they just have specific qualities that meet DXO's testing standards, there is so much that is not shown by these sorts of tests.

So, don't take it to heart that people don't take the same amount from these tests that you do, it's important to the people it's important to, you may think people are crazy for bashing the information, others may think you're crazy for showing charts when talking about a lens rather than beautiful images made with it, it's all good just different people with different requirements.

Enjoy your photography!

Mat

[barryfitzgerald]

DxO need to explain how they test more so people can understand it.
As with any test there is a margin for error.

My view is that is has limited use in the field for photographers so I don't really take it that seriously.
It's a bit like a computer benchmark..that's not always a great indicator of real world performance.

[Paulo Bizarro]

Thanks for the sum up. In my (simple) mind, what I take from this is that larger sensors naturally "win" in terms of dynamic range and colour depth. This is important to me, as I shoot a lot of landscapes-seascapes. And that is why I still use a FF DSLR for such subjects.

Rgds

[peterottaway]

Eric, I use the DxO software as the my basic processing software however their testing means very little to me as I rate colour fidelity and MP as two of my basic starting criteria. After all what the camera can produce is the limits of its performance and it is all down hill from there.That is why Canon produce better results in the tests than in my world.

I am using these as the starting points along with micro contrast for the inter-reaction between my cameras and lenses and the real world conditions that I have to actually take photographs in.

The DxO way to me is the same sort of technical mumbo jumbo as the ISO criteria for film speed.

[barryfitzgerald]

Thanks for the sum up. In my (simple) mind, what I take from this is that larger sensors naturally "win" in terms of dynamic range and colour depth. This is important to me, as I shoot a lot of landscapes-seascapes. And that is why I still use a FF DSLR for such subjects.

Rgds

Depends which full frame you use. If it's Canon then there are plenty of APS-C DSLR's with better dynamic range.
The problem with DxO is putting into perspective. numbers rarely mean much.

For example how to they come to their low light ratings? Image quality is subjective and most of us would have a print size target (at least I do) for high ISO images. But quality can vary here, IQ degrades how do DxO come up with these numbers?

It's all a bit vague for my liking.

[marcmccalmont]

I think DxOMark is one of many tools used to make a "smart" purchase
Before buying a camera or lens one reads dpreview, photozone etc to be informed.
Just one more intelligent metric to make an intelligent purchase decision.
Marc

PS: I'm going to purchase a new car, I read technical reviews, I read driving and functional reviews then I go to the dealer for a test drive and make a "smart" decision
So I decide on a Mazada Miata with skinny tires because it is fun to push on back roads at rational speeds over a Porsche 911 GTB. The Porsche came out high in the technical reviews, the Miata high in the functional reviews but at least I read the reviews and test drove the cars before writing the check!

[Paulo Bizarro]

Depends which full frame you use. If it's Canon then there are plenty of APS-C DSLR's with better dynamic range.
The problem with DxO is putting into perspective. numbers rarely mean much.

I use a Canon 6D and I am quite happy with it. Previously, I have used a 5DMKII. Never had a shot ruined because of lack of DR, but I do take some time to optimise exposure on the field, even sometimes using GND filters to balance "sky and land", or "sky and sea". I am currently working on some photos I took in the SW Portugal coast, these are sunsets over the sea, with some coastal cliffs as foreground. I had to use a GND to hold down the sky a bit, I suppose I am "old school" in that regard. Never had to up the shadows in post by much, so no problems there.

I have experience with good APS-C sensors, like Pentax K5 and Fuji XE-1 or X-Pro1, or X100, I have used them in the past too (still use the X100 for travel). What I can say is that with these cameras, I had to resort to highlight recovery sooner and more often, in Lightroom. The Canon 6D files have more room to play with. Are there better sensors out there for FF? I believe so, for sure, but the Canon FF is "good enough to me".

Rgds

[Telecaster]

IMO it's important not to succumb to simpletonistic interpretations of the info provided by any testing process. Personally I'd prefer that DxO not provide a single-number score for people to glom onto. But since they do you've gotta go past that and look at the more in-depth analysis. I consider their data useful but not determinative...as Marc observes, after the reading comes the test drive.

Sometimes I'll end up prefering a less technically capable camera over a more capable one due to ergonomic issues. Or maybe a "lesser" camera effortlessly delivers a pleasing or distinctive look that I have to wrestle with a "better" camera to achieve. A single number isn't gonna tell you about that stuff. DxO themselves are surely aware of this...sometimes I wonder, though, about segments of their readership.

-Dave-

[Dave Ellis]

I think DXOMark provides some useful information for those who are interested althought I too am not keen on their scoring systems. I'd prefer to make my own mind up based on the measurements. I like their sensor information, not so keen on their lens data. I can't fathom their Perceptual Mpix concept and would prefer to see basic mtf50/lwph info across the frame - just like the way dpreview used to present their data.

Dave

[ErikKaffehr]

Hi,

I pretty much agree with your view. I like the data but I feel the skeptical about the scoring system.

Best regards
Erik

I think DXOMark provides some useful information for those who are interested althought I too am not keen on their scoring systems. I'd prefer to make my own mind up based on the measurements. I like their sensor information, not so keen on their lens data. I can't fathom their Perceptual Mpix concept and would prefer to see basic mtf50/lwph info across the frame - just like the way dpreview used to present their data.

Dave

[Keith Reeder]

The scoring system is exactly the problem - guided, as it is, by some arbitrary and undisclosed weighting of the average of the three sub-scores it produces, a factor which unquestionably introduces an (unintentional? maybe) bias in favour of cameras that do well in one particular metric.

The tests themselves are inherently skewed in favour or specific use-cases which might well be entirely unrepresentative of the use to which many of us put our cameras - the very fact that many of their tests and their sensor scores are weighted towards 100 ISO, when many of us never use our cameras at 100 ISO, is a bias.

And as to lenses: did you know that the Canon 50mm f/1.8 II is a "better" lens than the Canon 600mm f/4 II based on DxO's scoring?

It's also true that DxO has made verifiable mistakes with its tests, but instead of publicly admitting to these errors, have quietly edited/fixed them unannounced, which is not very "scientific" for a company with the phrase "image science" in its logo tagline...


But the real problem with DxO's scores of course, is the use to which they're put by trolls trying to "prove" that their choice of camera is superior to another's, because DxO has scored its sensor higher.

Thanks for the sum up. In my (simple) mind, what I take from this is that larger sensors naturally "win" in terms of dynamic range...

No.

As Barry has pointed out, this is not intrinsically true, because it ignores improvements in sensor technology's "state of the art" - in the Real World, recent crop cameras often significantly outperform older full frame sensors in terms of dynamic range (shadow DR, anyway, which is all that anyone seems to be interested in these days) and high ISO noise handling.

[Keith Reeder]

- Low light ISO, the highest ISO setting that gives a good image

Huge problem right there, Erik: a "good" image by what criteria? To what standard?

A perfect example of the arbitrariness of the whole DxO process; and given that you can't have a good - or bad - high ISO image until the Raw data are actually converted into an image (and the conversion stage is vital to the end result, as we'd all agree), a completely misleading and disingenuous false equivalence - you cannot establish, purely from analysing Raw data, whether there's a good high ISO image in there.

I find this point particularly ironic, given just how damn' good DxO's Optics Pro is at converting high ISO images - DxO knows it's not telling the whole "low light" story here.

[ErikKaffehr]

Hi,

Raw images have measurable noise, just open a raw file with raw digger mark an area and it will measure the noise. It is measuring ISO that requires a calibrated light source.

A good description of DxO mark is published here: http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras.shtml
it is worth reading.

The formal definition of the "low light maximum ISO" is:

Low-Light ISO is then the highest ISO setting for the camera such that the Signal-to-Noise ratio reaches this 30dB value [32:1 ratio at 18% middle grey] while keeping a good Dynamic Range of 9 EVs [512:1 ratio] and a Color Depth of 18 bits [roughly 64×64×64 colors].

and

Low-Light ISO is normalized to an arbitrary reference resolution of 8 MPixels

The values may be arbitrary, but they are equal to all. Normalising the data to 8 MPixels means that resolution doesn't affect the rating.

DxO makes all these measurements, so they can make a good raw converter. They feel that this is a relevant way to present data. It is anyway more relevant than manufacturers data which may say 25000 ISO but say nothing about the resulting image quality.

Personally I almost never look at the DxO mark but look at the individual measurements.

A good sensor will rate well in all categories. A sensor with high readout noise will be hurt in the landscape rating. Canon's have high readout noise at low ISO, but it improves at high ISO because the signal from the sensor is amplified more before raw conversion.

Regarding the lens comparison, the 600/4 is sharper than the 50/1.8II according to DxO data, so I don't understand the rating.

I always check DxO data on lenses I want to buy, if available. It has saved me some money and a lot of frustration, I guess.

Best regards
Erik

Huge problem right there, Erik: a "good" image by what criteria? To what standard?

A perfect example of the arbitrariness of the whole DxO process; and given that you can't have a good - or bad - high ISO image until the Raw data are actually converted into an image (and the conversion stage is vital to the end result, as we'd all agree), a completely misleading and disingenuous false equivalence - you cannot establish, purely from analysing Raw data, whether there's a good high ISO image in there.

I find this point particularly ironic, given just how damn' good DxO's Optics Pro is at converting high ISO images - DxO knows it's not telling the whole "low light" story here.

[peterottaway]

One of the good points about these discussions is that it can bring out your generally unspoken assumptions about your photography. So whether you agree with the points raised or not is often not that important.

Keith Reeder states that he doesn't use ISO 100, that is my default EI setting. Some people will use Auto ISO with or without a manually adjusted limits, I usually don't.And even when I do, I will limit its upper speed to EI 400 unless it is a sport in demanding circumstances. Doesn't make me right or wrong just different.

But when words such as equalized or normalized or adjusted for a standardized output are used, it all becomes BS to me. Whether its DxO or DPR or anyone else.

[ErikKaffehr]

Hi,

Normalisation is a pretty smart thing. It says how things will look in print or for that part on screen at normal sizes.

Let's say that we happen to have same generation technology, but we have large pixels in one camera like the Sony A7s, while a camera like the Sony A7r has three times as many pixels. Pixel for pixel the A7s will blow the A7r away in noise, DR and ISO. On the other hand when printed the 36 MP Sony A7r image will be interpolated down to same size as the 12 MP A7s image, three times that many pixels will be used, and almost the entire per pixel advantage of the A7s will be compensated. DR gains a bit less when downsampling.

So the normalisation stuff makes it possible to compare sensors of different pixel count, while the unnormalised values correspond essentially looking at the image at actual pixels.

Best regards
Erik

But when words such as equalized or normalized or adjusted for a standardized output are used, it all becomes BS to me. Whether its DxO or DPR or anyone else.

[Paulo Bizarro]

No.

As Barry has pointed out, this is not intrinsically true, because it ignores improvements in sensor technology's "state of the art" - in the Real World, recent crop cameras often significantly outperform older full frame sensors in terms of dynamic range (shadow DR, anyway, which is all that anyone seems to be interested in these days) and high ISO noise handling.

Well, of course comparisons only make sense if we talk about concurrent sensors. Of course recent crop cameras have better performance than FF cameras of the past.

[Keith Reeder]

Well, of course comparisons only make sense if we talk about concurrent sensors. Of course recent crop cameras have better performance than FF cameras of the past.

Yeah, but people don't, Paulo: there's a routine presumption that any FF sensor is inherently "better" than any crop sensor, purely because its a FF sensor.

So it's not a matter of "of course..." - some folk really don't get this.

[Keith Reeder]

Hi Erik,

yeah, I understand completely that "noise" is objectively measurable: but what about subjective notions like noise "quality"?

It's entirely possible to have an image with a higher measurable noise value that still results in a "good" image (possibly a subjectively better image than from a camera with a smaller measured noise level) because the noise itself is (say) small-grained and unobjectionable, or "film-like": so I still balk at the idea of objectively quantifying something like the amount of noise and then using that metric alone - the quantity - as a determinant of high ISO image quality.

Again, until you're looking at the converted image, it's not indicative of anything meaningful to the end result.

A good converter can be a tremendous equaliser, and different converters render noise in very different ways, even if the amount of noise is objectively the same in all of them...

My basic point is that you cannot say "there's noise xx at ISO yyyy, therefore the image quality will be zz..."

[allegretto]

Erik

thank you for your insights "under the hood" on these test paradigms. I have come to learn much from you

But would never buy a lens or system based on these narrow tests

Rent the lens and body you like. If you like the look, buy it. Otherwise send it back to Roger. Once I settle on a body.. rinse and repeat for the lenses

As noted above by another 6D user, I have never felt inhibited by it's DR. In my 240 as well. It's about how the image appeals to me. I really don't care about "tests"

When I raced cars we never did road tests. We decided what class, got a vehicle and optimized it in real time… nothing was theoretical. All real results. Works for cameras too to some extent