Are the DXO Mark ratings a reliable, authority for objective information about the quality of photo gear?
The first thing to be aware of is that DxO only focuses on sensor performance and completely ignores key aspects of camera performance such as AF and usability.
Their sensor measurements are accurate and reflect IMHO the real world performance of cameras.
This being said, their first page summary numbers are just that, summaries, and can lead to endless discussions. They typically retain only the best performing data point of a sensor and do therefore over emphasise low ISO performance over high ISO one. This tends to display some cameras in extremely good light (typically cameras using Sony/Toshiba sensors) and is probably unfair for Canon sensors that perform well at higher ISOs.
On the other hand, digging a little bit more gives access to detailed curves of each metric for each ISO that can be very useful to identify the strong/weak points of a given sensor in terms of dynamic range for example.
As a summary, DxO provides useful information for those with a bit of understanding of how camera sensors work.
Cheers,
Bernard
They dont rate my Fuji, does it mean it's no good ?! ;D
Look at them as one of the sources of information, not the only one.
Hi,
Fuji has a non bayer sensor which is not supported by the DxO raw converter. Here DxO-s explanation:
http://support.dxo.com/entries/22223617-Will-the-camera-Fuji-X-Pro-1-X-E1-X10-XF1-X-S1-X-Pro1-X100s-and-X20-be-supported-
So the reason is not it is not good but the reason is it is not supported by DxO-s raw conversion. The Sigma cameras are not supported and reported either, for similar reasons.
Another point is that DxO doesn't evaluate all cameras. What they evaluate is mainstream cameras and some MFD cameras. But only just a few MFD-s of recent designs have been tested. But, would they test any of the Pentax 645Z, Phase One IQ-250 or the CMOS based sensors from Leaf or Hasselblad, they all would go the top. Doubling the sensor size would give 10-15 points in the DxO mark rating.
Best regards
Erik
According to DxOMark the Leica M9 sensor was the worst full frame sensor they had ever tested.
The M Typ 240 was rated a lot higher and actually quite well received.
Does that mean that everybody prefers the M 240 files over the M9 files? I didn't think so...
DxOMark gives an idea of the potential of the sensor but there's lot more, often very subjective, like rendering of lenses and colors...
The DxOMark Overall Score for sensors is something one should not put too much weight on - after all it's a subjectively decided arbitrary weighting of parameters. The measurement charts instead are very useful. If instead of the score we look at them, we'll notice that the peak performance of M9 sensor is not the worst of all full frame sensors they've tests - Canon 5D fares worse. Also The sensor resolution is ignored in these measurements - another area M9 beats 5D ;D
It is my understanding that Dxo only use targets for the lens tests. However, targets can only tell part of the story. It is my understanding that a high quality optical bench can provide important additional
information regarding lens performance.
I have no idea if their results are accurate or not, though I have seen no authority that
contradicts them. I will say that the results I seen on other websites with lens tests,
such as Photozone, seem to generally be consistent with the DxO findings.
I guess, that lens flare rather than sensor limits DR in real world shots.I'm not sure I agree with that, though I don't have any hard evidence at this time for one way or the other. Under normal shooting conditions the modern lens coatings do a terrific job to prevent almost all reflections. Also this would mean that details would be washed out long before read noise had an impact (for most cameras), and I'm not sure that's the case. Now that I write, I think I should test it someday - thank you for giving me an idea!
| Phase One P45+, Distagon 40/4 | Sony SLT99, 24-70/2.8 ZA |
| (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/Histograms/P45+.jpg) | (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/Histograms/SLT99.jpg) |
| (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/20131117-CF044323.jpg) | (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/20131117-_DSC3262.jpg) |
I'm not sure I agree with that, though I don't have any hard evidence at this time for one way or the other. Under normal shooting conditions the modern lens coatings do a terrific job to prevent almost all reflections. Also this would mean that details would be washed out long before read noise had an impact (for most cameras), and I'm not sure that's the case. Now that I write, I think I should test it someday - thank you for giving me an idea!
Thanks Bernard and everyone else - very helpful to get all your views on the topic. I'm amazed at how unfavorable the results are for Canon...supposedly one of the worlds better brands. What the heck......!
Hi,
My assumption is based on two kind of observations, somewhat illustrated below. I generally see this kind of pretty sharp drop off on the left side. The other point is the P45+ image has a bit more exposure but the darks are still brighter on the SLT99. The explanation I have come up with is lens flare. The noise images are pushed 4 stops with sharpening but without noise reduction.
Raw images on this page: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/80-my-mfd-journey-summing-up?start=5
Best regards
Erik
Ps. Keep on posting! :-)
Phase One P45+, Distagon 40/4 Sony SLT99, 24-70/2.8 ZA (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/Histograms/P45+.jpg) (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/Histograms/SLT99.jpg) (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/20131117-CF044323.jpg) (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/20131117-_DSC3262.jpg)
... The explanation I have come up with is lens flare...
Why lens flare? Isn't it simply greater dynamic range, i.e., the ability to show details/contrast in deeper shadows (in this case) and brighter highlights?
Are the DXO Mark ratings a reliable, authority for objective information about the quality of photo gear?
| (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/20131117-CF044323_vsmall.jpg) | (http://echophoto.dnsalias.net/ekr/Articles/MFDJourneyEOY/Noise/20131117-_DSC3262_vsmall.jpg) |
Why lens flare? Isn't it simply greater dynamic range, i.e., the ability to show details/contrast in deeper shadows (in this case) and brighter highlights?
If we assume that coating reduces reflection to 0.5%Modern coatings (for camera lenses) reduce reflection on 0.1% or less.
Modern coatings (for camera lenses) reduce reflection on 0.1% or less.
Older lenses are of course quite different.
Modern coatings (for camera lenses) reduce reflection on 0.1% or less.
I think the answer is "it depends".
I would think you are a bit optimistic on this, but I have no proof. I'd think the 0.5% figure comes from Zeiss and is pretty much confirmed by Imatest but I need to go back and check.You may be right - it's quite hard to find figures from manufacturers. The 99,9% I got from Canon after some digging. I do thing you're right!
That's per air/lens-element or lens-group surface, so it still adds upYes of course, but 99,9% adds up a lot slower than 99,5% ;)
... In addition, it also depends on how the inner surfaces of the lens barrel are constructed and whether the lens edges are blackened. Then it also matters how the lens surfaces are shaped, since they act like mirrors and can produce hot-spots. Also the AR-coating of the filter-stack elements and sensor cover glass plays a role, including the mirror-box construction which receives the reflection of the filter-stack.I had never thought of the shapes of the surfaces, interesting!
I have read a lot of people negating the validity and usefulness of DxOMark when their camera scores low. I have never seen anyone complaining about DxOMark when their camera scores among the best. The 'how my camera scores' variable usually weights more on many opinions than any objective argument.
Said that DxO is the most reliable source today for ranking camera sensors. It has a great advantage over any real world testing: they are compairing pears with pears in (nearly) all cameras in the market. People usually do flawed comparisions (even with a good intention), and no one has access to all cameras DxO has.
Rather than carry out their own comparisons, where practical, to confirm the accuracy of the DXO claims, at least approximately, they prefer to live in a state of denial.
I have read a lot of people negating the validity and usefulness of DxOMark when their camera scores low. I have never seen anyone complaining about DxOMark when their camera scores among the best. The 'how my camera scores' variable usually weights more on many opinions than any objective argument.
Said that DxO is the most reliable source today for ranking camera sensors. It has a great advantage over any real world testing: they are compairing pears with pears in (nearly) all cameras in the market. People usually do flawed comparisions (even with a good intention), and no one has access to all cameras DxO has.
Yep. Many photographers hate to be told by bystanders that they own a great camera because of the perceived underlying assumption that the great camera plays an important role in the quality of their images.
Yet, some of those participating in such threads who happen to own a "technically inferior" camera (measured for the sake of discussion by its DxO measurements results), often go to great length to deny the technical inferiority of their tool, as if they feared an association between that technical inferiority and the (lack of) "photographic quality" of their images.
Showing more confidence in their photographic skills would probably remove the need to defend their camera. ;)
I sort of had more fun when I was shooting with a D2x while the world was using a 1DsII. It never crossed my mind to question the fact that the 1DsII was a technically much superior camera (within the intrinsic differences of APS-C vs FF that are not as clearcut as it seems) but it made it fun to try to extract the last bit of quality from the D2x. Stitching already saved the day at the time. ;D
(https://farm1.staticflickr.com/117/291034889_eacf9398ff_o.jpg)
(https://farm1.staticflickr.com/184/462703260_50695687d1_o.jpg)
(https://farm1.staticflickr.com/107/291021806_46bb18e012_o.jpg)
Go figure.
Cheers,
Bernard
Off-topic:
Is that Siguniangshan/Dafeng and Huanglong near Chengdu, Sichuan?
Absolutely right, Guillermo. I've always had the impression that those who criticise the validity and accuracy of DXO results are simply trying to subjectively defend their own choice of a camera model which doesn't score well in the DXO tests.
Canon (because that's what you really mean) files are absolutely glorious at the image level, and nothing DxO and its fanboys have to say about it changes that fact.
Or do you seriously believe that Canon users don't care about image quality?
Arrant nonsense, Ray. People balk at DxO's outpourings because the evidence of their own eyes belies any real World legitimacy their nonsense is supposed to have.
Canon (because that's what you really mean) files are absolutely glorious at the image level, and nothing DxO and its fanboys have to say about it changes that fact.
Or do you seriously believe that Canon users don't care about image quality?
They get it in spades from Canon bodies. Despite what DxO might have to say about it.
... do you seriously believe that Canon users don't care about image quality?...
What a silly, silly question! Of course we do! That is precisely why we are so pissed off with Canon for not providing it at the state-of-the-art level. Instead, Canon presented yet another incarnation of the rotary phone model, very capable in its own right, though. It is just that the world has been using mobile phones for a while, you know ;)
Canon (because that's what you really mean) files are absolutely glorious at the image level, and nothing DxO and its fanboys have to say about it changes that fact.
What a silly, silly question! Of course we do! That is precisely why we are so pissed off with Canon for not providing it at the state-of-the-art level. Instead, Canon presented yet another incarnation of the rotary phone model, very capable in its own right, though. It is just that the world has been using mobile phones for a while, you know ;)
Can anyone put a printed Canon image and a printed Nikon image side by side and say which is which?
But back to DxO - even they do not make any claims about the meaningfulness of their results. They simply provide the data and let people argue over them. For example what does a difference of 75 and 73 really mean in practice?
Can anyone put a printed Canon image and a printed Nikon image side by side and say which is which?
But back to DxO - even they do not make any claims about the meaningfulness of their results. They simply provide the data and let people argue over them. For example what does a difference of 75 and 73 really mean in practice?
Then I bit the bullet, and bought a 1dsII, a 1dII, and five lenses. I kept my Nikon stuff. I enjoyed those Canons for more than a year. The Nikon announced the D3. I switched back.
Ouch! That was an expensive exercise! You bought a 1dsII, a 1dII, and five lenses, and used them for less than a couple of years? ;
IMO they should get rid of the overall DxOMark score - it's based on subjective weightings and doesn't tell you anything useful about a sensor.
Stick to the actual data and the objective derived measurements
IMO they should get rid of the overall DxOMark score - it's based on subjective weightings and doesn't tell you anything useful about a sensor.
I don't mind the overall rating as long as you understand what it is.
Stick to the actual data and the objective derived measurements (DR vs SO, col our depth vs ISO, efficiency, SNR) which actually tell you how a sensor performs.
And to understand what it is ..
Its a composite of it's best DR, Color Depth and ISO performance.
Exactly, it's a composite - jack of all trades and master of none.
I guess that I am more interested in Bernard's images than in the DXOMark ratings. As it should be.
I guess that I am more interested in Bernard's images than in the DXOMark ratings. As it should be.
Exactly, it's a composite - jack of all trades and master of none.
I find the numbers a lot more useful.
Photos are dependent on the skill and resources of the photographer. Just because photographer A produces great (or terrible) photos with camera A doesn't mean that photographer B will produce great (or terrible) images with camera A.
In contrast, the numbers and data tell me what each camera is and isn't capable of, and which piece of equipment will give me better results given the same shooting style and the same photographer.
Early photographers produced great work using primitive bodies and mediocre lenses. Blow them up to the size of a large print and you can quickly see all the technical limitations - lack of resolution, detailless shadows and blown-out highlights. Give the same photographers an IQ280, a set of Rodenstock lenses and a copy of Photoshop and they'd probably produce much better work - same compositional quality, but with much better detail and technical quality.
I guess that I am more interested in Bernard's images than in the DXOMark ratings. As it should be.
Hi,
There are a few problems with images:
- You don't know the processing that was done on the image
- Images are normally shown in small sizes
- The images are normally shown as JPEGs which contain much less data than the original raw, 8-bits/channel
It is not easy to do proper tests, and it is easier to them under lab conditions than under real world conditions. AFAIK I know the DxO data are very accurate, but they essentially measure sensor noise.
DxO also does lens testing, and again they have a quite obscure rating. But they also have a lot of measurement data. My lens purchases are essentially almost always based on lens tests.
That's essentially what I said - looking at photos taken by different people using different methods doesn't tell you much about the sensor.
That is what lab test like DxOMark bring to the table. They give you clues as to performance ceiling.
However, looking at real world photos gives you clues as to what can be done with the image. What it won't give you is the how to get there! And, of course, you must look at pictures in the output mode you intend. If you're target output is large scale printing, looking at 640 x 480 px screen images isn't gonna help much! Image making today, even Ansel Adams showed this in his day, isn't solely about what comes out of the camera. You want the best possible data out of the camera because it keeps more possibilities open, but the image making process extends through post processing and output. Give an expert in after camera image handling a less than stellar image file and he is likely to produce a final image better than most can with a stellar image file. Which is why he wants that stellar file to begin with as he can do more with it.
Erik mentioned using lens tests as the bases for lens purchases. I do as well, but not for selection. My method is to identify my need, Identify candidate options. Then I look at lens tests to 1) see if there is just some stellar performer in the options, 2) See if something just way out performs it's price point, 3) Identify lenses that look good on paper, but just perform poorly. From the remaining candidates, I then try to use test data to see how their lab performance might translate to my real world shooting style. For example, if I'm using it for landscapes, then wide open performance won't matter as much, and I might trade off some other characteristics for wide open performance. Finally, I look at ergonomics, functionality, durability even filter size. I standardized on 77mm awhile back for ND and CPL so if I get 2 similar options that I'm trying to decide between and 1 is 77mm and the other isn't then it might tip the balance. As a Nikon shooter, I am very pleased that both the 16-35mm f/4G VR and 18-35mm f/3.5-4.5G are both 77mm as are the 24-70mm f/2.8G and the 24-120 f/4G VR.
Do you mean you'll decide the purchase of one or another camera system based in how good are the pictures produced by their owners?. Or you simply meant that you enjoy more looking at pictures than looking at graphs and numbers?. If this is the case I'll tell you enjoying pictures is totally compatible with the usefulness of DxO data in deciding a purchase.
Simply the fact that they do the testing at only one aperture and one zoom setting and do a subjective IQ appraisal for their "score" says it all.
Simply the fact that they do the testing at only one aperture and one zoom setting and do a subjective IQ appraisal for their "score" says it all.