Are the DXO Mark ratings reliable?

[ErikKaffehr]

Hi,

My assumption is based on two kind of observations, somewhat illustrated below. I generally see this kind of pretty sharp drop off on the left side. The other point is the P45+ image has a bit more exposure but the darks are still brighter on the SLT99. The explanation I have come up with is lens flare. The noise images are pushed 4 stops with sharpening but without noise reduction.

Raw images on this page: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/80-my-mfd-journey-summing-up?start=5

Best regards
Erik

Ps. Keep on posting! :-)

Phase One P45+, Distagon 40/4	Sony SLT99, 24-70/2.8 ZA

I'm not sure I agree with that, though I don't have any hard evidence at this time for one way or the other. Under normal shooting conditions the modern lens coatings do a terrific job to prevent almost all reflections. Also this would mean that details would be washed out long before read noise had an impact (for most cameras), and I'm not sure that's the case. Now that I write, I think I should test it someday - thank you for giving me an idea!

[Fine_Art]

Thanks Bernard and everyone else - very helpful to get all your views on the topic. I'm amazed at how unfavorable the results are for Canon...supposedly one of the worlds better brands. What the heck......!

They still make impressive lenses. Maybe they should become a lens company.

[allegretto]

Now we are getting to where I 'live" Eric... thank you so much for your exceptional contribution

THIS post of yours just says it all...

Here is an example of two images... the "mere" +45 (I only mean there are much "bigger" backs). And here we are looking at graphs and speculating on lens flare so the graphs make sense.

Just look at the pictures..... there is little doubt which camera I would use.I don't need ANY graphs, or log tables... or DxO marks...nothing...

Damn... if things were just a bit different, this post right here would have me obsessing over P1 backs and carriers/lenses

Thank you...!!!

Hi,

My assumption is based on two kind of observations, somewhat illustrated below. I generally see this kind of pretty sharp drop off on the left side. The other point is the P45+ image has a bit more exposure but the darks are still brighter on the SLT99. The explanation I have come up with is lens flare. The noise images are pushed 4 stops with sharpening but without noise reduction.

Raw images on this page: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/80-my-mfd-journey-summing-up?start=5

Best regards
Erik

Ps. Keep on posting! :-)

Phase One P45+, Distagon 40/4	Sony SLT99, 24-70/2.8 ZA

[Slobodan Blagojevic]

... The explanation I have come up with is lens flare...

Why lens flare? Isn't it simply greater dynamic range, i.e., the ability to show details/contrast in deeper shadows (in this case) and brighter highlights?

[ErikKaffehr]

Hi Slobodan,

I don't have time for a long explanation this evening, but I'll try to explain when I have some more time.

Perhaps some person brighter than us comes up with an even better (or less faulty) explanation.

Best regards
Erik

Why lens flare? Isn't it simply greater dynamic range, i.e., the ability to show details/contrast in deeper shadows (in this case) and brighter highlights?

[Glenn NK]

Are the DXO Mark ratings a reliable, authority for objective information about the quality of photo gear?

I think the answer is "it depends".

Thom Hogan recently wrote in part:

The latest round of DxO fretting comes with the release of their Canon 7DII numbers. One headline on another site put it this way: “Canon EOS 7D Mark II DxOMark test score: identical to the 5 years old Nikon D300s camera.” Uh, okay. That’s referring to DxOMark’s “overall score” value, which if anyone can clearly explain how they come up with that single number, let alone what it actually means in a pragmatic sense, I’d be appreciative.

The DxOMark Overall Score is one of those faux statistics that attempt to put a lot of test data together with a lot of assumptions and come up with a single representative numeric value. Quick way to know that it’s faux: what’s the difference you’d see in a camera labeled “70” versus “72”? Right, thought so.

Attempting to rate a sensor (let alone a camera) with one number seems a bit of a fools game to me.

It's like rating the most beautiful women in the world on a numeric scale from 1 to 10.  What's being measured?  Physical beauty (which varies from culture to culture), personality (again this varies with cultures), or some other metric?  Then, give a weight to each trait and then combine them into one single number.  There just might be a bit of disagreement.

[ErikKaffehr]

Hi Slobodan,

Here comes a short explanation, while I think it is lens flare.

To begin with, lens flare measurements usually give a figure of about 0.5%, that would limit DR 1/200 which is 7.6 EV. Normally, only a small part of the image is very bright so lens flare effects are mostly not so severe.

Now, lets look at the P45+ histograms. Please note that I use logarithmic Y-axis, which show small pixel counts well. You can see a spike, which I would guess corresponds to the sky. Blue and red channels are around +2EV and the red channel is at around +1EV.

Looking at the green curve you see that it starts dropping rapidly at -4.5EV and reaches 5K (5000 samples of that intensity) at around -5EV, but it continues to drop smoothly and there is no noise floor. Just a single count below -8EV. Now check the red curves, you see that it crosses 5K pixels at -6.5 EV. On the left, there are a lot of spikes. What I think we see is just a manifestation of shot noise in the darkest areas.


Now let's look at the Sony Alpha 99 image:
Here the blue and green peaks are about 1/3 EV left of the P45+ image, indicating 1/3 EV less exposure. The 5K limit is reached about the same time but the drop to the noise floor is much steeper. I assume that we have some lens flare causing this. In addition the dark areas that are pushed +4EV indicate the Alpha 99 image is brighter, although it has less exposure.



Now, there could be some other explanations than lens flare. The histograms here come from RawDigger so they are not affected by raw processing. Different tone curves could make the shadows differently bright in the raw processor.

It may be reasonable that the 24-70/2.8 ZA lens has a bit more flare than the Distagon 50/4 CF FLE, the zoom has 17 elelements in 13 groups, that is 26 reflecting surfaces, while the Distagon has 9 lenses in 8 groups, that is 16 surfaces. Both are T* coated. If we assume that coating reduces reflection to 0.5% it would mean that
0.995^26 * 100 = 88% of the light reaches the sensor and 12% bounces around in the lens. With the Distagon in would be 0.995^16 * 100 = 92% would reach the sensor and about 8% bouncing around in the lens.

It could be argued that present generation T* is better than T* 20 years ago, but it could be also argued that baffling reflected light may be easier in a prime lens than in a zoom with groups moving around.

BTW, it is always good to try to explain things, you see things in a different light.

Best regards
Erik

Why lens flare? Isn't it simply greater dynamic range, i.e., the ability to show details/contrast in deeper shadows (in this case) and brighter highlights?

[Abe R. Ration]

Since there are lenses with even 30 elements which have more (due to reflections of reflections of reflections of...) than 5 times more reflections than a prime with 6 elements and the very complex lenses seem to allow quite a bit contrast, it's hard for me to think lens flare is much of an limiting issue nowdays when it comes to DR.

I did set up a small experiement. I used Sony A7, CV 50/1.5, a brightly lit half subject and very dark other half of the subject. The details in both subjects in the extreme ends of the dynamic range of the subject were very faint to naked eye (actually impossible to see at all with naked eye in the bright end and difficult/almost impossible in the dark end), yet the camera/lens had no problems with the details and bringing them up. The dark part went right to the noise floor, while the saturated part to saturation point. Below are relevant crops (with pushing, pulling and relevant contrast adjustments).

I withdraw this post once as I really should have measured the dynamic range of the subject as well, but didn't think of it at the time, silly me. I may do that another day when not this busy, and also might try to repeat the test with a much worse lens (i.e. my worst which is the paper weight of old Olympus Zuiko 50/1. to see if there are relevant differences.

Anyhow, here's the picture.

[Abe R. Ration]

If we assume that coating reduces reflection to 0.5%

Modern coatings (for camera lenses) reduce reflection on 0.1% or less.

Older lenses are of course quite different.

[ErikKaffehr]

Hi,

I would think you are a bit optimistic on this, but I have no proof. I'd think the 0.5% figure comes from Zeiss and is pretty much confirmed by Imatest but I need to go back and check.

Best regards
Erik

Modern coatings (for camera lenses) reduce reflection on 0.1% or less.

Older lenses are of course quite different.

[Bart_van_der_Wolf]

Modern coatings (for camera lenses) reduce reflection on 0.1% or less.

Hi,

That's per air/lens-element or lens-group surface, so it still adds up ... In addition, it also depends on how the inner surfaces of the lens barrel are constructed and whether the lens edges are blackened. Then it also matters how the lens surfaces are shaped, since they act like mirrors and can produce hot-spots. Also the AR-coating of the filter-stack elements and sensor cover glass plays a role, including the mirror-box construction which receives the reflection of the filter-stack.

Cheers,
Bart

[peterottaway]

I agree that care needs to be taken with the DxO summary information but it is more useful than some would allow. Unless you are using only one or two lenses, at a single EI, of similar subjects in much the same light.

Although a lot of my photography can be described as landscapes / cityscapes / technoscapes .They cover a high variety of conditions and lenses from an 18mm prime to a 70 - 400 zoom in many varied conditions and using different EI - usually 100 to 800 when I can. The DxO scores can give me a good starting point should I need to compare competing cameras.

Not the only important point where sensor performance is reasonably close - not having 57 different buttons and wheels and a 100 pages of menu to arrive at my preferred setup and being able to use decent adapters to keep on using previous lenses. Or to be able to buy in other lenses in other mounts is a wonderful freedom to me.

[dwswager]

I think the answer is "it depends".

I think the answer is that their testing methods are sound, but you might disagree with their scoring system.  This is why Hogan recommends looking at the data and use your own scoring.

But Hogan's post was in reference to the 7DmkII versus D300s comparison.  Which is irrelevant as one is new and one discontinued.  But it doesn't change the fact that when compared to the current top of the line DX Nikon, the $850 less Nikon D7100, the Nikon is preferred.  I would love for Nikon to release a D7200 or D9300 or whatever with  similar frame rate and feature set.  

[Guillermo Luijk]

I have read a lot of people negating the validity and usefulness of DxOMark when their camera scores low. I have never seen anyone complaining about DxOMark when their camera scores among the best. The 'how my camera scores' variable usually weights more on many opinions than any objective argument.

Said that DxO is the most reliable source today for ranking camera sensors. It has a great advantage over any real world testing: they are compairing pears with pears in (nearly) all cameras in the market. People usually do flawed comparisions (even with a good intention), and no one has access to all cameras DxO has.

[Abe R. Ration]

I would think you are a bit optimistic on this, but I have no proof. I'd think the 0.5% figure comes from Zeiss and is pretty much confirmed by Imatest but I need to go back and check.

You may be right - it's quite hard to find figures from manufacturers. The 99,9% I got from Canon after some digging. I do thing you're right!

That's per air/lens-element or lens-group surface, so it still adds up

Yes of course,  but 99,9% adds up a lot slower than 99,5%

I just checked out a couple of more complex lenses from DxOMark and their T-stops, and indeed it seems like my 99,9% was a bit optimistic, at least outside of the very high end.

... In addition, it also depends on how the inner surfaces of the lens barrel are constructed and whether the lens edges are blackened. Then it also matters how the lens surfaces are shaped, since they act like mirrors and can produce hot-spots. Also the AR-coating of the filter-stack elements and sensor cover glass plays a role, including the mirror-box construction which receives the reflection of the filter-stack.

I had never thought of the shapes of the surfaces, interesting!

[Ray]

I have read a lot of people negating the validity and usefulness of DxOMark when their camera scores low. I have never seen anyone complaining about DxOMark when their camera scores among the best. The 'how my camera scores' variable usually weights more on many opinions than any objective argument.

Said that DxO is the most reliable source today for ranking camera sensors. It has a great advantage over any real world testing: they are compairing pears with pears in (nearly) all cameras in the market. People usually do flawed comparisions (even with a good intention), and no one has access to all cameras DxO has.

Absolutely right, Guillermo. I've always had the impression that those who criticise the validity and accuracy of DXO results are simply trying to subjectively defend their own choice of a camera model which doesn't score well in the DXO tests.

Rather than carry out their own comparisons, where practical, to confirm the accuracy of the DXO claims, at least approximately, they prefer to live in a state of denial.

Now of course, it would be too difficult for most photographers to attempt to confirm the mathematical ratings for SNR and DR in terms of dB or EV and so on, with respect to a single model of camera, but that's not required.

All that's required is access to two models of camera that have been tested by DXO, the skill to take  ETTR shots with both cameras, and the skill to process RAW images. For example, if one wants to compare the dynamic range of two different camera models, one shoots a very high contrast scene and compares shadow noise, ensuring that both images under comparison are ETTR shots. To estimate the number of stops (or EV) of DR advantage that one camera has over another, one needs to take a number of shots of different exposures. If it is claimed that camera 'A' has 2 EV better DR than camera 'B', at a specific ISO setting, then a shot from camera 'A' which is 2 stops underexposed (with respect to a correct ETTR exposure) should have approximately the same shadow noise and detail as the ETTR exposure from camera 'B'.

[BernardLanguillier]

Rather than carry out their own comparisons, where practical, to confirm the accuracy of the DXO claims, at least approximately, they prefer to live in a state of denial.

Yep. Many photographers hate to be told by bystanders that they own a great camera because of the perceived underlying assumption that the great camera plays an important role in the quality of their images.

Yet, some of those participating in such threads who happen to own a "technically inferior" camera (measured for the sake of discussion by its DxO measurements results), often go to great length to deny the technical inferiority of their tool, as if they feared an association between that technical inferiority and the (lack of) "photographic quality" of their images.

Showing more confidence in their photographic skills would probably remove the need to defend their camera.

I sort of had more fun when I was shooting with a D2x while the world was using a 1DsII. It never crossed my mind to question the fact that the 1DsII was a technically much superior camera (within the intrinsic differences of APS-C vs FF that are not as clearcut as it seems) but it made it fun to try to extract the last bit of quality from the D2x. Stitching already saved the day at the time.  







Go figure.

Cheers,
Bernard

[allegretto]

I have read a lot of people negating the validity and usefulness of DxOMark when their camera scores low. I have never seen anyone complaining about DxOMark when their camera scores among the best. The 'how my camera scores' variable usually weights more on many opinions than any objective argument.

Said that DxO is the most reliable source today for ranking camera sensors. It has a great advantage over any real world testing: they are compairing pears with pears in (nearly) all cameras in the market. People usually do flawed comparisions (even with a good intention), and no one has access to all cameras DxO has.

Not limited to cameras however

cars, audio systems, clothing, food, travel, hotels, just about anything that gets "rated" by anyone is bound to create cognitive dissonance if you're on the short end, and care that you're on the short end...

Kind of human nature... as Hogan notes in another thread here along these lines. No one likes to feel "inferior" and some will do quite a bit of twisting to be off the spot.

[shadowblade]

Yep. Many photographers hate to be told by bystanders that they own a great camera because of the perceived underlying assumption that the great camera plays an important role in the quality of their images.

Yet, some of those participating in such threads who happen to own a "technically inferior" camera (measured for the sake of discussion by its DxO measurements results), often go to great length to deny the technical inferiority of their tool, as if they feared an association between that technical inferiority and the (lack of) "photographic quality" of their images.

Showing more confidence in their photographic skills would probably remove the need to defend their camera.

I sort of had more fun when I was shooting with a D2x while the world was using a 1DsII. It never crossed my mind to question the fact that the 1DsII was a technically much superior camera (within the intrinsic differences of APS-C vs FF that are not as clearcut as it seems) but it made it fun to try to extract the last bit of quality from the D2x. Stitching already saved the day at the time.  







Go figure.

Cheers,
Bernard

Off-topic:

Is that Siguniangshan/Dafeng and Huanglong near Chengdu, Sichuan?

[BernardLanguillier]

Off-topic:

Is that Siguniangshan/Dafeng and Huanglong near Chengdu, Sichuan?

Yes, exactly! Well spotted.

That was 8 years ago. I was back in the area 2 years ago, but only went to Jiyuzhaigou that time.







Cheers,
Bernard