Is DxO Mark on the mark?

[marcmccalmont]

Something has been bothering me about DxO Mark rankings and Michaels review of the A900 made me think that a fourth factor, resolution is missing from these ratings. Resolution counts, for a given size print the more resolution the more realistic the fine detail, i.e. foliage, hair etc. Perceived detail is a function of the pixels in each dimension or the square root of the pixel count. So I took DxO mark rankings and used it for 3/4 of the weighting and added 1/4 of the square root of the resolution and re-ranked the top 9 cameras. The formula I used was ((DxO Mark x 3) + (20 x sqrt(mpixel count)) / 4
10 mpx = 63.2, 12 mpx = 69.2, 13mpx = 72,  15 mpx = 77.4,  17mpx = 82.4,  21 mpx = 91.6,  24mpx = 98

Rankings with resolution as a factor:
1. A900   = 83.675
2. 1DsIII = 83.125
3. D3      = 77.75
4. D700   = 77.675
5. 1DsII  = 76.1
6. D90    = 71.75
7. 5D      = 71.175
8. Gx20  = 70.875
9. 1DIII  = 69.05

A ranking closer to what photographers are seeing! check my math too!
Marc

[ErikKaffehr]

Hi,

DxO Mark is absed on small prints. It does take resolution into account as a decrease of noise when the image is scaled down to reference size. I'm highly skeptical of cooking everything into a single figure of merit. There is a lot of data in their graphs which is quite useful, however.

Erik
 

Something has been bothering me about DxO Mark rankings and Michaels review of the A900 made me think that a fourth factor, resolution is missing from these ratings. Resolution counts, for a given size print the more resolution the more realistic the fine detail, i.e. foliage, hair etc. Perceived detail is a function of the pixels in each dimension or the square root of the pixel count. So I took DxO mark rankings and used it for 3/4 of the weighting and added 1/4 of the square root of the resolution and re-ranked the top 9 cameras. The formula I used was ((DxO Mark x 3) + (20 x sqrt(mpixel count)) / 4
10 mpx = 63.2, 12 mpx = 69.2, 13mpx = 72,  15 mpx = 77.4,  17mpx = 82.4,  21 mpx = 91.6,  24mpx = 98

Rankings with resolution as a factor:
1. A900   = 83.675
2. 1DsIII = 83.125
3. D3      = 77.75
4. D700   = 77.675
5. 1DsII  = 76.1
6. D90    = 71.75
7. 5D      = 71.175
8. Gx20  = 70.875
9. 1DIII  = 69.05

A ranking closer to what photographers are seeing! check my math too!
Marc

[fike]

Hi,

DxO Mark is absed on small prints. It does take resolution into account as a decrease of noise when the image is scaled down to reference size. I'm highly skeptical of cooking everything into a single figure of merit. There is a lot of data in their graphs which is quite useful, however.

Erik

Great work!  I found the absence of resolution pretty bothersome too. DXO should incorporate your idea.  I like to print big, and resolution matters.  I also like to crop, and this makes resolution matter.

...but the level of complexity involved in looking at multiple factors and then creating a scoring system to weight those factors is beyond the grasp of most people, including most photographers who consider themselves advanced camera users.  

I think the best way to evaluate a camera is to taste it like wine.  Hold it up to the light.  Smell it.  Put it in your mouth, swirl it around a bit.  Spit or swallow.

For complicated purchases like a camera (automobile, computer, house, spouse, etc...) your first purchase isn't likely to be perfect.  Each subsequent purchase will result in getting closer to your needs, but you will always be trading one attribute for another.  

I'm sorry that I always go off on this rant, but  I am sick of people trying reducing complex issues to scales, weighting and numbers.  Sometimes it can't be done.  But more importantly, it definitely can't be done universally.  Each person possesses needs and wants in different proportion to others.  DXO tries to do it universally.  That is misleading and bad for entry-level photographers who will all be deluded into thinking they need an A900 or 1DsIII or whatever.  Those cameras may not meet their needs, and certainly fail to consider cost as a factor in the value of a camera.  Would you recommend a 1DsIII to a beginner.  I don't think anyone would, no matter how much money someone had to throw into photography.  

These are great cameras that DXO is rating highly, but we could probably go on forever identifying attributes of a camera that aren't reflected in these numbers, most notably perhaps is ergonomics, oh and there is weather sealing, and lens selection, and size, and weight, and blah, blah, blah...

I could go on, but I will restrain myself.

sorry.

[marcmccalmont]

I also took the ranking weighted by resolution and divided by retail price to get a value rating:

• GX20  669
• D90    581
• 5D      356
• D700   288
• A900   278
• D3      181
• 1DIII   170
• 1DsIII 109
• 1DsII  170

Numbers do paint a pattern for evaluation although you are right there are a lot of intangibles to evaluate

Marc

[marcmccalmont]

!!!!!

[feppe]

5D MkII would end up at rank #3 with 82,15, right after 1DsIII.

[Ray]

Is it not assumed that the resolution of the sensor will always largely be determined by its pixel count, at least at base ISO? Is there really a need to test for the obvious? There will inevitably be slight variations depending on the strength of the AA filter and perhaps other factors, but I imagine that such variations will be overshadowed by differences in RAW converters and most of all, differences in the quality of lenses used.

What would be useful is a resolution test at high ISO because that's where the resolution differences between cameras of similar pixel count are likely to be significant. I found it interesting, for example, that dpreview claimed that the resolution of the 5D2 at ISO 3200 was a whisker less than the resolution of the D700 at ISO 3200, and not just with jpeg images, but RAW conversions also.

[BernardLanguillier]

Rankings with resolution as a factor:
1. A900   = 83.675
2. 1DsIII = 83.125
3. D3      = 77.75
4. D700   = 77.675
5. 1DsII  = 76.1
6. D90    = 71.75
7. 5D      = 71.175
8. Gx20  = 70.875
9. 1DIII  = 69.05

Marc,

Is the D3x in a category of its own?

Cheers,
Bernard

[Mark D Segal]

I also took the ranking weighted by resolution and divided by retail price to get a value rating:

• GX20  669
• D90    581
• 5D      356
• D700   288
• A900   278
• D3      181
• 1DIII   170
• 1DsIII 109
• 1DsII  170

Numbers do paint a pattern for evaluation although you are right there are a lot of intangibles to evaluate

Marc

Marc, I don't think this is a viable approach. I started-off the exercise which Michael published this evening doing something very similar - one formula which creates a single score for each camera in which a camera does better the higher the resolution, the better the sensor and the lower the price. Such formulae are easy enough to conjure and compute, but when I sat back to think about it, I decided that they embed implicit assumptions whcih aren't necessarily valid, especially (a) that the final "desirability" score would be a systematic continuous function of the fomula's inputs, ( the trade-offs between them work as the formula implicitly determines and © a certain amount of money has the same meaning for everyone using the formula. None of these are necessarily correct assumptions. Therefore we ended-up with the format provided in this evenings article. I'm sure there will be comment on this. Perhaps best to start it in a new thread.

[Mark D Segal]

Marc,

Is the D3x in a category of its own?

Cheers,
Bernard

For price, definitely.

For resolution and sensor quality, you can measure it against its closest rival in this evening's spreadsheet and decide for yourself whether the physical value-added is worth the added $$$. This is where the rubber hits the road and the final answer becomes totally subjective, because the price difference means more to some people than to others, and likewise for the quality differentials.

[marcmccalmont]

Marc,

Is the D3x in a category of its own?

Cheers,
Bernard

It must be, actually the original post was from Dec 3rd just a few months ahead of its time.
Marc

[marcmccalmont]

Marc, I don't think this is a viable approach. I started-off the exercise which Michael published this evening doing something very similar - one formula which creates a single score for each camera in which a camera does better the higher the resolution, the better the sensor and the lower the price. Such formulae are easy enough to conjure and compute, but when I sat back to think about it, I decided that they embed implicit assumptions whcih aren't necessarily valid, especially (a) that the final "desirability" score would be a systematic continuous function of the fomula's inputs, ( the trade-offs between them work as the formula implicitly determines and © a certain amount of money has the same meaning for everyone using the formula. None of these are necessarily correct assumptions. Therefore we ended-up with the format provided in this evenings article. I'm sure there will be comment on this. Perhaps best to start it in a new thread.

Your probably correct
Marc

[Ray]

Mark,
I didn't realise this topic was in response to your recent article on LL, which I've just read. I'm a bit nonplussed. Those who are experienced enough to understand the various aspects of image quality will tend to know what they want in respect of resolution, DR, high ISO performance, lenses etc., and will also be competent to judge whether the price can be justified in respect to their own economic circumstances.

Those who are totally ignorant of camera specifications and want a star rating from 1 to 5, or 1 to 10, will likely be unable to appreciate your reasoning.

Isn't this a paradox?

[fike]

Each time this topic is revisited, I keep thinking of new attributes that are not accounted for.  

This one I call "Aversion to Destruction Cost Threshold."

Many of us carry our cameras in some pretty crazy environments, like say in a small boat in the antarctic--lunatics.  Now, if I were going in a boat like that, I would be unwilling to carry $6,000 worth of camera around my neck, or even in a drybag.  I have always thought that the price of my camera must remain in a range that I am willing to risk its loss, theft, or destruction--otherwise I wouldn't be willing to take it to the wild-places where great photos are made.  My number is somewhere around $2,500.  Therefore I have a ceiling on my investment, no matter how good the next camera in the hierarchy is.  

This could be easily be simplified as a more-normal persons 'B-u-d-g-e-t.'

I think these tools can be really discouraging to people who have more modest means.  They somehow think that they can't possibly get the Canon 350D, because its numbers are stinkers.  No, they are not stinkers.  That was and is a good camera, that could be perfect for some folks' budgets, but because it is held up against superstars its apparent inferiority is highlighted.

It pushes people to want cameras beyond their means and as a result they get discouraged and don't get any camera. I have a  friend in that boat right now.  I have encouraged him to get a decent DSLR that is one generation older, but he is insistent that only the newest and best will do.  So, he has not camera.

[Mark D Segal]

Mark,
I didn't realise this topic was in response to your recent article on LL, which I've just read. I'm a bit nonplussed. Those who are experienced enough to understand the various aspects of image quality will tend to know what they want in respect of resolution, DR, high ISO performance, lenses etc., and will also be competent to judge whether the price can be justified in respect to their own economic circumstances.

Those who are totally ignorant of camera specifications and want a star rating from 1 to 5, or 1 to 10, will likely be unable to appreciate your reasoning.

Isn't this a paradox?

Yup - both of those types co-exist, and in-between them we expect there to be a large category of intelligent, undecided or just curious people who may appreciate the compact assistance this provides, given the plethora of choice out there. We'll see how it plays-out as the comments come in.

[Mark D Segal]

Each time this topic is revisited, I keep thinking of new attributes that are not accounted for.  

This one I call "Aversion to Destruction Cost Threshold."

Many of us carry our cameras in some pretty crazy environments, like say in a small boat in the antarctic--lunatics.  Now, if I were going in a boat like that, I would be unwilling to carry $6,000 worth of camera around my neck, or even in a drybag.  I have always thought that the price of my camera must remain in a range that I am willing to risk its loss, theft, or destruction--otherwise I wouldn't be willing to take it to the wild-places where great photos are made.  My number is somewhere around $2,500.  Therefore I have a ceiling on my investment, no matter how good the next camera in the hierarchy is.  

This could be easily be simplified as a more-normal persons 'B-u-d-g-e-t.'

I think these tools can be really discouraging to people who have more modest means.  They somehow think that they can't possibly get the Canon 350D, because its numbers are stinkers.  No, they are not stinkers.  That was and is a good camera, that could be perfect for some folks' budgets, but because it is held up against superstars its apparent inferiority is highlighted.

It pushes people to want cameras beyond their means and as a result they get discouraged and don't get any camera. I have a  friend in that boat right now.  I have encouraged him to get a decent DSLR that is one generation older, but he is insistent that only the newest and best will do.  So, he has not camera.

There is something called <insurance> for equipment. That reduces the risk you mention, to an extent depending on the terms of the insurance policy you buy.

Many normal people do have budgets, and that's why we put a price variable into the logic flow. The notion that good information destroys rational decision-making is indeed a rather strange hypothesis. Can you back it up with valid evidence? Economics is importantly about "choice" and optimizing decisions under constraints. Your friend may actually be making a valid, rational choice by holding off until he can afford what he wants. Other people would make compromises if they want something useful now but can't afford the "best" or the "bleeding edge". You know, I'd love to have a Bentley but it's beyond the pale, so I drive a Toyota .............