The DXO problem is that they insist that they have a measure, but it's not a real measure. They have a whole bunch of measures, but those measures don't include the whole universe of possibilities, and without the whole universe, you can't reach a valid conclusion. And it may be inherently *impossible* to measure the entire universe of possibilities. So, if a lot of people with good, trained photographic eyes say that a large P65 print is better than a large D3 print, and a DXO measurement says it's not, then the eyes are right, just as they're right when they see a bumblebee fly.
I agree with John here, and I like the analogy with high end audio as well. I'm an ex-audiophile, and was "in the business" earlier in my days, so I'm familiar with that "lifestyle" (or would that be "obsession" lol) as well as anyone and there are some definite parallels with this topic.
A few comments:
Today's society wants it easy and is used to getting things quickly. I remember (as a kid) the 70's, so I've seen the change.
Today we've got your instant coffee, a quick-start breakfast,the morning snack from the row of vending machines down the hall, fast food lunch, and a microwave dinner, all eaten far too quickly as we accelerate to an early grave (or a dozen prescription medications at the least) as a result of our stress, lack of exercise, and poor nutrition.
Social commentary aside, the whole "what camera do I buy" / "what lens do I buy" process has changed as well. Today we've got the internet, the old school magazines, cameras being sold by typically less-than-knowledgeable clerks at big box retailers, a few camera stores here and there, and much, much, much more marketing and advertising being shoved at our ears and eyeballs. Most of that wasn't around in the "old days", you know, back when we had to trudge our 600mm lenses to school up hill both ways through the snow and all that.
Salespeople in the 70's were usually trustworthy to some point while these days it's not common for me to hear in my local photo shop (said to a new buyer) "Oh, get brand XXXXXX, it's much better than brand YYYY" simple because there is a spiff (incentive payment to the salesperson) on that brand or product or a push of the sales clerks own bias, which may very well have been formulated by exposure to the marketing and a need to defend their own brand choice. I may be wrong, but my memories were that people weren't anywhere as defensive about their brand choice "back then" as they are now. Well, those days are gone, and the consumer is getting a lot more information from various sources aimed at them all at once, and understandably wants to make a "correct" purchasing decision. In 1970 you maybe talked to a pro you knew, talked to your salesperson, asked a friend, looked at an advert, and off you went. Now you've got half a dozen web sites and forums that discuss things subjectively, another half a dozen that do "objective" tests, plus the sales folks, friends, more advertising and marketing, and often it's hard to *quickly* get a clear message that screams "hey, Mister Consumer - obviously the Plektomatic Q45XDR-III 17.3 megapixel DSLR is the perfect camera, get it".
So in today's world, what's the holy grail? The single all encompassing quality grade that covers all bases, is totally defensible, and can't possibly be wrong. I'd argue, as I think John would, that this is impossible, but it is the holy grail. If there could be one test site that totally was beyond any doubt the sole deity, the supreme being, and had "the grade", then the decision process would be really, really simple. However, real life isn't always simple, and due to, as John points out, the vast universe of possibilities that determine what quality is, I seriously doubt it's going to happen, ever. But people want this easy number, because it relieves them of the far more time consuming option that I think today provides a better answer: a combined analysis of multiple subjective reviewers, a reading to understand each reviewers biases and tastes, an overview of the objective sites (plural, not just DXO or photozone or whatever in isolation, but a look at the results across a group of multiple test sites) and then some discussion with folks who own them, and if possible, renting of the proposed purchase to evaluate it. This takes time, and it takes effort, and it takes admitting that something that measures well might not actually shoot well in some situations, or perhaps, the things the photographer values highly in the complex interrelated object called "image quality" may be weighted differently versus something that is made to shine on a test chart.
Now, does this mean measurements have no place? Certainly not. I think what I'd personally like to see in the future is an *exploration* of how to quantify what we might see in terms of "image quality" beyond the current test methods. We could, for example, look at lens performance. Many folks only look at resolution (via photozone or slrgear or whatever) or an MTF graph for "the answer". Yet lens designers themselves (like the fellow from Zeiss in his MTF white paper and Dr. Caldwell, he of the famed Coastal 60mm Macro) have said MTF doesn't provide all the answers. So perhaps more research could be done in these areas - trying to correlate measurements with visual performance (not just lenses, but camera bodies too). Offhand, I don't think only evaluating sensor data exclusive to the raw conversion really helps us much here, but that's must my opinion.
I don't think we'll ever be able to fully quantify image quality, maybe not in my lifetime or the next generations, but I think it would still be valuable to at least try and make some progress. To circle around back to the high end audio analogy, Stereophile always has measurements and commentary on them along with the subjective review of the gear itself, and I get the distinct impression that the crew at Stereophile is very much aware of the differences between good measurement and good sound and tries to figure out why this is so. And perhaps that's what is needed "down the road" with photographic gear - more discussion of both subjective and objective together, with some dialog as to *why* something that scores .00002 geek-o-units of noise may not produce visible image quality as well as something that scores .000034 geek-o-units, or why a lens that knocks one out of the ballpark on photozone just doesn't draw that well when faced with a three dimensional object in the real world.
Like others, I do worry, that in the same manner that so many people use one and only one "objective" lens test site to judge lenses (even when research indicates divergent opinions to that site) that the DXOmark becomes the one and only factor, simply because it's an easy to digest number. Like I stated earlier, careful analysis of the wider spectrum of opinions, always taking into account reviewer bias as well as an admittance that "objective" tests may not be perfect, is a better method at this time.
As an aside, I'd also feel that a lot of people who make proclamations about image quality or lens sharpness (or whatever) need to include the *magnitude* of the difference. A lot of times I think people end up chasing the wrong thing - a guy who drops 8 grand on a D3X to shoot landscape, but then uses a 24-120 average-at-best lens on gitzo 1 series tripod is going to get creamed by a guy with a D2X, 100/2 Makro Planar and a stable support system. My own approach has always been lenses first, then bodies, and I've of late come to realize the importance of the support system as well. Bodies come and go, as will the arguments in the forums as to which one is better, but if one acquires a set of lenses that perform well and (importantly) have a "drawing" style that one feels suits their subjective tastes, they're off to a heck of a good start.