Michael et all,
When I made a living designing high end loudspeakers I experienced the insanity that Digital Photography is going through now. I had wished we had matured beyond this limited thinking. Michael is correct that high end audio got polarized as measure'rs and listeners, to the point that neither was correct or balanced. As one who could measure a loudspeaker accurately and accurately predict how it will sound and with the ability to listen and learn I have the following observations. Human senses are incredibly sensitive but subjective, you must be careful. One can accurately test and measure a device and have a very high correlation to how it sounds, listens or what ever. If there is a discrepancy one has to work hard to find a better test that will accurately describe what was observed. A quick example: I designed a speaker that tested well but sounded poorly and dug into the problem deeper. A friend took a look at one of the drivers with laser interferometry and found a 3 KHz standing wave dancing across the dust cap. Dave said "Marc if you increase the sampling rate of your spectrum analyser I think you will see them" I did and I saw the problem. I have spent more than 30 years testing, designing and listening to high end loudspeakers and can assure you that with the proper testing you can have a high correlation between the tests and the observations. I hope Digital Photography matures quicker than High End Audio and that all of us take a more balanced view. As I see it, all I observed this time is: trust your eyes the DxO mark tests are a step in the correct direction but fall short of telling the whole story. It is too early to give up on testing and fall into the trap that my eyes are capable of accurate repeatable non subjective observations. Lets just keep improving our testing of sensors (like was done with lenses) until the tests are good, valid and accurately represent what we see. IT CAN BE ACCOMPLISHED! TRUST ME. DON'T THROW IN THE TOWEL SO EARLY.
Sorry for the rant but we all lose if Digital Photography gets polarized into the lookers (photographers) vs the testers (engineers), not a balanced combination of both. In science some of the best lookers and listeners were also some of the best testers and engineers.
Quite right! I agree with this attitude. If the eyes are seeing something (or the ears hearing something) which seems at conflict with the test results, or at variance with the test results, it's because the tests are failing to measure some factor which is visible or audible.
DXO tell us broadly what they are measuring, SNR, DR, Tonal Range, Color Sensitivity etc. If there's some quality which the eye can pick up but which DXO is not measuring, then let's hear about it. Don't scrap the testing. Improve it.