It's almost a cliche to mention it, but the lessons learned from the good (bad) old days of high-end audio are salutary.
Stereo Review magazine did in depth technical reviews of amps and such down to four decimal points. Their reviews were so influential that manufacturers actually designed some products to test well, with 0.005 THD. The problem was that there was little if any correlation between test performance and how an amp sounded.
The Absolute Sound magazine took the other approach, eschewing lab tests and just relying on the judgement of experienced listeners. In this case most observers noted that this was much more closely correlated with the user's listening experience that the lab tests. The take-away was that people listen to music not numbers. The reasons are now well understood, though they wern't at the time.
Michael
There are many problems with high end audio, measurements is not one of them.
The reluctance to use blind testing and/or consulting the scientific literature comes to mind as the biggest reason why people are laughing at the industry, magazines and enthusiasts (such as myself). High end audio is (by the majority) seen as males in their 50s with more money than critical sense. I think that is sad.
If we did not have the "subjective revolution" in the late 70s/early 80s (before I started working part-time at a hifi shop), I claim that we would probably have better sound in our homes today. Right now, the hifi industry gets a better return-on-investement by marketing idiotic cables or other bling for the cult, rather than researching and developing the stuff that, you know, affect sound.
More likely they feel they do not need to use numerical lab results. Most top chefs probably dont feel the need to use timers and temperature probes. They see, feel, taste.
I believe that there are different lines of thought in cooking as well. Heston Blumenthal has a decidedly "scientific" approach to cooking.
http://en.wikipedia.org/wiki/Heston_Blumenthal"Molecular Gastronomy" gives some great examples of issues where expert cooks have come to the right _conclusions_, but their arguments are all wrong. I believe that this can be relevant to photography: a great photographer with a sound testing methology might conclude that doing so and so gives her better images. When she tries to _explain_ why doing this leads to better images, she may well be totally wrong, and when this "knowledge" works its way into photography books, websites and curriculum, people will inevitably apply her explanations in areas where they leads to bad practice. Depending on how much testing these people do on their own (and/or how much of a guru status the initial reporter have in the field), people might end up doing counter-productive things.
http://en.wikipedia.org/wiki/Molecular_gastronomy#Example_myths_debunked_or_explainedIn (classical) physics, there can only be one "right", all other must be wrong. In cooking or photography art, there can be several "rights", and the degree of "rightness" may be cultural/social. This may explain why so many great photographers have such strong opinions about the analysis and measurement of technical facets of their cameras.
-h