Pages: [1]   Go Down

Author Topic: Why numbers are important  (Read 3079 times)

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
Why numbers are important
« on: February 04, 2009, 01:17:10 am »

Michael et all,
When I made a living designing high end loudspeakers I experienced the insanity that Digital Photography is going through now. I had wished we had matured beyond this limited thinking. Michael is correct that high end audio got polarized as measure'rs and listeners, to the point that neither was correct or balanced. As one who could measure a loudspeaker accurately and accurately predict how it will sound and with the ability to listen and learn I have the following observations. Human senses are incredibly sensitive but subjective, you must be careful. One can accurately test and measure a device and have a very high correlation to how it sounds, listens or what ever. If there is a discrepancy one has to work hard to find a better test that will accurately describe what was observed. A quick example: I designed a speaker that tested well but sounded poorly and dug into the problem deeper. A friend took a look at one of the drivers with laser interferometry and found a 3 KHz standing wave dancing across the dust cap. Dave said "Marc if you increase the sampling rate of your spectrum analyser I think you will see them" I did and I saw the problem. I have spent more than 30 years testing, designing and listening to high end loudspeakers and can assure you that with the proper testing you can have a high correlation between the tests and the observations. I hope Digital Photography matures quicker than High End Audio and that all of us take a more balanced view. As I see it, all I observed this time is: trust your eyes the DxO mark tests are a step in the correct direction but fall short of telling the whole story. It is too early to give up on testing and fall into the trap that my eyes are capable of accurate repeatable non subjective observations. Lets just keep improving our testing of sensors (like was done with lenses) until the tests are good, valid and accurately represent what we see. IT CAN BE ACCOMPLISHED! TRUST ME. DON'T THROW IN THE TOWEL SO EARLY.
Sorry for the rant but we all lose if Digital Photography gets polarized into the lookers (photographers) vs the testers (engineers), not a balanced combination of both. In science some of the best lookers and listeners were also some of the best testers and engineers.
Marc
Logged
Marc McCalmont

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Why numbers are important
« Reply #1 on: February 04, 2009, 01:43:22 am »

Quote from: marcmccalmont
Michael et all,
When I made a living designing high end loudspeakers I experienced the insanity that Digital Photography is going through now. I had wished we had matured beyond this limited thinking. Michael is correct that high end audio got polarized as measure'rs and listeners, to the point that neither was correct or balanced. As one who could measure a loudspeaker accurately and accurately predict how it will sound and with the ability to listen and learn I have the following observations. Human senses are incredibly sensitive but subjective, you must be careful. One can accurately test and measure a device and have a very high correlation to how it sounds, listens or what ever. If there is a discrepancy one has to work hard to find a better test that will accurately describe what was observed. A quick example: I designed a speaker that tested well but sounded poorly and dug into the problem deeper. A friend took a look at one of the drivers with laser interferometry and found a 3 KHz standing wave dancing across the dust cap. Dave said "Marc if you increase the sampling rate of your spectrum analyser I think you will see them" I did and I saw the problem. I have spent more than 30 years testing, designing and listening to high end loudspeakers and can assure you that with the proper testing you can have a high correlation between the tests and the observations. I hope Digital Photography matures quicker than High End Audio and that all of us take a more balanced view. As I see it, all I observed this time is: trust your eyes the DxO mark tests are a step in the correct direction but fall short of telling the whole story. It is too early to give up on testing and fall into the trap that my eyes are capable of accurate repeatable non subjective observations. Lets just keep improving our testing of sensors (like was done with lenses) until the tests are good, valid and accurately represent what we see. IT CAN BE ACCOMPLISHED! TRUST ME. DON'T THROW IN THE TOWEL SO EARLY.
Sorry for the rant but we all lose if Digital Photography gets polarized into the lookers (photographers) vs the testers (engineers), not a balanced combination of both. In science some of the best lookers and listeners were also some of the best testers and engineers.
Marc

Quite right! I agree with this attitude. If the eyes are seeing something (or the ears hearing something) which seems at conflict with the test results, or at variance with the test results, it's because the tests are failing to measure some factor which is visible or audible.

DXO tell us broadly what they are measuring, SNR, DR, Tonal Range, Color Sensitivity etc. If there's some quality which the eye can pick up but which DXO is not measuring, then let's hear about it. Don't scrap the testing. Improve it.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Why numbers are important
« Reply #2 on: February 04, 2009, 01:50:09 am »

Hi,

Thanks for a posting offering a lot of insight!

Just want to point out that many "gold ears" buy anything, from golden power cables to epitaxial USB cables, much of which beats common sense.

I'd suggest that Michael doesn't agree with the DxO rating on MFDBs, because that rating does not match his experience. Nevertheless I still think that he takes a balanced view.

Best regards
Erik



Quote from: marcmccalmont
Michael et all,
When I made a living designing high end loudspeakers I experienced the insanity that Digital Photography is going through now. I had wished we had matured beyond this limited thinking. Michael is correct that high end audio got polarized as measure'rs and listeners, to the point that neither was correct or balanced. As one who could measure a loudspeaker accurately and accurately predict how it will sound and with the ability to listen and learn I have the following observations. Human senses are incredibly sensitive but subjective, you must be careful. One can accurately test and measure a device and have a very high correlation to how it sounds, listens or what ever. If there is a discrepancy one has to work hard to find a better test that will accurately describe what was observed. A quick example: I designed a speaker that tested well but sounded poorly and dug into the problem deeper. A friend took a look at one of the drivers with laser interferometry and found a 3 KHz standing wave dancing across the dust cap. Dave said "Marc if you increase the sampling rate of your spectrum analyser I think you will see them" I did and I saw the problem. I have spent more than 30 years testing, designing and listening to high end loudspeakers and can assure you that with the proper testing you can have a high correlation between the tests and the observations. I hope Digital Photography matures quicker than High End Audio and that all of us take a more balanced view. As I see it, all I observed this time is: trust your eyes the DxO mark tests are a step in the correct direction but fall short of telling the whole story. It is too early to give up on testing and fall into the trap that my eyes are capable of accurate repeatable non subjective observations. Lets just keep improving our testing of sensors (like was done with lenses) until the tests are good, valid and accurately represent what we see. IT CAN BE ACCOMPLISHED! TRUST ME. DON'T THROW IN THE TOWEL SO EARLY.
Sorry for the rant but we all lose if Digital Photography gets polarized into the lookers (photographers) vs the testers (engineers), not a balanced combination of both. In science some of the best lookers and listeners were also some of the best testers and engineers.
Marc
« Last Edit: February 04, 2009, 01:53:32 am by ErikKaffehr »
Logged
Erik Kaffehr
 

NikosR

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 622
    • http://
Why numbers are important
« Reply #3 on: February 04, 2009, 01:54:03 am »

I'm pasting here my response to Michael in another (relevant) thread because I think it can contribute to this, more general, discussion:


'I don't subscribe to this point of view. I like to understand WHY I'm seeing what (I think) I'm seeing and WHY I'm not seeing what I should be seeing. Subjective evaluation and technical testing go hand in hand otherwise we're talking about magic.

Sure, technical measurements might not cater for what one is seeing but in this case questioning and investigation should proceed concurrently in two ways:

1. Make sure I'm seeing well (i.e. am I looking into the right things, am I not masking differences by introducing other variables when the differences would be obvious if I was looking at something else etc. etc.)

2. Check that I'm measuring the right things in the right way and if not try to improve on what and how I'm measuring.

Both of the above are equally important. After all, it's obvious that the sun is revolving around the earth isn't?'
« Last Edit: February 04, 2009, 01:54:24 am by NikosR »
Logged
Nikos

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Why numbers are important
« Reply #4 on: February 04, 2009, 01:57:25 am »

Yeah,

That was obvious for greater part of human history. Is astrology really more reliable after Copernicus?

Best regards
Erik
Quote from: NikosR
Both of the above are equally important. After all, it's obvious that the sun is revolving around the earth isn't?'
« Last Edit: February 04, 2009, 02:01:32 am by ErikKaffehr »
Logged
Erik Kaffehr
 

Eric Myrvaagnes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 22814
  • http://myrvaagnes.com
    • http://myrvaagnes.com
Why numbers are important
« Reply #5 on: February 04, 2009, 09:46:30 am »

Quote from: ErikKaffehr
Just want to point out that many "gold ears" buy anything, from golden power cables to epitaxial USB cables, much of which beats common sense.
Durn it! I guess I gotta go get my hearing aids gold-plated now! 
Logged
-Eric Myrvaagnes (visit my website: http://myrvaagnes.com)

Tklimek

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 284
Why numbers are important
« Reply #6 on: February 04, 2009, 11:37:16 am »

Hey.....you know what would be cool?  A site much like the DxO information (which collects all of the relevant various data points) and then lets you set up your OWN categories.  In other words....for example...let's say there are 6 data points (totally making this up):

  • Resolution
  • Price
  • Dynamic Range
  • Color Depth
  • Low Light ISO

As a user of the site, you could pick as many or as few of the categories you were interested in; so for example let's say I was interested in:

  • Price
  • Dynamic Range
  • Resolution
  • Low Light ISO

Then you would have to "weight" each of your choices, and site would then re-calculate the "score" for the cameras in it's database for which it had relevant information and THEN present the various graphs and charts and scores.  That would be pretty cool!  I think that a hack like myself for instance might place a different "weight" on "Price" than perhaps someone like....Warren Buffet!

Getting back to the original topic here, I think that the data SHOULD be collected but should be able to be organized and presented by how the USER thinks it is important, and not the collecting organization.  Am I off base here?

Cheers.....

Todd in Chicago
Logged
Pages: [1]   Go Up