Pages: 1 [2] 3   Go Down

Author Topic: DxO bashing - a bit of perspective  (Read 14330 times)

Lightsmith

  • Full Member
  • ***
  • Offline Offline
  • Posts: 197
Re: DxO bashing - a bit of perspective
« Reply #20 on: May 18, 2014, 01:55:40 am »

Eric's comments would lead me to conclude that the DXO "bashing" is justified. There are so many compromises made in their methodology as to make the end results not ones I would rely on to make a lens or camera purchase any more than I would use the feds fuel economy numbers when selecting a vehicle to buy.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DxO bashing - a bit of perspective
« Reply #21 on: May 18, 2014, 02:26:25 am »

That is of course the save yourself a lot of money question.

Best regards
Erik

Yeah, but people don't, Paulo: there's a routine presumption that any FF sensor is inherently "better" than any crop sensor, purely because its a FF sensor.

So it's not a matter of "of course..." - some folk really don't get this.
Logged
Erik Kaffehr
 

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DxO bashing - a bit of perspective
« Reply #22 on: May 18, 2014, 09:44:35 am »

Eric's comments would lead me to conclude that the DXO "bashing" is justified. There are so many compromises made in their methodology as to make the end results not ones I would rely on to make a lens or camera purchase any more than I would use the feds fuel economy numbers when selecting a vehicle to buy.

What I like about DxO is that their actual measurements eg. DR usually agree with what you get if you take a raw file from your own camera. This has been confirmed time over time by the geeks on this forum. Their data is real. For the rest .... if you believe that a single number can sum up a camera or a musical instrument, then I guess you should marry a shoe-size ;)

Edmund
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: DxO bashing - a bit of perspective
« Reply #23 on: May 19, 2014, 06:06:02 pm »

Maybe someone who has a real beef with their process can post a sample from the maligned camera or lens in question that shows the problem.

Nyself I dont have a problem with what they recommend. I have put my money there with good results. I do feel some lenses they rate poorly are better than presented. I'm talking about some old stuff. Overall DxO seems to know their stuff.
Logged

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
Re: DxO bashing - a bit of perspective
« Reply #24 on: May 19, 2014, 06:57:14 pm »

What I like about DxO is that their actual measurements eg. DR usually agree with what you get if you take a raw file from your own camera. This has been confirmed time over time by the geeks on this forum. Their data is real. For the rest .... if you believe that a single number can sum up a camera or a musical instrument, then I guess you should marry a shoe-size ;)

Edmund
I see nothing wrong with marrying a "10" !
Marc :)
Logged
Marc McCalmont

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DxO bashing - a bit of perspective
« Reply #25 on: May 20, 2014, 11:17:11 am »

Hi,

The usual tests for ladies don't measure bitchness.

Best regards
Erik

I see nothing wrong with marrying a "10" !
Marc :)
Logged
Erik Kaffehr
 

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DxO bashing - a bit of perspective
« Reply #26 on: May 20, 2014, 02:50:40 pm »

Hi,

The usual tests for ladies don't measure bitchness.

Best regards
Erik


How can one measure something that doesn't exist? ;)
We're skating on thin ice here - there may some ladies reading their husbands posts ...
Let's go back to bashing DxO and cameras :)

Edmund
« Last Edit: May 20, 2014, 03:00:45 pm by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Manoli

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2296
Re: DxO bashing - a bit of perspective
« Reply #27 on: May 20, 2014, 03:36:20 pm »

How can one measure something that doesn't exist?

Well, the British think tank DEMOS recently tried ...

http://takimag.com/article/misogyny_among_the_fairer_sex_jim_goad/print#axzz32GcPsljO
Logged

The View

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1284
Re: DxO bashing - a bit of perspective
« Reply #28 on: June 01, 2014, 02:43:05 am »

The problem with DxO is that those are lab measurements that do not conform with field conditions.

Also, a great lens has an artistic quality, that cannot be measured, only experienced.

And the human eye has its own way of seeing things, different from a machine, and the qualities there can't always pinned down by simple numbers.

The reason these lab results are so popular: everybody can easily compare lenses and cameras, like you can compare the horse power of cars.

There is a difference between criticism and bashing that needs to be respected.


Camera developers have let lab results rule for a long time - and with them, their marketing departments. First we had the Megapixel wars, then the High ISO wars.

I wish camera quality would be tested by those who actually use them on a high level - not by lab technicians.

We would probably get cameras that are not as much designed for specs, but for artistic potential in image making.
Logged
The View of deserts, forests, mountains. Not the TV show that I have never watched.

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Re: DxO bashing - a bit of perspective
« Reply #29 on: June 01, 2014, 08:59:24 am »

The problem with DxO is that those are lab measurements that do not conform with field conditions.

Do you have an actual example of these supposed gaps?

My view is that the correlation is excellent.

Cheers,
Bernard

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DxO bashing - a bit of perspective
« Reply #30 on: June 01, 2014, 06:39:02 pm »


I wish camera quality would be tested by those who actually use them on a high level - not by lab technicians.


We're all waiting for you to do it :)

Edmund
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Keith Reeder

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 253
    • Capture The Moment
Re: DxO bashing - a bit of perspective
« Reply #31 on: June 15, 2014, 05:27:12 am »

I wish camera quality would be tested by those who actually use them on a high level - not by lab technicians.
 
They frequently are.

And then these Real World tests/reviews are summarily dismissed (often in support of some specific agenda) in favour of "but DxO says...", as if the Real World experience - and perhaps the hard-won skill of the user in maximising the image potential of the camera in question - are irrelevances.
Logged
Keith Reeder
Blyth, NE England

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DxO bashing - a bit of perspective
« Reply #32 on: June 15, 2014, 06:25:20 am »

 
They frequently are.

And then these Real World tests/reviews are summarily dismissed (often in support of some specific agenda) in favour of "but DxO says...", as if the Real World experience - and perhaps the hard-won skill of the user in maximising the image potential of the camera in question - are irrelevances.
The problem with such real-world reports is that they often point in orthogonal directions. If tester A says that camera 1 is better than camera 2, while tester B says the opposite and neither provide sufficient background for themselves or their testing methology, it is really hard to conclude based on the tests.

I am a big fan of such real-world tests for the stuff that cannot possibly be measured. Such as ergonomy and user-interface.

-h
Logged

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DxO bashing - a bit of perspective
« Reply #33 on: June 15, 2014, 07:03:18 am »

The problem with such real-world reports is that they often point in orthogonal directions. If tester A says that camera 1 is better than camera 2, while tester B says the opposite and neither provide sufficient background for themselves or their testing methology, it is really hard to conclude based on the tests.

I am a big fan of such real-world tests for the stuff that cannot possibly be measured. Such as ergonomy and user-interface.

-h

It's not only that. Some perceptual aspects can be quantified eg. distorsion, some with great difficulty eg eg. color can be quantified initially by colorimetry; so testers ignore those hard aspects until they are quantified, rather than saying: This thing looks good/bad. A typical example is perceived "upscalability" of an image from a camera - some upscale well, some fall apart.

Edmund
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DxO bashing - a bit of perspective
« Reply #34 on: June 15, 2014, 07:30:01 am »

It's not only that. Some perceptual aspects can be quantified eg. distorsion, some with great difficulty eg eg. color can be quantified initially by colorimetry; so testers ignore those hard aspects until they are quantified, rather than saying: This thing looks good/bad. A typical example is perceived "upscalability" of an image from a camera - some upscale well, some fall apart.

Edmund
I think that a more significant source of confusion is :
1) Some (possibly artistically good) photographer test the "DR" of a camera and finds it to be good/poor
2) Some "lab technicians" tests the "DR" of the same camera and finds it to be poor/good

Chances are, the two are not really testing the same thing using equivalent methods, so no wonder that they come to diverging conclusions. There is the very real possibility that the "lab technicians" are reporting something true but irrelevant. But there is also the possibility that the "seasoned veteran" plainly does not understand the image processing (how many concert pianists understands how a grand piano works?), or is using a processing workflow that is relevant to him and nobody else.

I do believe in reporting thoroughly _what_ you did, and _why_ you concluded like you did. Ideally, someone ought to be able to repeat your experiment and if they share taste and choice of subjects with you should come to the same conclusions.

-h
Logged

michael

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5084
Re: DxO bashing - a bit of perspective
« Reply #35 on: June 15, 2014, 09:05:54 am »

It's almost a cliche to mention it, but the lessons learned from the good (bad) old days of high-end audio are salutary.

Stereo Review magazine did in depth technical reviews of amps and such down to four decimal points. Their reviews were so influential that manufacturers actually designed some products to test well, with 0.005 THD. The problem was that there was little if any correlation between test performance and how an amp sounded.

The Absolute Sound magazine took the other approach, eschewing lab tests and just relying on the judgement of experienced listeners. In this case most observers noted that this was much more closely correlated with the user's listening experience that the lab tests. The take-away was that people listen to music not numbers. The reasons are now well understood, though they wern't at the time.

Michael
 
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Re: DxO bashing - a bit of perspective
« Reply #36 on: June 15, 2014, 09:01:48 pm »

It's almost a cliche to mention it, but the lessons learned from the good (bad) old days of high-end audio are salutary.

Stereo Review magazine did in depth technical reviews of amps and such down to four decimal points. Their reviews were so influential that manufacturers actually designed some products to test well, with 0.005 THD. The problem was that there was little if any correlation between test performance and how an amp sounded.

The Absolute Sound magazine took the other approach, eschewing lab tests and just relying on the judgement of experienced listeners. In this case most observers noted that this was much more closely correlated with the user's listening experience that the lab tests. The take-away was that people listen to music not numbers. The reasons are now well understood, though they wern't at the time.

Michael
 

The lesson to be learned here, Michael, is that the results of any lab tests, whatever the nature of the subject or object that is under study, have to be interpreted.

Such interpretation requires an understanding of the subjective significance of any numerical results, and how that subjective significance might be affected by other factors, such as the type of music, the loudness of the music, the frequency of the sound, the acoustic properties of the listening environment, and perhaps most significantly the hearing acuity of the listener.

Those who are interested in the technical aspects of hi fi soon learn that differences of 0.005 THD and 0.05 THD are irrelevant to their listening experience whatever the circumstances, just as those who are interested in the technical aspects of sensor performance soon learn that differences in Dynamic Range of 0.1 EV are irrelevant to their visual experience.

I suspect that those who attach little or no credence to the DXOMark test results are mostly jpeg shooters, or those who simply don't have the photographic skills to confirm for themselves the significance of the DXO results.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DxO bashing - a bit of perspective
« Reply #37 on: June 16, 2014, 12:14:42 am »

Hi,

For the audio comparison, there is a lot of hype in that area.

Regarding the DxO tests, they present a lot of info, but all of that info relates to noise. DxO mark essentially ignores resolution, for instance. I am pretty sure that the measurements are accurate.

Now, I would say noise is an important characteristic of cameras, but other factors may be more important, especially if noise related data is good enough. Lenses, AF and handling may have a significant role.

Best regards
Erik





The lesson to be learned here, Michael, is that the results of any lab tests, whatever the nature of the subject or object that is under study, have to be interpreted.

Such interpretation requires an understanding of the subjective significance of any numerical results, and how that subjective significance might be affected by other factors, such as the type of music, the loudness of the music, the frequency of the sound, the acoustic properties of the listening environment, and perhaps most significantly the hearing acuity of the listener.

Those who are interested in the technical aspects of hi fi soon learn that differences of 0.005 THD and 0.05 THD are irrelevant to their listening experience whatever the circumstances, just as those who are interested in the technical aspects of sensor performance soon learn that differences in Dynamic Range of 0.1 EV are irrelevant to their visual experience.

I suspect that those who attach little or no credence to the DXOMark test results are mostly jpeg shooters, or those who simply don't have the photographic skills to confirm for themselves the significance of the DXO results.
Logged
Erik Kaffehr
 

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: DxO bashing - a bit of perspective
« Reply #38 on: June 16, 2014, 01:22:54 am »

The lesson to be learned here, Michael, is that the results of any lab tests, whatever the nature of the subject or object that is under study, have to be interpreted.

Such interpretation requires an understanding of the subjective significance of any numerical results, and how that subjective significance might be affected by other factors, such as the type of music, the loudness of the music, the frequency of the sound, the acoustic properties of the listening environment, and perhaps most significantly the hearing acuity of the listener.

Those who are interested in the technical aspects of hi fi soon learn that differences of 0.005 THD and 0.05 THD are irrelevant to their listening experience whatever the circumstances, just as those who are interested in the technical aspects of sensor performance soon learn that differences in Dynamic Range of 0.1 EV are irrelevant to their visual experience.

I suspect that those who attach little or no credence to the DXOMark test results are mostly jpeg shooters, or those who simply don't have the photographic skills to confirm for themselves the significance of the DXO results.

More likely they feel they do not need to use numerical lab results. Most top chefs probably dont feel the need to use timers and temperature probes. They see, feel, taste.

The science is important, nobody credible would say we would have the current level of tools without the science. DxO tries to use a repeatable scientific methodology for comparisons. The deciding factor for everyone is judgement. Some people have it based on a variety of things. Some people lack judgement in a variety of things.

IMO DxO numbers are better at telling you what is good equipment than what is bad equipment.
Logged

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DxO bashing - a bit of perspective
« Reply #39 on: June 16, 2014, 03:40:32 am »

It's almost a cliche to mention it, but the lessons learned from the good (bad) old days of high-end audio are salutary.

Stereo Review magazine did in depth technical reviews of amps and such down to four decimal points. Their reviews were so influential that manufacturers actually designed some products to test well, with 0.005 THD. The problem was that there was little if any correlation between test performance and how an amp sounded.

The Absolute Sound magazine took the other approach, eschewing lab tests and just relying on the judgement of experienced listeners. In this case most observers noted that this was much more closely correlated with the user's listening experience that the lab tests. The take-away was that people listen to music not numbers. The reasons are now well understood, though they wern't at the time.

Michael
There are many problems with high end audio, measurements is not one of them.

The reluctance to use blind testing and/or consulting the scientific literature comes to mind as the biggest reason why people are laughing at the industry, magazines and enthusiasts (such as myself). High end audio is (by the majority) seen as males in their 50s with more money than critical sense. I think that is sad.

If we did not have the "subjective revolution" in the late 70s/early 80s (before I started working part-time at a hifi shop), I claim that we would probably have better sound in our homes today. Right now, the hifi industry gets a better return-on-investement by marketing idiotic cables or other bling for the cult, rather than researching and developing the stuff that, you know, affect sound.
More likely they feel they do not need to use numerical lab results. Most top chefs probably dont feel the need to use timers and temperature probes. They see, feel, taste.
I believe that there are different lines of thought in cooking as well. Heston Blumenthal has a decidedly "scientific" approach to cooking.
http://en.wikipedia.org/wiki/Heston_Blumenthal

"Molecular Gastronomy" gives some great examples of issues where expert cooks have come to the right _conclusions_, but their arguments are all wrong. I believe that this can be relevant to photography: a great photographer with a sound testing methology might conclude that doing so and so gives her better images. When she tries to _explain_ why doing this leads to better images, she may well be totally wrong, and when this "knowledge" works its way into photography books, websites and curriculum, people will inevitably apply her explanations in areas where they leads to bad practice. Depending on how much testing these people do on their own (and/or how much of a guru status the initial reporter have in the field), people might end up doing counter-productive things.
http://en.wikipedia.org/wiki/Molecular_gastronomy#Example_myths_debunked_or_explained

In (classical) physics, there can only be one "right", all other must be wrong. In cooking or photography art, there can be several "rights", and the degree of "rightness" may be cultural/social. This may explain why so many great photographers have such strong opinions about the analysis and measurement of technical facets of their cameras.

-h
« Last Edit: June 16, 2014, 04:01:46 am by hjulenissen »
Logged
Pages: 1 [2] 3   Go Up