Pages: [1] 2 3   Go Down

Author Topic: DXOMark  (Read 19496 times)

fike

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1413
  • Hiker Photographer
    • trailpixie.net
DXOMark
« on: November 17, 2008, 09:54:18 pm »

When I read Michael's update today about the DXOMark site, I was pretty excited, but after just a few minutes at the site messing around with all the sliders and nifty flash gadgetry, I was left kind of bored.  

Is this it?  Is this what photography has been reduced to?

That word "reduced" seems to be what everyone is shooting for.  We want to reduce cameras down to one number, one rating, a top ten of a sort.  

Top to bottom, image quality is becoming a business of analyzing a gnats @ss of variance.  Michael comments about what one is to make of a 0.1 difference between two cameras.  NOTHING.  

We needed tools like this 4 years ago, but today they become more noise.  Everyone is always trying to fight the last war...fight the last battle.  In my opinion, DXOMarkis a tool that would have been great to fight the first round of the DSLR wars.
Logged
Fike, Trailpixie, or Marc Shaffer

Misirlou

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 711
    • http://
DXOMark
« Reply #1 on: November 18, 2008, 12:57:47 am »

I don't completely agree. Yes, there is still way too much pixel peeping going on these days, but if nothing else, DxOmark should provide some facts to turn down the heated debate of late about increasing pixel density. We've had a number of people arguing for a long time that increased pixel density is nothing but marketing hype, and others who have said that they aren't seeing the noise problems they've been told to expect. It's great to see some effort to finally analyze that question rigorously. I find that useful, and current.

Sure, some people might obsess over that single sensor number on DxOmark, but they could have just as easily obsessed over pixel count before, or frame rate, or any other silly thing. Ever heard of the great debate over the relative merits of Xenotars and Planars in Rolleiflex TLRs from the 50's? And it's not like we don't have any reviewers to tell us about the other important characteristics of new equipment. I rely a lot on Michael's valuable insights on camera handling, for example. Plenty of other sites are available to talk about purely artistic considerations.
Logged

feppe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2906
  • Oh this shows up in here!
    • Harri Jahkola Photography
DXOMark
« Reply #2 on: November 18, 2008, 01:51:02 am »

Perhaps (hopefully) the greatest contribution of DXOmark is to draw attention to something else than megapixels. If camera manufacturers start using DXOMark in their marketing hype, that's so much better than pushing out yet another high-mp camera.

Not to say that DXOMark is the end-all of objective sensor quality metrics, but it certainly is more rigorous than a mere megapixel count.

Buyers are always welcome to ignore such measures, and go for ergonomics, FPS or weather sealing.

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DXOMark
« Reply #3 on: November 18, 2008, 05:48:09 am »

Quote from: fike
We want to reduce cameras down to one number, one rating, a top ten of a sort.
We don't need to to that since digital cameras already _are_ numbers, and there is no possible discussion about that. A different story is Photography, which is an art and as such can never be expressed as numbers.

So IMO the new site is the closest one to the real camera we can find now on the Net.

BR


imagico

  • Newbie
  • *
  • Offline Offline
  • Posts: 47
    • http://www.imagico.de/
DXOMark
« Reply #4 on: November 18, 2008, 06:59:17 am »

I think criticising a quantified test generally as inappropriate reduction of photography or pixel peeping misses the most important point.  There are essentially two useful ways to approach the matter of equipment testing:

  • Evaluation based on competence and trust.  Someone who you consider competent and skilled gives you a subjective opinion.  What you can gain from such a test much depends on how well the tester's and your priorities and view of things comply.
  • Evaluation based on scientific method.  Key element here is what is commonly called intersubjective verifyability - a testing can only be called scientific if it can be reproduced independently by others.  This requires the used procedure to be documented in minute detail.  Therefore the scope of such testing is usually limited to very few properties of the equipment (like sensor properties in case of digital cameras).

The problem about DXOMark to me seems that for falling into the first category their competence is not too widely accepted (most would consider an experienced photographer they know much more qualified).  For the second category their method is not described well enough to actually assess its reliability.  Don't get me wrong, it seems lightyears better than what you can find on dpreview but still not something i would put trust in.

And i think this is a bit of what Michael tries to address with his closing remarks - if there is a minute difference between the rating of different cameras you would need to know precisely how these numbers were determined to be able to interpret their significance.
Logged
Christoph Hormann
photolog / artificial images / other stuff

fike

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1413
  • Hiker Photographer
    • trailpixie.net
DXOMark
« Reply #5 on: November 18, 2008, 07:16:28 am »

I think my issue is with the fact that the seem to reduce the camera to a few properties of the sensor, not the whole camera system.  For example, I don't see any measure of a camera's ability to resolve detail.  My guess is that they would consider resolving power as a property of the camera sensor AND LENS, but to me that is the camera system.  Without looking at all the parts from lens and capture through post-processing (if any) to printing.  

It is like judging the success of a person's life by their SAT scores (college admission exams) instead of by the composite of their life's work and achievements.
Logged
Fike, Trailpixie, or Marc Shaffer

MarkL

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 475
DXOMark
« Reply #6 on: November 18, 2008, 07:28:42 am »

Quote from: fike
Top to bottom, image quality is becoming a business of analyzing a gnats @ss of variance.  Michael comments about what one is to make of a 0.1 difference between two cameras.  NOTHING.

It's amusing that there is a 0.1 difference between the D700 and D3 which use the same sensor.
Logged

NikosR

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 622
    • http://
DXOMark
« Reply #7 on: November 18, 2008, 02:08:48 pm »

Quote from: MarkL
It's amusing that there is a 0.1 difference between the D700 and D3 which use the same sensor.

Not amusing at all. Could be explained in various ways. One is sample deviation. Not sure how DxO deal with this. Other reason might be differences in the off the chip circuits.
Logged
Nikos

barryfitzgerald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 688
DXOMark
« Reply #8 on: November 18, 2008, 06:11:43 pm »

So far nobody has inserted a Ken Rockwell phrase
lol ;-)

Being dead honest, I take these numbers with a pinch (bucket) of salt. Esp the dynamic range ones..10.1 stops on a G9, right..ok..cough cough..
And these 12 stops DSLR's have printable IQ do they? I suspect not. Most of the tests I have seen involve pulling around the image both ends, so much so that even a 6"x4" would be pushing it.
I can appreciate that with cameras, there will always be an element of testing, and the like. However, really we are going a bit too far with this.

I therefore declare that site to be.."of little importance in the real world"

There was a time you would go to the shop and grab a few cameras, play around and the feel of one would just fit you, and you bought it. Aaaah back when life was simple..
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DXOMark
« Reply #9 on: November 18, 2008, 07:10:23 pm »

I think perhaps some folks have missed the whole point of pixel peeping. It's not so that we can get excited about small differences and make purchasing decisions based on such small differences. It's so we can ignore such differences, remove them from the equation and concentrate on more relevant factors such as availability of good wide-angle lenses, autofocus accuracy, bracketing options etc etc.

In order to determine whether or not differences in performance are significant, someone has to do the testing, and hopefully as precisely and rigorously as possible. Subjective appraisal is just too variable.
Logged

fike

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1413
  • Hiker Photographer
    • trailpixie.net
DXOMark
« Reply #10 on: November 18, 2008, 07:26:35 pm »

Quote from: Ray
I think perhaps some folks have missed the whole point of pixel peeping. It's not so that we can get excited about small differences and make purchasing decisions based on such small differences. It's so we can ignore such differences, remove them from the equation and concentrate on more relevant factors such as availability of good wide-angle lenses, autofocus accuracy, bracketing options etc etc.

In order to determine whether or not differences in performance are significant, someone has to do the testing, and hopefully as precisely and rigorously as possible. Subjective appraisal is just too variable.

If that is what you are up to, then great. I think that is fine.  I think what I object to is when people use it to justify hyperbolic statements of superiority or inferiority of a particular camera of preference.  

It's the system baby.  From the brain, down the arms, through the lens and camera to the paper.  Everything matters.  And everything matters differently for different people and applications.
Logged
Fike, Trailpixie, or Marc Shaffer

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DXOMark
« Reply #11 on: November 18, 2008, 08:05:58 pm »

Quote from: fike
It's the system baby.  From the brain, down the arms, through the lens and camera to the paper.  Everything matters.  And everything matters differently for different people and applications.

True! But it's difficult to change one's arms and brain. Changing the camera is easy. With brain and arms the same, differences attributed to the camera can be spectacular, provided one skips a few upgrades.

On the other hand, one could go along with Ken Rockwell's hypothesis that the camera doesn't matter. One could spend a few years studying art, composition and lighting techniques, and then also take spectacularly improved images with the same old-fashioned camera. Such improvements could then be attributed to the brain and arms rather than the camera   .
Logged

The View

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1284
DXOMark
« Reply #12 on: November 21, 2008, 02:57:48 am »

The Nikon 40Dx beats the Canon 40d in color depth by 0,3.

Does that mean the colors of the Nikon D40 are better? Or even equal? Is color depth all that is to color? Isn't there also how colors are interpreted and recorded?

I see those guys measure a Rembrandt and a Picasso and then judge the artistic quality by the color depth of the paints those masters used.



If this low-end numbers ranking persists, manufacturers may choose to build cameras that fare well in tests that can be understood by Joe Blow, but don't deliver a truly aesthetic image quality.

Image quality is a matter of definitions.

DXO sensor reduces it to a couple of technical data.


Compare: what satisfactory work Ansel Adams could do with the older films with the thicker emulsion. He mentioned it in his book "The Negative", that modern films, with their thin emulsions, did not offer the same kind of aesthetic quality that his older, less advanced films had.

This is a comparison, and I guess you know how it's meant.

You could build a camera that records in maximum color depth, but gives unpleasant colors.

And the data would still show it's the top camera. That's the danger of the numbers game.
« Last Edit: November 21, 2008, 03:08:00 am by The View »
Logged
The View of deserts, forests, mountains. Not the TV show that I have never watched.

The View

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1284
DXOMark
« Reply #13 on: November 21, 2008, 02:59:09 am »

I'd rather read a review of a knowledgeable and aesthetically educated photographer on how a camera performs in that photographer's work.

For this you need education and knowledge and a lot of experience.

Pixel peeping can be understood by Joe Blow, it can also be done by Joe Blow.

Let's hope we won't get any Joe Blow cameras out of this.



PS: Safari crashed three times visiting that site.
« Last Edit: November 21, 2008, 03:02:09 am by The View »
Logged
The View of deserts, forests, mountains. Not the TV show that I have never watched.

The View

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1284
DXOMark
« Reply #14 on: November 21, 2008, 03:01:05 am »

Quote from: fike
Is this what photography has been reduced to?

That word "reduced" seems to be what everyone is shooting for.  We want to reduce cameras down to one number, one rating, a top ten of a sort.  

(...)

DXOMarkis a tool that would have been great to fight the first round of the DSLR wars.


I agree.
Logged
The View of deserts, forests, mountains. Not the TV show that I have never watched.

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
DXOMark
« Reply #15 on: November 21, 2008, 04:00:01 am »

0.1 is a negligable difference!

DXO is reporting results of stringent testing based on "raw" data, in my book that is a good thing. Most test results reported are for JPEG, which I never use.

I'd suggest that the tests are informative.

Best regards
Erik


Quote from: NikosR
Not amusing at all. Could be explained in various ways. One is sample deviation. Not sure how DxO deal with this. Other reason might be differences in the off the chip circuits.
Logged
Erik Kaffehr
 

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DXOMark
« Reply #16 on: November 22, 2008, 06:47:43 pm »

Quote from: The View
Image quality is a matter of definitions.

DXO sensor reduces it to a couple of technical data.


Compare: what satisfactory work Ansel Adams could do with the older films with the thicker emulsion. He mentioned it in his book "The Negative", that modern films, with their thin emulsions, did not offer the same kind of aesthetic quality that his older, less advanced films had.

This is a comparison, and I guess you know how it's meant.

You could build a camera that records in maximum color depth, but gives unpleasant colors.

And the data would still show it's the top camera. That's the danger of the numbers game.

There are plenty of definitions at the DXOMark website. What's your problem? You don't like an over all assessment expressed as a number? If you think there is some important quality of the sensor that DXO is not measuring, tell us what it is. Pleasantness of color is not a property of the sensor. The undeveloped RAW image does not have any colors. It's all data. Colors exist in the mind. What may be a pleasant color to you might not be to someone else. Getting pleasant colors is presumably what you try to do when you convert your RAW file and do postprocessing in Photoshop. If you want very accurate colors, then it's usually necessary to calibrate your camera.

The point should be stressed that DXO is not making an over all assessment of camera systems, but of camera sensors. The sensor is analagous to the old-fashioned film. I've still got a folder containing information on the properties of various types of film, which I downloaded years ago from the manufacturers websites. It includes not only the spacial frequency properties of the films, but curves for spectral dye density, and spectral sensitivity. For example, Provia 100F has greatest sensitivity in the blue layer.

In the early days of Kodak DSLRs, which were ridiculously expensive, it was suggested by at least one manufacturer that it might be possible to build a camera with a replaceable sensor. Instead of buying a new camera every couple of years as sensor technology improved along the lines of Moore's Law, one would just buy a new sensor. Nothing came of the idea for very good reasons. New cameras contain all sorts of improvements in addition to improved sensor performance, which are useful in their own right.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
DXOMark
« Reply #17 on: November 22, 2008, 08:03:06 pm »

Well, it seems that Apple needs to do some homework on "Safari", doesn't it? And yes, Firefox has also issues.

Erik

Quote from: The View
PS: Safari crashed three times visiting that site.
Logged
Erik Kaffehr
 

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DXOMark
« Reply #18 on: November 22, 2008, 11:48:03 pm »

Quote from: ErikKaffehr
Well, it seems that Apple needs to do some homework on "Safari", doesn't it? And yes, Firefox has also issues.

Erik

Erik,
I think you'll agree, these are side-line issues and people should recognise them as such. The question is, are the DXO results accurate? If anyone thinks they are not, then let them provide the evidence. In the absense of contrary eveidence, this is all idle speculation.
Logged

The View

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1284
DXOMark
« Reply #19 on: November 25, 2008, 09:30:53 pm »

Quote from: Ray
Pleasantness of color is not a property of the sensor. The undeveloped RAW image does not have any colors. It's all data. Colors exist in the mind. What may be a pleasant color to you might not be to someone else. Getting pleasant colors is presumably what you try to do when you convert your RAW file and do postprocessing in Photoshop. If you want very accurate colors, then it's usually necessary to calibrate your camera.

Colors exist in the mind?

So what?  Anything that exists in the real world exists also in the mind, expressed in the mind's way to comprehend, understand, and categorize things, putting them into relations to other phenomena.

So what you say is without any sense.

Or did you intend to say "color exists (only) in the mind"?

That would be less than having said something bearing no sense. It would be wrong.

Of course, color is what we call a certain sensation, when reflected light of a certain wave length hits our retina. Depending on that wave length we can see it in different colors, or, like infra red, we can't see it at all.

The same goes for a sensor. Instead of a retina being connected to a brain, it is connected to a computer chip and records the light data in zeros and ones.

If there would not be a strong correlation between light of a certain wave length hitting the sensor and its being categorized/interpreted as a certain color, color photography, and especially color management would be impossible.

Of course, sensors can be built with different sensibilities, just as lenses can be built sharp or cheap.

I do not have the impression that DxoSensor does any in depth analysis how color is seen by a sensor.


PS: it is also wrong to say a RAW file doesn't contain any color, just data.

It's data about color, even though easy to edit. Any RAW file will tell you what color a person's face is or what a color a car has. You just have to process it, knowing from which camera it came, so you can interpret the data about color, and you know what color it is. That's a part of color management.

Data, that cannot be linked to something tangible or concrete, is worthless.

In the same vein you could say that there's no music on a CD, just data.
« Last Edit: November 25, 2008, 09:31:09 pm by The View »
Logged
The View of deserts, forests, mountains. Not the TV show that I have never watched.
Pages: [1] 2 3   Go Up