3 Feb, 2009 - Eyes vs. Numbers - Which to Believe

[NikosR]

This is a sensible write up. I cannot help but agree with most of the basic ideas underlined in this article. The DxO MFDB issue is a challenge since one needs to consider what is being measured. If we agree that MFDB raw is half baked, aren't we doing the dSLR camera injustice by considering theirs 'baked enough'? After all, neither raw file is the final 'viewable' result. Where does one draw the line? DxO draws it at the raw level, they do produce a levelled comparison and one is free to interpret their data at will.

If you're comparing raw files (in an attempt to disengage from the intricacies of raw development) why would the DxO comparison be invalid or unfair to the MFDBs? On the other hand, indeed it sounds like it is...since they are intended to be used with their dedicated raw converters... but who says Nikons (for example) are not intended to be used with NX?  Maybe it would have been best for DxO to refrain from getting in that mousetrap in the first place.

On the other hand, I think I partially disagree with Michael's assertion about DxO not taking resolution into account. In some sense they do, in some sense they don't.

They do, because they publish results normalised for a standard 'print' size of 8x10. In this respect increased resolution results in an increase of their measured metrics (as they explain in their site).

On the other hand, they don't consider the increased USEABILITY of the larger resolution files (e.g. being able to print larger). In this respect they fail, in a similar way as not taking the price into consideration. But for both of these items it is hard (impossible?) to devise objective measurements on the raw data or even just a meaningful coefficient or weighting factor to apply to the final results. We are in the realm of subjective criteria and intended use which cannot be objectively measured.

I believe, and I think Michael agrees with me, that both objective measurements and subjective evaluations are appropriate and complimentary. After all, even in the High End audio world, measurements are still being used which often correlate with the subjective impression,s as any reader of the standard publications like Stereophile will be able to attest to, although measurements have progressed a lot from the time of simple S/N, Total Harmonic Distortion and turntable Wow & Flutter measurements.

[Gary Gray]

I have to agree with Michael's assessment of the significance of rating camera sensors with DxO.  I've always believed that the proof was in the print. (no pun intended)  The DxO thing is an interesting point of reference, but I've never seen any correlation between a DxO mark rating of a camera and how a photograph appeals to the eye.  The very things that make a photograph interesting are the things that can't be measured.  Normalizing resolution to a 8x10 print is pointless, at least for me.  I only make 8x10 prints for family members who want snapshots I took.   Just about any camera will look like just about any other camera at 8x10, so what's the point of the exercise?  I give Michael credit for at least taking a stance on the issue.  He's actually got a fairly good "eye" for judging the significance of things. I think so anyway.

This is a sensible write up. I cannot but agree with the notions underlined in this article. The DxO MFDB issue is a challenge since one needs to consider what is being measured. If we agree that MFDB raw is half baked, aren't we doing the dSLR camera injustice by considering theirs 'baked enough'? After all, neither raw file is the final 'viewable' result.

If you're comparing raw files (in an attempt to disengage from the intricacies of raw development) why would the DxO comparison be invalid? On the other hand, indeed it sounds like it is... Maybe it would have been best for DxO to refrain from getting in that mousetrap in the first place.

On the other hand, I think I partially disagree with Michael's assertion about DxO not taking resolution into account. In some sense they do, in some sense they don't.

They do, because they publish results normalised for a standard 'print' size of 8x10. In this respect increased resolution results in an increase of their measured metrics (as they explain in their site).

On the other hand, they don't consider the increased USEABILITY of the larger resolution files (e.g. being able to print larger). In this respect they fail, in a similar way as not taking the price into consideration. But for both of these items it is hard (impossible?) to devise objective measurements on the raw data or even just a meaningful coefficient to apply to the final results. We are in the realm of subjective criteria and intended use which cannot be objectively measured.

I believe, and I think Michael agrees with me, that both objective measurements and subjective evaluations are appropiate and complimentary. After all, even in the High End audio world, measurements are still being used which often correlate wuth the subjective impression,s as any reader of the standard publications like Stereophile will be able to attest to, although measurements have progressed a lot from the time of simple S/N and Wow & Flutter measuremants.

[NikosR]

It might be useful to quote here a portion of the DxO press release regarding the availability of their MFDB measurements. I only quote this because I think it provides some food for discussion and some counterarguments to some of Michael's thoughts:



'How is it possible to make valid comparisons between cameras with widely disparate sensor sizes? It's possible by reviewing the cameras' rankings for each of the three separate metrics that make up the DxOMark Sensor scale (Color Depth, Dynamic Range, and Low-Light ISO).

For example, if medium-format cameras do not receive top marks on the overall DxOMark Sensor scale because of their inherent Low-Light ISO limitations, DxO Labs has found these models' Color Depth and Dynamic Range performance to be very striking when compared to high-end DSLRs. Also, despite the clear challenge from DSLRs across all DxOMark sensor metrics, medium-format cameras still lead the way for large-print photography because of their very high resolution performance capability.'

(ΒΤW I don't think that even Michael would argue with the notion that most MFDB's do suffer comparatively at high ISO, even after their output is 'baked' in the proprietary or 3rd party raw converters.)


PS. I do believe that DxO measurements CAN be largely trusted and DO relay much of what can be exposed by subjectively evaluating the cameras, though one really HAS to understand what is being measured and how it is presented.

I believe the biggest problem in the DxO Mark testing is the inclusion of their DxO Mark SCORE (a marketing invention, no doubt). This is what confuses people, since it is produced by quite arbitrarily weighting the different measurement results and this is what often keeps people from really delving deeper into the DxO Mark measurements.

[michael]

Nikos,

DxO's argument about MF backs and high ISO does not address the issue of reporting on raw data that has not been noise reduced on one type of camera and yet doing so on another. The playing field isn't level.

Michael

[NikosR]

Nikos,

DxO's argument about MF backs and high ISO does not address the issue of reporting on raw data that has not been noise reduced on one type of camera and yet doing so on another. The playing field isn't level.

Michael

Michael,

I'm not sure I'm getting your point. Failure of MFDB manufacturers to optimise their S/N ratio by using various methods in both the analog and the digital domain, both in HW and in Firmware (assuming that dSLR manufacturers do indeed perform digital noise reduction prior to committing to raw) and relying only at digital level noise processing at post, cannot surely be discounted as 'it's just the way they work'.  Post processing noise reduction can be also applied to the dSLR raw output skewing the viewable results as we all know.

 Where do you draw the line and why?

[andrewh]

Michael,

I'm not sure I'm getting your point. Failure of MFDB manufacturers to optimise their S/N ratio by using various methods in both the analog and the digital domain, both in HW and in Firmware (assuming that dSLR manufacturers do indeed perform digital noise reduction prior to committing to raw) and relying only at digital level noise processing at post, cannot surely be discounted as 'it's just the way they work'.  Post processing noise reduction can be also applied to the dSLR raw output skewing the viewable results as we all know.

 Where do you draw the line and why?

The issue is which sensor can get the ultimate best quality - however achieved (We are not talking about in camera JPEG here). MF backs take a different approach to Nikon which also takes a different approach to Canon. DxO's measurements could favour a particular approach (and certainly favour low resolution/density chips like the D3 since they are not concerned by resolution but are concerned with noise). It would be interestiog to consider what a sensor designer would do to artiifcially maximise his DxO score. I am sure the scientists at Canon and Sony coiuld come up with a super high score on a chip that delivered horrible pictures (THD = 0.001%)

Andrew

[michael]

You call it a "failure" on the part of MFB makers.

I'm not an engineer, but I have spent quite a bit of time talking with the guys that design the circuit boards, write the firmware and also the software, and if you were sitting with them at the pub over a beer and described their work as a "failure" I think you'd get a bit of an argument. They have solid engineering reasons for doing what they do, just as Canon, Nikon et al have for their approaches.

The point is really simple. DxOMark's tests are done on raw data from the camera, therefore any maker that choses to do their noise normalization in software rather than in firmware is penalized, just as are readers who are unaware of this fact.

Michael

[NikosR]

You call it a "failure" on the part of MFB makers.

I'm not an engineer, but I have spent quite a bit of time talking with the guys that design the circuit boards, write the firmware and also the software, and if you were sitting with them at the pub over a beer and described their work as a "failure" I think you'd get a bit of an argument. They have solid engineering reasons for doing what they do, just as Canon, Nikon et al have for their approaches.

The point is really simple. DxOMark's tests are done on raw data from the camera, therefore any maker that choses to do their noise normalization in software rather than in firmware is penalized, just as are readers who are unaware of this fact.

Michael

Sorry, but if your data is inherently comparatively noisy it is a failure in my book. Your argument above assumes all S/N optimisation takes place in the digital domain, which is NOT true and also assumes that dSLR manufacturers do perform noise reduction processing in the digital domain before commiting to raw which AFAIK is not proven and even if its true, good for them.

If I take your argument to its natural limit it would mean that testing or comparing raw data IS NOT a valid point of comparison for ANY digital camera. Fair enough, we are at the mercy of the raw converters then be them proprietary or 3rd party, which is opening a whole new can of worms as you very well know.

[michael]

Yes, they indeed do do noise reduction in the digital domain, and yes, indeed good for them.

But the cake ingredients and the baked cake are not the same thing. As for me, I prefer my cakes baked, which is an analogous way of saying – it's what one sees that matters, the rest is just baking – to mangle a metaphor.

Michael

[NikosR]

Yes, they indeed do do noise reduction in the digital domain, and yes, indeed good for them.

But the cake ingredients and the baked cake are not the same thing. As for me, I prefer my cakes baked, which is an analogous way of saying – it's what one sees that matters, the rest is just baking – to mangle a metaphor.

Michael

I don't subscribe to this point of view. I like to understand WHY I'm seeing what (I think) I'm seeing and WHY I'm not seeing what I should be seeing. Subjective evaluation and technical testing go hand in hand otherwise we're talking about magic.

Sure, technical measurements might not cater for what one is seeing but in this case questioning and investigation should proceed concurrently in two ways:

1. Make sure I'm seeing well (i.e. am I looking into the right things, am I not masking differences by introducing other variables when the differences would be obvious if I was looking at something else etc. etc.)

2. Check that I'm measuring the right things in the right way and if not try to improve on what and how I'm measuring.

Both of the above are equally important. After all, it's obvious that the sun is revolving around the earth isn't?

[douglasf13]

I don't subscribe to this point of view. I like to understand WHY I'm seeing what (I think) I'm seeing and WHY I'm not seeing what I should be seeing. Subjective evaluation and technical testing go hand in hand otherwise we're talking about magic.

Sure, technical measurements might not cater for what one is seeing but in this case questioning and investigation should proceed concurrently in two ways:

1. Make sure I'm seeing well (i.e. am I looking into the right things, am I not masking differences by introducing other variables when the differences would be obvious if I was looking at something else etc. etc.)

2. Check that I'm measuring the right things in the right way and if not try to improve on what and how I'm measuring.

Both of the above are equally important. After all, it's obvious that the sun is revolving around the earth isn't?

  I like to know why as well, but the point is that from capture to output, there are many variables in between that make drastic differences.  It seems out of sorts when someone points to reviews/numbers to show why their A900/D3x/5Dii is the best thing out there, and then they go off and use Lightroom to process.  RAW conversion itself is so nuanced that it deserves it's own large section in any technical review of a camera.  

[Digital Darkroom in Dutch]

Beyond Color Depth, Dybanic Range and Low-Light ISO, not only Resolution should be taken into account, but also Individual Taste and Cultural Habit when the quality of various digital camera's is being compared.

And what is more, how does the printer perform? Michael referred back to the audio world. Well, not only the amplifier's specs made the quality of the sound but also the  speakers.

The evaluation has to be based on measurable data together with unmeasurable factors. As long as this is reckognised, we should be glad to have the high quality DxO data as a sound basis to start our evaluation from. Furthermore, the measurable data are a reference point for camera manufacturers.

The proof of the pudding is in the eating. It may look delicious, but how does it taste?

[Jack Varney]

The MFDB backs do not produce jpegs, therefore since Raw conversion is mandatory and noise reduction can be done here, there is no reason for on chip noise reduction. If the DSLRs didn't produce jpegs I wonder if they would bother with noise reduction on chip.

[Panopeeper]

DxO's argument about MF backs and high ISO does not address the issue of reporting on raw data that has not been noise reduced on one type of camera and yet doing so on another

If the DSLRs didn't produce jpegs I wonder if they would bother with noise reduction on chip

The only DSLRs applying noise reduction to the raw data are some Sonys. I don't believe DxO have used Sony NRed raw files for anything. The following captures show the effect of Sony NR on the noisiest channel in the sample, namely the red (I don't know which level of NR was selected, I guess it was High).

No other DSLRs are performing noise reduction on the raw data, except single pixel related cleansing, which has nothing to do with the generally accepted meaning of "noise reduction".

[NikosR]

The MFDB backs do not produce jpegs, therefore since Raw conversion is mandatory and noise reduction can be done here, there is no reason for on chip noise reduction. If the DSLRs didn't produce jpegs I wonder if they would bother with noise reduction on chip.

I think that both Gabor and I have responded to this argument, put forward by Michael in his essay, quite satisfactorily in various threads. In a nutshell, it sounds very much like a a red herring argument.

[pegelli]

In some other threads the analogy between the DxO testing of sensors and HiFi testing from years ago was pointed out. In there people were "hearing" stuff differently from what the measurements showed, which in turn led to better measurements (don't know if they ever fully aligned)

So don't forget that DxO with these measurements and scores is only a few month old and the fact people are "seeing" stuff differentlty from what the measurements show is for me not surprising at all. Let's reward and encourage DxO with positive input for going out there in the first place, rather than bash them for not getting it fully right (according to some). I'm sure as time goes on these measurements and alignment with what we see will improve.

Before DxO the only comparisons that were mostly available were subjective noise comparisons on in camera jpg's in photo magazines. From my perspective DxO is a huge leap forward.

Also for me in the end the sensor is only one aspect of a camera, it's also range of available lenses (both in your own closet and as well as the ones on the market), ergonomics, weight/size, available accessories, etc etc.

So to answer the question "eyes or numbers" it's still eyes for me but the numbers are pretty darn usefull as well, imperfect as they may be.

[Ray]

This disparity, disconnect, or whatever you want to call it, between DXO measurements and and visual experience, should be demonstrated so we can all understand, and see for ourselves, what the issue is about, precisely.

Michael has already demonstrated that on a particular type of subject, at an A3+ size, there's virtually no discernible difference between a G10 and a P45+, despite the fact that the DXO results imply that there should be a difference.

Michael now seems to claim something along the lines of the opposite. That despite the DXO test results indicating that the D3X is very slightly better than, or the same as, the P45+ at the normalised size of 8x12", the eyes tell a different story, that the P45+ is clearly better, and presumably not just in terms of resolution which we already understand has to be better.

I can't dispute this unless I see some comparisons along the lines of the G10/P45 comparison. Seeing is believing.

C'mon Michael. Back up your assertions with some sample comparisons   .

[Nick Rains]

This disparity, disconnect, or whatever you want to call it, between DXO measurements and and visual experience, should be demonstrated so we can all understand, and see for ourselves, what the issue is about, precisely.

Michael has already demonstrated that on a particular type of subject, at an A3+ size, there's virtually no discernible difference between a G10 and a P45+, despite the fact that the DXO results imply that there should be a difference.

Michael now seems to claim something along the lines of the opposite. That despite the DXO test results indicating that the D3X is very slightly better than, or the same as, the P45+ at the normalised size of 8x12", the eyes tell a different story, that the P45+ is clearly better, and presumably not just in terms of resolution which we already understand has to be better.

I can't dispute this unless I see some comparisons along the lines of the G10/P45 comparison. Seeing is believing.

C'mon Michael. Back up your assertions with some sample comparisons   .

I've read a lot of these threads about subjective or objective, apple and oranges, DxO this and dpi that  yada yada yada...

No-one, and I mean no-one, has ever actually come up with a comparison that is a true level playing field both technically and aesthetically. This is because such a beast does not exist.

The only way you can come close is to very tightly define the task for which the camera is to be used - it is merely a tool at the end of the day. Only then can you attempt to make a meaningful judgement. For instance a D3X is 'better' than a P45 on a Cambo Wide at sport and wildlife (but again only if we are talking about hand held, long lens, high shutter speed work). It's not a better camera, it's just better at that specific task.

This is why DxO testing and rating can never be broadly meaningful - all it does is rate camera sensors by it's own narrow set of rules which may or may not be relevant to the real world. By it's own rules the D3X is indeed better than a P45 back.  I could make the argument that my old 400D is clearly superior to a 1DsMk3 - if I use price and weight as my criteria.

Even print making is suspect as a criteria - size does matter. I'd venture to suggest that a 6x4 print from a high street lab would look much the same from a D3X or a P45 - although forn low light work the D3 would clearly 'win'. So if your test criteria is 6x4 prints then the D3 wins. OTOH if we set the print criteria at 30x40 then the balance shifts. The P45 will look superior generally, except again for low light or action work.

There are too many variables but at the very least I'd like to see the task, or intended end use, included in any discussions about camera superiority - without it any arguments are pointless.

[barryfitzgerald]

I have been strongly against raw noise reduction, there was a major issue with this on the A700 (which was addressed via firmware update)
Though if you apply high ISO NR on the sony's, it does effect the raw data (why they do this, is unknown, should be just jpeg really) Off does not appear to have any influence on the raw.

Back on topic, hard to disagree with anything in the article, logical and sensible..I never went on numbers myself, so yes..the real world is more important. I take the numbers with a pinch of salt..so should most

[BernardLanguillier]

In some other threads the analogy between the DxO testing of sensors and HiFi testing from years ago was pointed out. In there people were "hearing" stuff differently from what the measurements showed, which in turn led to better measurements (don't know if they ever fully aligned)

Yet, it remains to be shown how exactly their results differ from what one sees...

Other than the fact that a P65+ A2 print looks nicer than the same print from an A900/D3x, I have not read any convincing example from Michael to explain what exactly he means by this supposed gap between perception and numbers.

If anything, he commented in a recent thread that most backs and DSLRs have the same 12 stop DR in real world applications, which is basically exactly what DxO is saying also.

Is the disconnect real?

Cheers,
Bernard