3 Feb, 2009 - Eyes vs. Numbers - Which to Believe

[Nick Rains]

Is the disconnect real?

Cheers,
Bernard

Maybe  

Actually I think that a master printer / digital guru could get a better print out of a D3X file than a lesser being with a P45. Up to say 20x16, maybe 20x30. if you want to go bigger then I think the P45 will 'overtake' the D3.

My opinion is that almost all modern sensors in mid and top line cameras are pretty similar in their tonal range and a well made print within the native res of the original file will look excellent. I think you'd be really hard pushed to pick a P45, P65, D3X, a D3, a 5D, a 5DMk2 etc if you were shown prints from each at A3 size (420x297mm).

[BernardLanguillier]

Maybe  

Actually I think that a master printer / digital guru could get a better print out of a D3X file than a lesser being with a P45. Up to say 20x16, maybe 20x30. if you want to go bigger then I think the P45 will 'overtake' the D3.

My opinion is that almost all modern sensors in mid and top line cameras are pretty similar in their tonal range and a well made print within the native res of the original file will look excellent. I think you'd be really hard pushed to pick a P45, P65, D3X, a D3, a 5D, a 5DMk2 etc if you were shown prints from each at A3 size (420x297mm).

Absolutely, but DxO never claimed to be taking resolution into account.

What you write supports the contention that DxO numbers are actually a good representation of what our eyes see and there is in fact no disconnect.

Cheers,
Bernard

[ErikKaffehr]

According to the manual it won't be effective below ISO1600. According to Sony it was added because DRO induces extra noise. I'm not certain about any of this.

Best regards
Erik

I have been strongly against raw noise reduction, there was a major issue with this on the A700 (which was addressed via firmware update)
Though if you apply high ISO NR on the sony's, it does effect the raw data (why they do this, is unknown, should be just jpeg really) Off does not appear to have any influence on the raw.

Back on topic, hard to disagree with anything in the article, logical and sensible..I never went on numbers myself, so yes..the real world is more important. I take the numbers with a pinch of salt..so should most

[Nick Rains]

Absolutely, but DxO never claimed to be taking resolution into account.

What you write supports the contention that DxO numbers are actually a good representation of what our eyes see and there is in fact no disconnect.

Cheers,
Bernard

I don't doubt that they are accurate - it's the significance that I dispute.  They are only a good representation of what we see in very general terms, as soon as you apply a 'task' then they can fall down badly.

Hence my assertion that there can be no broadly meaningful tests, only highly specific ones.

[BernardLanguillier]

I don't doubt that they are accurate - it's the significance that I dispute.  They are only a good representation of what we see in very general terms, as soon as you apply a 'task' then they can fall down badly.

Hence my assertion that there can be no broadly meaningful tests, only highly specific ones.

That's true indeed.

Cheers,
Bernard

[Peter McConvill]

A long time lurker I decided to finally register and add my voice.....

I agree with MR's take on the DxO Mark application for a bunch of reasons.  In addition to MR's points I suppose my real concern is the assumption that you can combine probably hundreds of engineering specs for a sensor into just three or four measures, take a single data point of those measures and extrapolate that out to create a single number that says (or at least implies) that one sensor will always have better IQ than another in all shooting situations.

Many test sites (and photographers) make the mistake of assuming that performance at the extremes is indicative of performance in the 'core' operating areas when in fact the opposite is very often the case.    How often have you read something like the XX lens has better IQ than YY because YY is slightly softer in the corners when wide open while totally ignoring that neither lens is at their best then and that in their respective sweet spots YY actually moves ahead.  Similarly with sensors, there are sensors that produce a lot of 'shot' noise but control thermal noise very well, hence they are extremely clean at low isos but fall away fast as iso rises.  Others have well controlled shot noise so work well at high isos but have more thermal noise so are actually noisier low down.  In this situation how can any single data point seek to define IQ?

So all in all I think we need to be extremely wary of any of these quasi engineering level tests.

But this doesnt mean these tests (whether its DxO Mark or DPReviews infamous 100% crops) are entirely pointless.  They are useful as diagnostic tools to try an understand why we see what we see.  If you see a photo from xx and yy and you find you prefer yy but cant quite understand why then these 100% crops or DxO data analyses can help.  If you are trying to optimise your particular camera's processing then examining these sorts of tests can assist.  But really thats about it.  These tests can explain what you have seen but cannot tell you what you will see.

[Ray]

This is why DxO testing and rating can never be broadly meaningful - all it does is rate camera sensors by it's own narrow set of rules which may or may not be relevant to the real world. By it's own rules the D3X is indeed better than a P45 back.  I could make the argument that my old 400D is clearly superior to a 1DsMk3 - if I use price and weight as my criteria.

Nick,
You would be spot on if you were to make the argument that your old 400D is superior to the 1Ds3 in terms of price and weight. Who could argue with that, assuming that a lower price and a lower weight actually is superior, or better, or more desirable?

DXO not only rates the cameras it tests by its own narrow set of rules, but also by the rules and jargon that most of us use when assessing, discussing and arguing about a camera's performance. Issues such as resolution are largely a matter of pixel count and are also greatly influenced by choice of lens, so I can't see much point in their testing the sensor for resolution. Cameras without an AA filter seem to have a marginal resolution advantage at the expense sometimes of additional moire.

DXO tests for signal-to-noise, dynamic range, tonal range and color sensitivity at various ISOs. They produce two sets of results; one at a pixel for pixel view on the screen, and the other at a downsampled size of 8x12" at 300 ppi, which represents an 8mp image.

What else should they be testing? My personal view is that they should have two sets of normalized images, 8x12" and 16x24". But maybe that would introduce additional problems regarding the manner of interpolation.

If DXO is failing to test for certain important factors that influence image quality, at the sensor level, then what are they? We all know that the final print can be influenced by a whole range of post processing factors, including the choice of raw converter and the operator's skill in Photoshop.

One can hardly expect DXO to objectively review the performance of converters since they are in the business of producing their own raw converter.

[Ray]

How often have you read something like the XX lens has better IQ than YY because YY is slightly softer in the corners when wide open while totally ignoring that neither lens is at their best then and that in their respective sweet spots YY actually moves ahead.

Hardly ever. All good lens reviews test lens sharpness at the centre and at the borders, at most apertures, or at least at maximum aperture and at F8. Good resolution in the centre is generally more highly valued.  When rating lens performance with a single figure, a weighted rating will be applied that gives higher points for better centre performance and a smaller penalty for poorer edge performance.

[barryfitzgerald]

I am going to make a point here, made more than a few times before.

I see no "quality" variable for some tests, esp thing like DR. It's all very well bashing out a 12.5 stops number, but if the image is unprintable, then it's pointless statistics, v of real world use. That is just one example, of why I would not take them too seriously. Of course the quality aspect is open to debate too, hence the never ending problem.

I can see some point to tests for cameras, and the technology improves etc, but I wouldn't want to nail hard numbers down on them.

The good news is that nobody has yet, made a software solution for "measuring a great photograph" It could provide statistics on composition, exposure. colour balance, noise, sharpness..and provide an in depth analysis, even offer some suggestions as to how to improve the shot. That way we could simply batch run our photos through it, and it could auto discard the bad ones, and rank the good ones. Wow..now I gave somebody an idea ;-) lol.

Sometimes, you just need to trust your eyes..and use that as the main influence

[LEdgars]

There are nothing wrong with DxoMark measurements, but rather interpretation. There is so many misinterpretations around forums. To understand what exactly mean each number you should study a DxoMark measurement data technology.
May be I’m wrong, but DxoMark technologies is made (and described) by physics/engineers, not photographers. They manly studied photosite physical characteristics, but not whole sensor as a tool for capturing. For example for photographers much more important is to know sensor resolution changes at different ISOs than error of photosite response to light (SNR18%). Furthermore for me it is much more interesting to know whole sensor dynamic range, than only in shadow.

[JRSmit]

Being involved in high end audio on a professional basis, all i can say that measurements as commonly shown in brochures and reviews have little if any bearing on the sound quality delivered. For the electronics, it is simple, the more feedback the lower the distortion. BUT, no one has been able to prove that is then sounds better, on the contrary. The best sounding electronis are still those that are already well designed and wel performing without feedback. In other words, how the measured result is achieved (what is in the black box) is not shown directly in the measurements, but is fundamental in the perceived performance. None of the measurements typically used in audio have any bearing on how human hearing perceives sound.
When the audio world went from analog to digital, initally the quality of the digital processed sound was basically very bad. Perceived as harsh, flat (no 3-dimensionality), lifeless, unbalanced, definitely not engaging.
One of the marketing themes was cheap reproduction electronics, no need for very expensive turntables et to get high quality. Yet within a few years the best sounding CD-players were as expensive, horrendously expensive, as the best turntables and cartridge combo's.

It actually took over 20 years to be able to fully deliver to the red-book standard (also known as the CD). But commercially it was the only viable option.
Actually already in the analog area quality was compromised to get more minutes of sound on an LP, and to make it more easy on the electronics.
I was once confronted by music makers that their standard was that it should sound great on a ghetto blaster, this was about CD's.
Adding some amount of distortion is a common practice to emphasise a particular sound in a recording.
At the end of the day a good music reproduction on a system with a good source (LP, CD, SA-CD, etc) is perseiced as emotionally engaging, 3 dimensional, and even after playing it several times a continuous discovery of new facets in the reproduced audio "image" or picture if that is a better suiting word.

Bringing this to the picture capturing and reproduction, in other words photography. There is a strong parallel.

Putting reproducable measurements and results does not give insight in the perceived picture quality, bears little relation with the visual perception, but becomes a major element in selecting a piece of equipment. Commercially it is very hard to stay in the analog solution.
However it is not just the digital part of the total chain. With some friends and aquintances we once compared pictures, using same slid film make and type, same scene, but with a Leica M and with a Canon A1 with FD lenses. Slides were put in a projector tray in random order and projected. A 100% selection of canon vs leica by all persons present. Not on color, sharpness etc. Simply on 3-dimensionality and engagement. Yet the Canon lenses measured in reviews equal or better.


Wrapping it ll up, the digital photogaphic technology is still not there, numbers are just one of the elements subjective evaluation is still the only significant element in the evaluation, and will probably remain as such, but only to those that dare to rely on it.

Jan R. Smit

[JRSmit]

This is a sensible write up. I cannot help but agree with most of the basic ideas underlined in this article. The DxO MFDB issue is a challenge since one needs to consider what is being measured. If we agree that MFDB raw is half baked, aren't we doing the dSLR camera injustice by considering theirs 'baked enough'? After all, neither raw file is the final 'viewable' result. Where does one draw the line? DxO draws it at the raw level, they do produce a levelled comparison and one is free to interpret their data at will.

If you're comparing raw files (in an attempt to disengage from the intricacies of raw development) why would the DxO comparison be invalid or unfair to the MFDBs? On the other hand, indeed it sounds like it is...since they are intended to be used with their dedicated raw converters... but who says Nikons (for example) are not intended to be used with NX?  Maybe it would have been best for DxO to refrain from getting in that mousetrap in the first place.

On the other hand, I think I partially disagree with Michael's assertion about DxO not taking resolution into account. In some sense they do, in some sense they don't.

They do, because they publish results normalised for a standard 'print' size of 8x10. In this respect increased resolution results in an increase of their measured metrics (as they explain in their site).

On the other hand, they don't consider the increased USEABILITY of the larger resolution files (e.g. being able to print larger). In this respect they fail, in a similar way as not taking the price into consideration. But for both of these items it is hard (impossible?) to devise objective measurements on the raw data or even just a meaningful coefficient or weighting factor to apply to the final results. We are in the realm of subjective criteria and intended use which cannot be objectively measured.

I believe, and I think Michael agrees with me, that both objective measurements and subjective evaluations are appropriate and complimentary. After all, even in the High End audio world, measurements are still being used which often correlate with the subjective impression,s as any reader of the standard publications like Stereophile will be able to attest to, although measurements have progressed a lot from the time of simple S/N, Total Harmonic Distortion and turntable Wow & Flutter measurements.

The problem is they sort of translate that to a standard print size as a normalisation of performance.
I have made quite recently pictures of my mother, with a Nikon D3, with a Panasonig TZ3 and a Leica M. The Leica picture is Fuji provia, scanned in a very modified Nikon LS50-ED scanner at max resolution. I use PWP (Picture Window Pro) to print all three on a 4x6 size, with significant difference in perception of the picture shown, the scanned leica beating the nikon, the nikon beating the panasonic.
Where then does this fit into the DxO approach?

[Panopeeper]

I have been strongly against raw noise reduction, there was a major issue with this on the A700 (which was addressed via firmware update)
Though if you apply high ISO NR on the sony's, it does effect the raw data (why they do this, is unknown, should be just jpeg really) Off does not appear to have any influence on the raw

As the captures I posted above demonstrate this, NR Off is now effective on the raw data.

[BernardLanguillier]

Furthermore for me it is much more interesting to know whole sensor dynamic range, than only in shadow.

You might want to do a bit of reading on dynamic range... highlights saturation point is always the start point for DR computation so DxO's DR measurement is not just looking at shadows.

Cheers,
Bernard

[EricV]

(michael @ Feb 3 2009, 04:02 PM)
The point is really simple. DxOMark's tests are done on raw data from the camera, therefore any maker that choses to do their noise normalization in software rather than in firmware is penalized, just as are readers who are unaware of this fact.

(NikosR @ Feb 3 2009, 07:18 AM)
Sorry, but if your data is inherently comparatively noisy it is a failure in my book. Your argument above assumes all S/N optimisation takes place in the digital domain, which is NOT true and also assumes that dSLR manufacturers do perform noise reduction processing in the digital domain before commiting to raw which AFAIK is not proven and even if its true, good for them.

Let's consider a very specific example of pixel data processing which improves sensor response ("noise" and hence dynamic range) without harming resolution.

[blockquote]1) Take a calibration image with no light and examine the response of every pixel.  You will probably see some systematic pixel-to-pixel offset variability.  Correct all subsequent images by subtracting the measured offset from every pixel.  (Add back a few counts of constant offset if you do not want to clip blacks at the sensor noise level.)

2) Take a calibration image of a uniform illumination source and examine the response of every pixel.  You will probably see some systematic pixel-to-pixel gain variability.  Correct all subsequent images by normalizing every pixel to its measured gain. [/blockquote]These calibrations could be performed in camera, before the raw file is written, or they could be performed later, during processing of the raw file.  An image which has these calibrations applied at any stage should be given credit for the resulting noise reduction.

I have no idea whether any camera system currently performs these particular calibrations.  If something like this is being done, at different processing stages in different systems, then Michael's point is completely valid.

[Panopeeper]

You might want to do a bit of reading on dynamic range... highlights saturation point is always the start point for DR computation so DxO's DR measurement is not just looking at shadows.

This is a recurring issue (probably a leftover ingrained from the film era). When I am saying that I need far underexposed images for the DR measurement and I am not interested in seeing any high DR scenery, then invariably some readers become incredulous.

[Panopeeper]

These calibrations could be performed in camera, before the raw file is written, or they could be performed later, during processing of the raw file.  An image which has these calibrations applied at any stage should be given credit for the resulting noise reduction.

I have no idea whether any camera system currently performs these particular calibrations.  If something like this is being done, at different processing stages in different systems, then Michael's point is completely valid.

LOL, this is good. The only system I know of doing this is MFDB (part of the raw pre-processing out of camera).

Most DSLRs offer long exposure noise reduction, which is a dark frame substraction, but it works usually only from 1sec upward, and it's effect is questionable compared to "manual" dark frame substraction.

[BernardLanguillier]

This is a recurring issue (probably a leftover ingrained from the film era). When I am saying that I need far underexposed images for the DR measurement and I am not interested in seeing any high DR scenery, then invariably some readers become incredulous.

It would seem that some people around, including very high end gear owners, believe that there is some magic analog quality to raw files that make highlights behave intependantly from the strict digital world where 255,255,255 is pure white.

One aspect where highlights mesurements might make sense in DR computation though is if sensors don't really believe linearly near clipping, which I have been suspecting for quite some time.

Cheers,
Bernard

[Eric Myrvaagnes]

It would seem that some people around, including very high end gear owners, believe that there is some magic analog quality to raw files that make highlights behave intependantly from the strict digital world where 255,255,255 is pure white.

Yes. I've been thinking of marketing a specially modified version of PS, perhaps called Ph*t*Sh*pPlus, in which all of the numeric scales run from 0 to 300 instead of just to 255. I'll bet I could sell quite a few!   

[Panopeeper]

One aspect where highlights mesurements might make sense in DR computation though is if sensors don't really believe linearly near clipping, which I have been suspecting for quite some time.

1. The "pure" pixel values are not always linear, but that is in the entire range, with the greatest effect in the very shadows. The black frame shots I asked for would be partly for the purpose of finding out how much this is.

2. I don't know of any case, where the values of individual pixels are not linear specifically at the high end. However, there are cases, particularly earlier Nikons (like the D200) with pixels, which do not saturate at the same level (same Canons too are exhibiting this, but within a narrower range). There is a demonstration of this phenomenon in http://www.cryptobola.com/PhotoBola/Exposure.htm

As current raw processors do not treate the pixels with different saturation levels (at least I don't know of that), this may cause an apparent nonlinearity.