DXOmark ranks DB image quality well below DSLR!

[Ray]

My ZD has more DR than my 5D2. It looks rather obvious. I didn't try to test it by looking at numbers and so forth and if you do it takes a lot of effort I assume to do it right. And when you bring up the shadows in the ZD it works much better than the 5D2.

I was taking some difficult shots in a forest with directly sunlight on a face but the rest of the body in shadows. The ZD did quite well with the scene. The ZD feels more like a nice quality motion picture with the 5D2 feeling like a TV show video by comparison. That is sort of how I relate to it.

It is sad that the 5D2 has that pattern noise down bellow. I assume that doesn't show up with DXO.

I almost always use the ZD at ISO 50.

Bill

Unless you can show us 100% crops of the same scene with the same lighting, the same FoV and the same shutter speed, we can't be sure about the validity of your impressions. You need to expose your methodology for us to make an assessment, otherwise it's all speculation.

[Plekto]

I stand by my remark that putting too much trust in sites like Dx0 or other similar "tools" might lead one to a purchase one later regrets or at the least, does not understand. In my case, I have shot the tools that are reasonable considerations for my purposes; I can see the differences in each; and I have purchased what will work for me (within my budget).

But the thing is that Dx0 is at least giving us SOME scientific data.  The camera makers flatly lie and throw tons of FUD and marketing claims at us.  I don't know of any other site that tests cameras based upon scientific methods.

So it is useful for at least that much, no matter what one may think of it one way or the other.

P.S. I still wish Dx0 would test the Sigma/Foveon sensors.

[billthecat]

Ray,

It's too much work for me to setup such a comparison. But perhaps I will if I have some free time this 3day weekend. Currently you have a subjective testimonial that Bill says the ZD has more DR than the 5D2 especially in the shadows. Easton the car photographer also said the same thing a while back.

Bill

Unless you can show us 100% crops of the same scene with the same lighting, the same FoV and the same shutter speed, we can't be sure about the validity of your impressions. You need to expose your methodology for us to make an assessment, otherwise it's all speculation.

[Carsten W]

There are some real issues with the DxO measurements, but you ignore those.

Carstenw,
What are they? If you don't specify them, it's understandable I would ignore them. I'm not a mind-reader.

I listed some in my post. The single-most damning flaw is that they don't publish their exact process, so that their numbers are only black magic to others.

I've never ignored resolution. Check my posts again.

I didn't say that you did. I said that Gabor was saying that resolution was important.

Is this the source of the confusion? The single number assessments under the 'overview' heading? If you are at all serious you will look at the individual graphs that detail performance at various ISO settings. The single numbers in the overview are weighted.

Man, you are one of the worst readers I have ever seen!  What do you think I set up three variables in three unknowns for? I even said so: to find out what the weights are. The weights made no sense, and it throws even more doubt onto their claims when one can't even figure out how they combine their three individual ratings into their one final rating.

I'll go back and figure out the rating weights again, and see if they have fixed anything.

For example, it has been established by me, and later confirmed by DXOmark, (that sounds arrogant, but I did show my results first) that the D3 has approximately a 1/3rd to 1/2 a stop DR advantage over the 5D at high ISO. So how come there's such a huge difference in the single number rating for low-light ISO? The D3 is rated at 22.9 and the 5D at 13.68. That's a huge difference. It doesn't seem to corelate with the graphic results under the DR heading. What does DXO mean by the expression 'low-light ISO'? It clearly doesn't mean, 'high-ISO performance'. I've got no idea what subjective elements have influenced these single number ratings. I recommend ignoring them.

Right, and so our opinions get closer. There are too many question-marks over their numbers. Some seem to match what we ourselves feel (like your D3 results), and others are way off. Their process is too secretive. I recommend using the DxO site only as another anecdotal data point, until they get more serious about this and publish proper methods. I am all for giving credit to DxO for doing the community a great service, but only once they have given up their secretiveness and list reproducible results.

[Carsten W]

Unless you can show us 100% crops of the same scene with the same lighting, the same FoV and the same shutter speed, we can't be sure about the validity of your impressions. You need to expose your methodology for us to make an assessment, otherwise it's all speculation.

That is right, but why don't you also think that this is true for DxO?

[Panopeeper]

It's too much work for me to setup such a comparison. But perhaps I will if I have some free time this 3day weekend. Currently you have a subjective testimonial that Bill says the ZD has more DR than the 5D2 especially in the shadows. Easton the car photographer also said the same thing a while back.

It is imaginable, that the ZD had greater DR than the 5D2, though it is improbable. However, Bill said :-) It feels like the ZD has many stops more dynamic range than the 5D2 but DXO doesn't show that , and I dear say flatly, that this is not so.

Pls keep in mind, that the ZD creates only 8 levels for the tonality over the 9th stop of the DR. Theoretically, if the noise is very-very low, one could say there are four levels for the 10th stop, two levels for the 11th stop, and one level for everything above. In other words, if the sensor of the ZD were so good, the ZD would be bit depth challenged, like the 5D classic.

[ejmartin]

It is imaginable, that the ZD had greater DR than the 5D2, though it is improbable. However, Bill said :-) It feels like the ZD has many stops more dynamic range than the 5D2 but DXO doesn't show that , and I dear say flatly, that this is not so.

Pls keep in mind, that the ZD creates only 8 levels for the tonality over the 9th stop of the DR. Theoretically, if the noise is very-very low, one could say there are four levels for the 10th stop, two levels for the 11th stop, and one level for everything above. In other words, if the sensor of the ZD were so good, the ZD would be bit depth challenged, like the 5D classic.

I don't think bit depth is the issue.  Here are deep shadows of my 1D3 at ISO 100:



The 1D3 only has about 11.7 stops of DR at the pixel level; 14-bit tonal depth is wasted on it.  In the above image, the bottom left square of the GM chart is about 9 stops down from RAW saturation.  There is pattern noise all over the place, and it keeps the shadows from being clean to an extent that does not show up in standard measures of DR using std dev of pixels, because it is correlated noise.  For another set of examples involving the 5D2, see the FM thread I linked to earlier, or

http://www.diglloyd.com/diglloyd/free/Push...acks/index.html

If Canon cleaned up their low ISO amplification a bit, my image could look like this:



This is the same scene with the same illumination, shot at the same Tv/Av, but at ISO 1600.   The shadows could be so much cleaner if Canon was willing to put in the effort.

[Panopeeper]

The 1D3 only has about 11.7 stops of DR at the pixel level; 14-bit tonal depth is wasted on it

On what criterion is this figure based on?

[Ray]

The single-most damning flaw is that they don't publish their exact process, so that their numbers are only black magic to others.

DXO are in the business of marketing their own RAW converter. I imagine that the exact processes they use to get their measurements will be patented. The'tool' they use to get their measurements is called a "DXO Analyzer". The information they gather about a sensor is crucial to the design of their RAW converter. Do camera manufacturers publish full details of their sensor technology? C'mon! Be sensible!  

However, DXO do publish broad descriptions of their methodology, under the heading 'Technologies'. There, for example, they describe the two different ways of measuring ISO sensitivity (in accordance with ISO standard 12232) and tell you which method they use, which is 'sensitivity for full saturation'. For those who are interested in maths, there are a also few formulas there to get excited about.

I didn't say that you did. I said that Gabor was saying that resolution was important.

Everyone reading this thread knows that resolution is important. I knew that resolution is important when I was a small boy running around in short pants. Why criticise a review for the aspects of camera performance that are not addressed. DXO also do not address price, weight and waterproofing, which are also important factors. I don't need to be told that the P65+ has higher resolution than the D3X. I can work that out for myself. However, I cannot work out for my self that the D3X has a higher DR than the P65+ without going to a lot of trouble and expense in hiring equipment. This is the sort of information that is therefore valuable.

Sensor resolution very closely follows pixel count. I'm not aware of any sensor with a significantly higher pixel count than another which also does not deliver higher resolution than the other. When sensors have a similar pixel count, there will likely be some variation in resolution depending on the strength of the AA filter, or whether the sensor has one or not. However, sensor resolution is also affected by the brand of RAW converter used, and perhaps more significantly, the 'system' resolution of any 'real-world' image, which most practicing photographers would consider more important, will depend very much on the quality of the lens used. There are probably very good reasons why DXO have not addressed resolution.

Man, you are one of the worst readers I have ever seen!  What do you think I set up three variables in three unknowns for? I even said so: to find out what the weights are. The weights made no sense, and it throws even more doubt onto their claims when one can't even figure out how they combine their three individual ratings into their one final rating.

I'll go back and figure out the rating weights again, and see if they have fixed anything.

It won't necessarily help you if you figure out the rationale behind their weighting, or the algorithm used to determine a single-figure value. The essential point about a weighting process is that there is a subjective opinion at its core. If you discover the method by which they arrive at those single-figure assessments, you will at least be in a position to determine if DXO are consistent in the application of their ratings, but you may disagree completely with the subjective decision behind their approach.

That is why I ignore their 'overview' ratings, and I would recommend you do too. The results that count are under the five headings, ISO Sensitivity, SNR 18%, DR, Tonal Range, Color Sensitivity. DXO provide full definitions of those terms. I can't understand why you are complaining   .

If you think any of their figures (apart from the subjective weightings) are way out, please demonstrate the fact and provide the evidence. That's not too much to ask is it?  

[Plekto]

I think it really comes down to the fact that a lot of people were lied to by the DB makers for years(eg - fake ISOs above ~400) and don't want to deal with the results of the bit camera makers' DSLRs.  It's not like Sony is a tiny company, after all.  One would expect the giants to eventually overtake the small DB makers or come very close - and for less cost, too.  If for no other reason than market share and advertising, which allows them to sell 10x as many units.

This happens in every case.  Take the Audi R8.  It's a mainstream auto maker that suddenly has a car that's nipping at the heels of the Ferrari and similar boutique cars.  Well, it's not too surprising if you think about just their R&D budget and number of employees.

Dx0 is good for at least that much in any case, because it clearly shows glitches and lying.  For example:

http://www.dxomark.com/index.php/eng/Image...se-One/P45-Plus
Click on the ISO Sensitivity tab.  Whatever their testing methods, there plainly is something the maker isn't telling us going on.

[ejmartin]

On what criterion is this figure based on?

Max engineering DR at ISO 200; 5 ADU read noise out of 16383 max signal.  16383/5~11.7 stops.

[ThierryH]

Why has one to read such misleading and wrong information about digital back manufacturers?!

I can at least stand here for Sinar (one of the digital back manufacturer) and affirm that this is absolut bullshit and disrespectful at the least. There might be information which doesn't come accross, there might be information that is not understood, there might be information that is not explained, there might even be information which is not given, because not relevant, but to say that digital back manufacturers are lying and purposely lying to their customers or potential customers is a serious accusation.

Sinar has answered and still is answering and addressing questions or issues without lying to endusers.

Unfortunately one can write whatever one wants under the cover of anonymity, without consequence.

For the record.

Best regards,
Thierry

... I think it really comes down to the fact that a lot of people were lied to by the DB makers for years(eg - fake ISOs above ~400) ...

[Panopeeper]

I think it really comes down to the fact that a lot of people were lied to by the DB makers for years(eg - fake ISOs above ~400) and don't want to deal with the results of the bit camera makers' DSLRs

Keep in eye, that DSLR manufacturers are doing the same, with the same success. Read the diverse forums and see, how many owners believe to shoot with ISO 6400 and above with the 5D2, 12800 and 25600 with the D3, etc., while they are in fact only underexposing.

I think it really comes down to the fact that a lot of people were lied to by the DB makers for years(eg - fake ISOs above ~400) and don't want to deal with the results of the bit camera makers' DSLRsDx0 is good for at least that much in any case, because it clearly shows glitches and lying.  For example:

http://www.dxomark.com/index.php/eng/Image...se-One/P45-Plus
Click on the ISO Sensitivity tab.  Whatever their testing methods, there plainly is something the maker isn't telling us going on.

I'm afraid DxO is wrong in this case (too). According to the few P45+ files I have, the higher ISOs are *not* fake. I wonder if they saved the cost of renting a camera for test purposes, thinking that the P45 Plus is acting like the P45.

[Panopeeper]

There might be information which doesn't come accross, there might be information that is not understood, there might be information that is not explained, there might even be information which is not given, because not relevant, but to say that digital back manufacturers are lying and purposely lying to their customers or potential customers is a serious accusation

Thierry, forget about the term used by Plekto, let's deal only with the factual background.

The eMotion 54 LV is said to have ISO up to 400, according to you in an earlier post. However, the fact is, that the e54 does not have different ISOs at all; ISO 100 and above are plain underexposures, and the so-called ISO is nothing more than metadata telling to the raw processor to boost the intensity. Many customers (I dare say the vast majority of the customers) don't know this. Don't you think that if they knew this, they would be more conservative in their using of ISO 100 and above?

[Panopeeper]

Max engineering DR at ISO 200; 5 ADU read noise out of 16383 max signal.  16383/5~11.7 stops.

How do you calculate it for Nikon cameras?

(Btw, you forgot to substract 1020, but the difference is not much.)

[Ray]

I'm afraid DxO is wrong in this case (too). According to the few P45+ files I have, the higher ISOs are *not* fake. I wonder if they saved the cost of renting a camera for test purposes, thinking that the P45 Plus is acting like the P45.

Gabor,
How do you reconcile the above statement with the following statement from Emund Ronald about the high ISO performance of his new P45+?

Although indications are one can go pretty far down and have usable pictures, it would seem that ISO 1600 is better reached by underexposing ISO 100 than by underexposing from ISO 400.

http://luminous-landscape.com/forum/index....c=21763&hl=

[ejmartin]

How do you calculate it for Nikon cameras?

For the read noise, one can either access the masked pixels and look at their std dev, or fit a straight line to the noise variance vs exposure, whose intercept is the read noise variance (take the sqrt to get the std dev).  For the latter method, it is best to take differences of pairs of raw files in order to eliminate systematic errors.

[Panopeeper]

How do you reconcile the above statement with the following statement from Emund Ronald about the high ISO performance of his new P45+?

I don't. However, it seems (as far as I can measure it from the few raw files I have), that the higher ISOs of the P45+ could be fake with the same result, i.e. increasing the ISO does not reduce the noise. Like ISO 3200 with the 5D2: it is not fake but useless, except for JPEG (one could say it is an analog fake, while 6400 is fake analog .

[Panopeeper]

For the read noise, one can either access the masked pixels and look at their std dev

But they don't record any (except for the D300). Did you think I asked specifically the Nikons by chance?

or fit a straight line to the noise variance vs exposure, whose intercept is the read noise variance (take the sqrt to get the std dev).  For the latter method, it is best to take differences of pairs of raw files in order to eliminate systematic errors.

I find this not reliable enough even with the latter method, particularly with 12bit depth. A small difference in the "read noise" causes a large difference in the engineering DR.

[Carsten W]

DXO are in the business of marketing their own RAW converter. I imagine that the exact processes they use to get their measurements will be patented. The'tool' they use to get their measurements is called a "DXO Analyzer". The information they gather about a sensor is crucial to the design of their RAW converter. Do camera manufacturers publish full details of their sensor technology? C'mon! Be sensible!  

Okay, if they don't want to publish their methods, then they also cannot expect to be used as anything more than another data point. In this case, their values should in no way change the way people do research before buying a new camera, and in that case, what good are the numbers at all?

Everyone reading this thread knows that resolution is important.

Then why doesn't DxO attack it in some way? This I don't understand. One hypothetical example: Let's say that a 15MP has slightly higher dynamic range than the P65+. By downsampling the P65+ image to 15MP, you can then boost the DR of the image, getting superior results. In other words, just looking at DR in isolation is not so useful. You must look at DR and resolution together, or at least, at the same time, to get the whole picture for your personal needs.

It won't necessarily help you if you figure out the rationale behind their weighting, or the algorithm used to determine a single-figure value. The essential point about a weighting process is that there is a subjective opinion at its core. If you discover the method by which they arrive at those single-figure assessments, you will at least be in a position to determine if DXO are consistent in the application of their ratings, but you may disagree completely with the subjective decision behind their approach.

Sure, there is subjectivity everywhere, and that is fine, but there is too much black magic on the DxO site to properly separate fact from opinion, which reduces the usefulness of their published numbers. They need to be more open here, even if they don't reveal everything.

If you think any of their figures (apart from the subjective weightings) are way out, please demonstrate the fact and provide the evidence. That's not too much to ask is it?  

This is a forum, not a panel of scientists. We are all trying to provoke the others into doing all the work  I currently have 6 cameras, but only one is digital, so I cannot compare anything.