DxOMark's new resolution magnitude: P-Mpix

[Guillermo Luijk]

DxOMark has defined a composite magnitude called Perceptual Mpix (or P-Mpix) to represent the output resolution/sharpness one can expect from a given combo: lens + sensor. According to DxOMark's definition:

"P-Mpix is the unit of a sharpness measurement. The number of P-Mpix of a camera/lens combination is equal to the pixel count of a sensor that would give the same sharpness if tested with a perfect theoretical optics, as the camera/lens combination under test."

Looking for new photo gear? DxOMark's Perceptual Megapixel can help you!



I wonder at what aperture is this magnitude defined. The one that provides max resolution for each lens? any thoughts?. I think it's a very interesting benchmark to quantify the effective resolution of a particular camera and/or lens, but the definition is still a bit vague.

Regards

[RFPhotography]

They've had that rating for a while.  Since they don't really provide any information as to how the number is derived, it would seem to be a bit of folly to put much stock in it.  They don't, as far as I recall, provide any information as to how they define or create the theoretical 'perfect' optics.  Seems more a marketing ploy than a reliable value.  Not unlike their tests which present camera performance in terms of a common number (8 MP, if I recall) as the headline number.

[jrsforums]

In addition, they only test one lens.  Makes it very difficult to tell if the results are from a typical lens or from an outlier, either very good or bad.

[Guillermo Luijk]

Seems more a marketing ploy than a reliable value.

If their figures are based in measures there is no marketing here. Another story is that they are not explaining for which precise situation the figures are calculated.

[BJL]

It would be nice if they:
- stated the aperture used [my guess: they try several and report the maximum],
and
- if/how they weight between center and edges [my guess; if their MTF graphs were center only, this is too, because they are reprocessing their MTF data into a single number.]

It would also be fun if they reported values for a sequence of aperture values: then it would help to show that increasing the pixel count does not make resolution worse at small apertures in any relevant sense, or "prevent" use of small apertures when abundant DOF is desired!

[NancyP]

I tend to pay attention to Roger Cicala, because as owner of a lens rental business, he can test 10 or 15 copies of a lens to get an idea of consistency of performance. Most reviewers have a sample size of n = 1.

[Fine_Art]

Their number troubled me a bit too when I first started using it. It is very useful for sorting lenses you would consider buying. I trust it to give better lenses a higher number, that is about as much credence as I give it. I am not sure what the absolute number means relative to the sensor megapixels. I have lenses that it rates relatively poorly (Minolta primes) that give good detail at the pixel level much higher that their number states. I did use it to buy a nikon 85 1.8G with good results.

If you really want to see how your lens operates at various apertures use the SLRGear Imatest diagrams. If you click on the graph it opens up into a window where you can move sliders around for various apertures or zooms.

http://www.slrgear.com/reviews/index.php

The downside of this is you mentally file away f4 to f8 for most lenses based on the large purple center (extreme sharpness) making you focus stack pictures for future processing that you never do. I have many landscapes like that. I also have f8 to f16s that I end up using while part of me grumps about the lost resolution.

[Bart_van_der_Wolf]

If their figures are based in measures there is no marketing here. Another story is that they are not explaining for which precise situation the figures are calculated.

Hi Guillermo,

They indeed leave a number of questions unanswered, but they do give a rough indication as to which parameters play a role in their score.

A great metric backed up by science and the industry
DxOMark’s new Perceptual MPix measurements are based on acutance and human contrast sensitivity function (CSF) published in recently-released image quality standards from the International Standards Organization (ISO) and the International Imaging Industry Association (I3A). A member of the working groups involved in image quality, DxO Labs has been working diligently with giants in the digital imaging industry such as AMD, Nokia, Kodak, Nvidia, Fujifilm, HP, RIM, Intel, Microsoft, Google, and others.

DxOMark has also relied on the very recent scientific research of CNES (the French space agency) with respect to the optimization of digital acquisition systems, notably those for satellite imagery.

Human perception (visual acuity and contrast sensitivity) play a role, MTFs play a role, the sensor's sampling density and AA-filter play a role, and since it's a relative score versus a theoretically optimal lens, it can be used to compare combinations of camera and lens.

However, as always when a single number metric is used, it doesn't tell the whole story, such as how consistent the score is across the image (center versus edges/corners). It also doesn't tell anything about the viewing distance/image magnification, bokeh rendering,  vignetting, etc., etc.

Cheers,
Bart

[hjulenissen]

Is the theoretically perfect reference lens still affected by diffraction, or are they idealizing away that as well?

-h

[Bart_van_der_Wolf]

Is the theoretically perfect reference lens still affected by diffraction, or are they idealizing away that as well?

Hi,

I think we can assume so, but we do not know its weighting in the total metric. Diffraction is the same if the aperture is the same, regardless of lens (give or take some iris shape influences), so one can still compare lenses when this parameter stays constant. Of course, a different sensor may be able to sample the diffraction pattern at a different density, which would affect the MTF. The MTF will include lens distortions and diffraction effects.

Therefore, if MTF is used as part of the equation (which is likely, because of the need to relate that to the Contrast Sensitivity curve of the human eye), diffraction effects will be in there as well.

Cheers,
Bart

[Ray]

I wonder at what aperture is this magnitude defined. The one that provides max resolution for each lens? any thoughts?. I think it's a very interesting benchmark to quantify the effective resolution of a particular camera and/or lens, but the definition is still a bit vague.

Regards

Good question, Guillermo. My guess is the P-Mpix would have to refer to the performance of the lens at it's sharpest aperture, otherwise such ratings would not make sense.

The purpose of the rating is to evaluate the perceived sharpness of a lens in conjunction with a particular camera or sensor. At small apertures (big f/stop numbers), all lenses are equal. It would serve little purpose to create an average performance of a lens, from its maximum aperture to F22 for example, in order to assess the differences between a first  rate prime used with a low resolution sensor, and a medium quality zoom used with a high resolution sensor.

What I find a bit puzzling is the 'Best at' scores at the top, as in my attached image, showing the DXO comparison between the Nikkor 24-120mm on the D800, and two high quality primes on the 12mp D3.

These scores describe the Nikkor 24-120 as being best at 35mm and F4. Now, according to Photozone, this lens actually is sharpest at 35mm and F4, but only in the centre. At the borders, the lens is much better at 24mm. Can one conclude that 'best at' refers to centre sharpness?
Not if one considers the Photozone results for the Nikkor 85mm and Sigma Art 35mm. These lenses are definitely not sharpest at maximum aperture, whether at the centre or the borders.

Clicking on the question mark next to the DXO 'Best at' scores, I get the following notice:

DxOMark Score
The DxOMark Score reports average lens-camera performances over the whole focal length and aperture ranges.
The DxOMark Score is reported using a gauge that shows the score itself as well as the range of scores over the focal range. With this gauge, photographers can view the homogeneity of the lenses image quality over their focal range.
The DxOMark Score is measured for defined exposure conditions corresponding to low-light scene with 150 lux illumination and an exposure time of 1/60s. These conditions were chosen as we believe low-light performances are very important for today’s photography and it is also important for photographers to know how well lenses perform at the widest aperture.

There's some serious confusion here.

[RFPhotography]

If their figures are based in measures there is no marketing here. Another story is that they are not explaining for which precise situation the figures are calculated.

Not at all.  Companies use measurements in marketing all the time.  It's a matter of what's being measured.  Don't know about Europe  but here in Canada cell phone companies engage in marketing based on measurements all the time.  They will all claim, for example, the fastest 4G speeds.  It's not possible that they can all be the fastest.  But they all do the measurements differently or measure different things so as to be able to make the claim.  Without the information needed to know how DxO does its lens testing,  without knowing what and how it's measuring, it's difficult to know the validity of the results.  Another member asked much the same question I did about the 'theoretically perfect optics' when he asked if they theorise away diffraction.

[Guillermo Luijk]

The point is they are not selling anything.

[RFPhotography]

The point is they are not selling anything.

No?  They don't sell software?  Being known as the top review site wouldn't have any impact on their software sales?  What about planning for the future?  Things we don't know about yet.  They don't have a design service?  That wouldn't benefit from have a very strong reputation as a review site and the number of people who read and (maybe) rely on their tests? 

[Guillermo Luijk]

No?  They don't sell software?  Being known as the top review site wouldn't have any impact on their software sales?  What about planning for the future?  Things we don't know about yet.  They don't have a design service?  That wouldn't benefit from have a very strong reputation as a review site and the number of people who read and (maybe) rely on their tests?

Come on Bob, they are not selling any of the stuff (cameras and lenses) they are qualifying through their measurements. Is that clear now?. So there is no reason why this new benchmark should be just marketing without a real technological justification.

I think lately they have become a bit more obscure in the explanation of where they figures come from, and IMO that is not the best approach to gain reputation. For example they used to provide the SNR curves but don’t do it anymore:



At this moment I find the information about how the P-Mpix are calculated insufficient to rely on it in choosing a particular camera and/or lens.

[Guillermo Luijk]

Good question, Guillermo. My guess is the P-Mpix would have to refer to the performance of the lens at it's sharpest aperture, otherwise such ratings would not make sense

I agree, but I wonder if P-Mpix models the behaviour of the lens across the entire sensor surface (which would make sense since they are comparing camera + lens, and that includes the format), or only at the centre (where sharpness reaches its max).

[RFPhotography]

Come on Bob, they are not selling any of the stuff (cameras and lenses) they are qualifying through their measurements. Is that clear now?. So there is no reason why this new benchmark should be just marketing without a real technological justification.

No, they never were.  Of course that's clear.  It was never in question.  But they are selling their technology to other commercial users.  So in that regard, being regarded as the 'best' at what they do is beneficial.  It's a big picture view.  I don't expect many on here to understand it.

I think lately they have become a bit more obscure in the explanation of where they figures come from, and IMO that is not the best approach to gain reputation. For example they used to provide the SNR curves but don’t do it anymore:



At this moment I find the information about how the P-Mpix are calculated insufficient to rely on it in choosing a particular camera and/or lens.

Agreed.  I think that was clear from the outset. 

[Dale_Cotton2]

Probably I'm missing something here, but I don't even bother with the P-Mpix score. I go into the measurement tab's data displays, and look at the field maps (particularly the one for sharpness). Here I can see how each parameter varies over the sensor area for each focal and aperture. Presumably, the same data they use to generate their raw converter lens profiles, so likely pretty good.

Sure, it would be nice to have a valid, relative ranking for each tested lens to help with decision making. But how can you know how closely their criteria resemble your criteria? For one thing: they make their measurements at 50 Lux, or some such low light level. Does this matter? Would the numbers be different if done at the light levels typical for daylight work? I have no idea. Even the second-level Lens Metric Scores seem too simplistic. If I shoot landscapes at f/8 or f/11 nearly all the time, do I care what the mean value for all measured apertures and focal lengths happens to be? How many zooms aren't soft in the corners at wide angles or sacrifice sharpness at one end of their range versus another?

But even sticking to the actual measurement displays you have to be careful. Nearly didn't buy the RX100M2, given the DxO data. What wasn't at all obvious to me is that they are reporting the results without the metadata distortion corrections applied. You can't even view the uncorrected output in Lightroom and certainly not from the OOC JPEGs. That's like holding a paralympics but refusing the use of prosthetic devices. Surely distortion correction is essential to small camera zoom lens design? At the very least make it very clear that the numbers only apply to the uncorrected output of the lens. Put a red warning note on each screen in such cases.

[Ray]

Sure, it would be nice to have a valid, relative ranking for each tested lens to help with decision making. But how can you know how closely their criteria resemble your criteria?

Hi Dale,
If you have the time and own some of the cameras and lenses that DXO have used for their tests, you could verify the results for yourself, making allowances for minor variations in lens quality control.

For example, I own a D800E, D700 and the 3 lenses shown in the DXO test results above, in my reply #10.

What I find fascinating is that the P-Mpix of the medium quality Nikkor 24-120/F4 zoom, when used on the 36mp D800, is about the same as the P-Mpix ratings for the excellent 85mm and 35mm primes when those lenses are used on a 12mp full-frame camera.

Now I don't have my D700 with me at present, just the D800E and the D7100, so I'm unable to carry out such tests for myself. However, what I would expect to find is that the zoom set at 35mm and 85mm, used on the D800E, would provide a very similar image quality to those primes used on the 12mp D700, when such lenses are used at their sharpest apertures, and the D800 images are downsampled to 12mp.

If this were to be correct, it would be useful information for those who may already be satisfied with the pixel count of their camera but who would like to upgrade their lenses. Instead of buying a few expensive primes to use with one's D700, one could just buy a D800 and continue using the 24-120 zoom on the new camera. If one doesn't need 36mp, no problem, just downsample the images to 12mp and get the image quality of a first rate prime used on the old camera.

The primes may always have the advantage of a wider aperture, but the zoom has, in many circumstances, the more significant advantages of flexibility of focal length range and image stabilization.

[Dale_Cotton2]

Ray: I have the time but not the lenses. The only camera+lens combo I have that DxO has tested is the RX100M2 with its built-in zoom that I ranted about above.

Whatever the P-Mpix is, it isn't simply a report on sharpness/res/acutance. If you go to the Measurements tab for a given lens, then look at the DxOMark Score Map chart, then look at the Sharpness chart for that same lens, you'll see a different pattern of colours. The lens I'm looking at right now is the Nikon AF-S DX Nikkor 18-55mm f/3.5-5.6G VR mounted on the D7000. The score map only shows even hints of green at the bottom left, which is wide open and wide angle. But the sharpness chart shows no green at all and seems to indicate max sharpness is yellow at f/8 in the wide angle range. Nor can I find any pattern of transmission, distortion, vignetting, or CA that seems to concentrate at the bottom left.

Colour me totally mystified.