DxO marks

[Rory]

Great article.  A few minor things to correct:

• the D800/D800E are not APS-C products (in figure 2a discussion)
• in the blue comma example, the background may be black for some readers
• "by simulating the wedding photographer’s shooting without with only ambient light in the ambience"
• This allows Canon to switch to switch from off-chip analog to digital converters to per-column ADCs

[svein]

Great article.  A few minor things to correct:

Agree really interesting article, appreciate the time spend researching and writing it.

A couple of typos on sensor sizes:

• "smallest (28 mm2) sensors as used in the Fujifilm X-10 or Canon S-110"
• "so-called 1/2.33” sensors used in the Fujifilm X-10 and Pentax Q (crop factor of 5.6×),"

X10 use a (so called) 2/3" sensor which is slightly bigger than the one in S110, and quite a bit bigger than the one in Pentax Q.

[Peter van den Hamer]

Thanks, Rory and Svein, for spotting these errors. I will see if I change the blue comma - it is indeed different on screen (on this site) than on paper.

[DaveCurtis]

Brilliant article.

Amazing to see how the sensors have evolved over the years.

I am now waiting for Canon to up the ante on DR front

[Ray]

Excellent article, but a bit daunting for someone who may be just trying to get beyond the overall scores which are weighted and sometimes a bit misleading.

I would take issue with the following comment;

Another way to look at the relevance of resolution: nowadays, unless you use expensive lenses on a relatively cheap camera, cameras tend to have enough resolution to handle what the lenses can project onto the sensor. And for most uses, 12-18 MPixels is more than enough anyway. So a properly designed noise benchmark can be used to predict image quality as long as you keep an eye on whether you have enough resolution for your needs.

I'm not happy with this concept that a camera can have enough resolution to handle what the lens can project onto the sensor.

The recorded resolution is always a product of both lens resolution and sensor resolution. Increase the resolution of either one, and the recorded resolution will also increase.

In effect, when one upgrades to a sensor with more megapixels, one automatically upgrades all one's lenses as a free bonus.

To take an example from DXOMark's lens database. The relatively cheap, standard Canon 50mm F1.4 on the 12.7mp 5D has a resolution of 55 lp/mm, according to DXO's methodology.

That same lens on a Canon 5D3 with less than double the the pixel count of the 5D, has a resolution of 63 lp/mm. If the 5D3 were to have the resolution of a D800, ie. another 50% increase in pixel count, that same lens would have a recorded image of perhaps around 66 lp/mm.

The point I would make is that probably no amount of money could buy a 50mm lens that could deliver 66 lp/mm or even 63 lp/mm on a 5D. Such a lens has not been produced.

The presence of more pixels on the sensor also has the effect of upgrading one's lenses in terms of the range of effective focal lengths in relation to the standards of a lower pixel-count sensor.

For example, supposing one were to consider that the 16mp of the Nikon D7000 is sufficient for one's purposes and that the 36.3mp of the D800 is far more than one would ever use because one never makes prints larger than A3+. How short-sighted would one be?

What the upgrade from a D7000 to a D800 does in effect, is not only upgrade the resolution of one's lenses, but also effectively converts all one's prime lenses into telephoto lenses in relation to those standards of the lower-pixel-count D7000, and extends the range of all one's zoom lenses in relation to the standards of the D7000. Such an upgrade is almost priceless.

For example, a 50mm, high quality prime on a D800 becomes in effect an extremely high quality 50-75mm DX zoom by D7000 standards.

That great lens, the Nikkor 14-24/2.8, on the D7000 becomes effectively a 21-36mm DX zoom. However, on the D800, by the DX standards of the 16mp D7000, it becomes a high quality 14-36/2.8 DX zoom. No such lens exists, and if it did, it would be hugely expensive at the same quality.

[HSway]

A nice Saturday’s read yesterday for me. It reminded me of a smell of academic periodicals' pages. I mean the physical smell, long before the internet. Actually, I realized, seldom I read on the web articles similar to these Saturday’s delights these 19th cent. established periodicals were bringing me.

The dxomark is really great and we are lucky to have its systematic data in this useful format at disposal. Thinking a moment about it, I, in fact, can’t imagine the digital photography community being without this well-arranged evaluative system. Many good points in the article, the dxomark is certainly not a complete answer to any question but is good to have it and I imagine the confusion, major errors and bias on the web would have much easier life without it.

Thanks for the article,

Hynek

[AlanRussell]

Peter- Many many thanks for a really superb article.  Best thing I've read on the best Photo Site in the Universe.  (So good that I am replying after many years of just lurking on LuLa.....).

The one thing my fuzzy little brain has trouble with is the rather huge Elephant in the Room with all of this (which, your article  correctly didn't cover- being a discussion of DxO marks specifically) -  i.e. diffraction softening - surely, in everyday practical terms, the biggest limitation to image quality, along with resolving power of the lenses.

Cramming more and smaller pitch sensels onto chips means that we will be limited to a very restricted range of apertures with these new high MP count dslrs such as the D800.

As a vague rule of thumb I've always worked on the 1/Pixel Pitch in Microns = smallest aperture worth using e.g. 8microns gives f8 (depends on how fussy you are with circles of confusion and all that of course).

The whole issue of how many MP we can cram on to a 35mm sensor, once discussed regularly seems to have been sidelined as SNR ratios have improved but surely we are close to some sort of nyquist limit already - as Scotty from Star Trek said "ye canny change the laws o physics, Jim".

I speak as a product photographer who needs both depth of field and the highest resolutions possible and who, being Scottish, would much prefer to spending £3K on a DSLR to £30K on a new back to achieve both decent Dof and Res.

I know very little of these laws o physics but I'd really appreciate your thoughts..

Yours aye,

Alan

[ErikKaffehr]

Hi,

You could check a few of my articles:

http://echophoto.dnsalias.net/ekr/index.php/photoarticles/68-effects-of-diffraction

http://echophoto.dnsalias.net/ekr/index.php/photoarticles/49-dof-in-digital-pictures?start=1

On the other hand, diffraction is very beneficial to sharpening. So optimal sharpening can recover a lot of contrast lost to diffraction.

Check this article by Tim Parkin: http://www.onlandscape.co.uk/2012/07/diffraction-limited/ he finds, among other things, that a D800 image at f/22 outresolves his Sony Alpha 900 at f/8, when both images are correctly sharpened.

By the way, going MFD will not help with diffraction. On MFD you need to stop down more DoF so you loose more to diffraction, it's a zero sum game. The best option you have is to use Scheimpflug or merge several images into one.

Both methods are covered here: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/29-handling-the-dof-trap

Best regards
Erik

Peter- Many many thanks for a really superb article.  Best thing I've read on the best Photo Site in the Universe.  (So good that I am replying after many years of just lurking on LuLa.....).

The one thing my fuzzy little brain has trouble with is the rather huge Elephant in the Room with all of this (which, your article  correctly didn't cover- being a discussion of DxO marks specifically) -  i.e. diffraction softening - surely, in everyday practical terms, the biggest limitation to image quality, along with resolving power of the lenses.

Cramming more and smaller pitch sensels onto chips means that we will be limited to a very restricted range of apertures with these new high MP count dslrs such as the D800.

As a vague rule of thumb I've always worked on the 1/Pixel Pitch in Microns = smallest aperture worth using e.g. 8microns gives f8 (depends on how fussy you are with circles of confusion and all that of course).

The whole issue of how many MP we can cram on to a 35mm sensor, once discussed regularly seems to have been sidelined as SNR ratios have improved but surely we are close to some sort of nyquist limit already - as Scotty from Star Trek said "ye canny change the laws o physics, Jim".

I speak as a product photographer who needs both depth of field and the highest resolutions possible and who, being Scottish, would much prefer to spending £3K on a DSLR to £30K on a new back to achieve both decent Dof and Res.

I know very little of these laws o physics but I'd really appreciate your thoughts..

Yours aye,

Alan

[Peter van den Hamer]

The recorded resolution is always a product of both lens resolution and sensor resolution. Increase the resolution of either one, and the recorded resolution will also increase.

In effect, when one upgrades to a sensor with more megapixels, one automatically upgrades all one's lenses as a free bonus.

Ray,

"Product of" would obviously be wrong, but I guess you meant "function of".
But that claim is like saying "for a given lens, increasing sensor resolution will not reduce overall image sharpness" - which is kind of hard to disagree with 

So, given that you recommend upgrading to a high res sensor to utilize the full capabilities of lenses (which I kind of did myself: 6 MPix APS-C 10D to 21 MPix FF 5D2), let's check how much this helps using your own example: a Canon 50mm/1.4 lens, used on full-frame cameras, and measuring the max/max/max resolution using DxOMark data.

This gives 55 line_pairs/mm on a 12.7 MPix Canon 5D. Assuming the lens could keep up with the increased resolution of the 5D2 (there is no DxO lens data for 5D3 yet), it would give 71 lp/mm (= 55 lp/mm * sqrt(21.1/12.7)).

Firstly, the highest resolution for any lens on any camera measured by DxOMark so far is only 67 lp/mm. So we can't expect 71 lp/mm for a lowly Canon 50/1.4 (successor is expected). Instead, the measured data for the 50mm/1.4 lens on the 5D2 is 63 lp/mm. This is a 15% increase compared to the three years older 5D design.

This confirms your (somewhat unrefutable) claim that higher resolution sensors contribute to higher resolution images. But it also shows that lenses don't really keep up with sensor resolution increases:

• decreasing the pixel pitch by 30% (=increasing the MPixel count by 75%) only results in a 15% lineair resolution increase.
• some of that 15% increase is probably due to unrelated camera improvements. Compare the pricey 85mm/1.4D on the Nikon D300s to the D700: both have a 12 Mpixel sensor resolution, but 8% lp/mm overall resolution difference (AA filter? crosstalk? processing?). So a modern 12 MPixel full-frame camera would presumably give better results than the old Canon 5D design. So the contribution of sensor resolution to the 15% measured overal resolution improvement might be below 10%. We could test this as soon as DxOMark tests the lens on the 5D3 (newer, roughly same resolution as 5D2).
  
• the 5D had huge pixels for its time. The 8 MPixel 350D was already out. If you scale the full frame 5D down to 1.6x APS-C, you get a mere 5 MPixel camera. So the 5D had unusually low resolution for its time. A simplistic calculation for the Canon 5D says that it cannot outresolve 59 lp/mm (59 pairs/mm * 2 lines/pair * 24mm * 59 * 2 * 36mm = 12 MPixels).
  
• the single figure resolution numbers provided by DxOMark are at each lens' best aperture, at the best zoom setting, and in the middle of the image. This max/max/max measurement obviously flatters the true capabilities of the lens.
• we are doing the excercise at full-frame. Many people have smaller sensors. An APS-C sensor has 1.5 or 1.6 times smaller pixels than a full-frame sensor with the same resolution. This means that "increasing resolution to get the most out of your lenses" will probably give even less benefit for APS-C or smaller cameras. Arguably you need more than a 12 MPixel medium format camera, but these are not for sale, and users are less likely to crop there.
  

What the upgrade from a D7000 to a D800 does in effect, is not only upgrade the resolution of one's lenses, but also effectively converts all one's prime lenses into telephoto lenses in relation to those standards of the lower-pixel-count D7000, and extends the range of all one's zoom lenses in relation to the standards of the D7000. Such an upgrade is almost priceless.

The story that if you migrate from APS-C to full-frame, that you should increase resolution if you plan to crop back to APS-C makes a lot of sense, assuming that you owned an FX lens on a DX camera. Incidentally, the 36.6 MPixel D800(E) and the 16 MPixel D7000 both have a pixel pitch of 4.8-ish micrometer. The numbers are so alike, it may not be a coincidence (some product manager said "scale D7000 sensor to full-frame!"). The blue scaling line in Figure 6 shows that the D800 even performs pretty comparably to 2 (actually 2.25) D7000 sensors tiled side-by-side.

Peter

[Peter van den Hamer]

Actually, I realized, seldom I read on the web articles similar to these Saturday’s delights these 19th cent. established periodicals were bringing me.

I have written scientific articles in the past (not on image sensors, mind you). But the 19th century is even before my time.

[Peter van den Hamer]

huge Elephant in the Room with all of this (which, your article  correctly didn't cover- being a discussion of DxO marks specifically) -  i.e. diffraction softening - surely, in everyday practical terms, the biggest limitation to image quality, along with resolving power of the lenses.

The topic is indeed a bit out of scope but has actually been taken into account and is briefly mentioned twice in the article. So it is NOT an elephant in the room. I prefer cats anyway as they eat less and are cleaner.

1) See Erik Kaffehr's answer.

2) http://www.cambridgeincolour.com/tutorials/diffraction-photography.htm has a tutorial and an online calculator.

3) Note also Falk Lumo's site (quoted in the article). His equivalence criterion (summarized in text and a large table in my article) actually tells you what MF camera setup can be replaced by which full-frame configuration with the same depth of field and the same diffraction. It should help thrifty Scotchmen determine whether they can get away with a D800E instead of a MF camera. If you have a precise enough question ("am I right that X should match Y w.r.t. Z?"), chances are that Falk will help out. http://www.falklumo.com/lumolabs/index.html

[Ray]

Ray,

"Product of" would obviously be wrong, but I guess you meant "function of".

Peter,

Either product or function would be appropriate. No sensor, no recorded image. No lens, no recorded image. Let's not quibble.

But that claim is like saying "for a given lens, increasing sensor resolution will not reduce overall image sharpness" - which is kind of hard to disagree with

I'm saying more than that. However bad the lens may be, the fact that increasing sensor resolution will not reduce overall image sharpness, is a given. No need to raise the point.

What I'm saying is that 16 or 18mp, whether full frame 35mm or cropped format, is not sufficient to capture the full resolution projected by a good lens at its sharpest aperture and other apertures close to its sharpest aperture, such as F8 or F4, if F5.6 is the sharpest aperture.

In fact, my own tests comparing the 10mp Canon 40D (25.6mp full frame equivalent) with the 15mp Canon 50D (38mp full frame equivalent) show that even at F16 the 38mp image (equivalent) reveals more resolution than the 25.6mp image. At F8 and F5.6 there's no contest. Nikon can safely raise the pixel count of the D800 in future models to, say, 60mp without complaints that there is no resolution advantage.

This gives 55 line_pairs/mm on a 12.7 MPix Canon 5D. Assuming the lens could keep up with the increased resolution of the 5D2 (there is no DxO lens data for 5D3 yet), it would give 71 lp/mm (= 55 lp/mm * sqrt(21.1/12.7)).

You are correct that the figure I quoted of 63 lp/mm for the 5D3 is in fact for the 5D2. Typo error. However, the difference of 1mp between the 5D2 and 5D3 sensor is of no practical significance. One would expect the 5D3 to produce the same resolution of 63 lp/mm with that 50mm lens.

However, there is no reason to presume that any increase in resolution due to the increase in pixel count should be proportional to the square root of the increase. This would only occur if one had a perfect lens without any aberrations or diffraction effects. There is a law of diminishing returns at work. Each doubling of pixel count produces a progressively smaller increase in the resolution of the recorded image, using the same good lens at its sharpest aperture.

Of course, the sharpness of any lens at F22 and beyond is abysmal. 

[AlanRussell]

"So it is NOT an elephant in the room. I prefer cats anyway as they eat less and are cleaner"

Thanks Peter - I apologise for my , ahem, terminological inexactitude re pachyderms, and I was NOT trying to disparage at all!

I agree that "cat in the room" might have been a better term (they have been adopted by builders of Passiv House Standard buildings as a useful unit of heating BTW)  - I WAS aware of your references within the article and thank you very much for reminding us of the detail and for pointing out the other sources of information.

It's interesting to postulate whether  the number of MP the manufacturers will be able to pack into a 36x24mm sensor size will be limited by diffraction softening / physical dificulties in making sensels increasingly small / lens resolution or marketing issues....

Once again thanks for the article and discussion.

Yours aye a Scotsman who quite likes Scotch (pedantic, moi?).

Alan

[HSway]

I have written scientific articles in the past (not on image sensors, mind you). But the 19th century is even before my time.

That sounds quite believable; it was more a time for our great-great-grandfathers to write. By the way, the Dxomark article looks to be written entirely in modern style  )
Checking it up I see the first issue of my favourite one came out in 1853, founded by Jan Evangelista Purkyně. The second one a while after in 1871.  I haven‘t seen one for years, world is changing, hopefully their level is still what it used to be.

Hynek

[VidJa]

Great article Peter, very clear and precise.

it shows exactly why I never minded upgrading to the latest of great cameras, unlike other members of my local photography club.

I used to have a D50 with only 6 MP. A few years a go I made the mistake to forget switching my camera back from 1.5 MP (which I used to take literally thousands of product photos for a website) to its full resolution. Of course I took a great picture at this 1.5 MP. A bit of scaling and  interpolation gave me a really acceptable 11"x16" print.

Unfortunately I broke the rugged D50 by dropping it on a concrete floor. A good excuse** to finally go for a new cam. The D3200 was a perfect excuse and with the difference between the D7000 I could actually ditch* my old 50mm, and 105 micro for new successors.

Despite all the little thingies you get with more expensive cameras I never felt unease with the manual mode ;-)***

*sell it to someone.......
** wife acceptance factor
*** The difference between 'taking a picture' or 'making a picture' is called Photography.

[Peter van den Hamer]

The D3200 was a perfect excuse and with the difference between the D7000 I could actually ditch* my old 50mm, and 105 micro for new successors.

Yup. At low ISO, the D3200 actually beats both the D3 and the Canon 5D3. At high ISO, the full frame cameras should win (with a suitable lens).

Peter

[Dave Millier]

The point about increased sensor resolution revealing lens aberrations more clearly is important.

The Sigma SD1 is a case in point.  Most commentators are reporting that the DP2m is superior to the SD1 despite the fact they use the same sensor.  The reason is the lens. The DP2m lens is excellent and well matched to the sensor.  Your standard back catalogue of Sigma lenses with a few exceptions really struggle with the SD1m.  The problem is that while most lenses are sharp in the centre, a lot of lenses are not so sharp at the edges.  The SD1 is revealing this amongst the Sigma range.  Extremely sharp and detailed centres coupled with fuzzy edges/corners look worse than the same lenses used with the older generation of cameras where the resolution is insufficient to really expose the difference between centres and edges. Softer all over seems better than sharp in one place and blurry elsewhere.  

David

[bjanes]

Peter,

Thank you for an excellent post. One area that was not covered is the difference between pixel binning in the sensor (hardware binning) and binning post capture by downsampling the image (software binning). At very low levels of illumination where read noise predominates over photon noise, software binning can not make a small pixel sensor perform as well as a larger pixel sensor with the same total sensor area.

Consider the IQ180 sensor with 80 MP. One could use software binning in Photoshop to yield a 20 MP image, combining 4 pixels into one. At low levels of illumination where the SNR is 1:1 and the noise is almost entirely read noise, one could use software 4:1 software binning to obtain a SNR of 2:1 where 4 pixels (each with a read noise component) are combined into one. On chip binning using the Sensor+ technology would yield a SNR of 4:1 since only one read noise component would be present (see this Phase one tutorial--click on the Sensor+ page).

Regards,

Bill

[Ernst Dinkla]

Peter,

A lot of work to decide what your next camera will be ;-)   Good reading.

For the Color Depth at Low Iso measurements the label Portrait did not cover the content well, we all agree. But I considered it a good measurement for art reproduction jobs with ample (full spectrum) light allowing optimal settings for all components including a low Iso setting. The jobs where you would expect color calibration, profiling from camera to print and larger print sizes, the last revealing chroma noise when available which could influence a 1:1 reproduction of colors (with the original still there to check against). The dynamic range of a sensor will not be challenged when reproducing reflective art originals so is of less importance. Maybe "Portrait" could be replaced with "Still Life" or "Reproduction".

The Print "filter" on the data may not reflect daily practice for camera users that work with large format printers. I have not checked that again but I got the impression some years ago that the print filter could be too conservative in time. The filter brings sensor qualities much closer to one another than what large prints can reveal. Recent development in extrapolation and sharpening routines in the printer driver software etc should be counted too. The results more progressive than what your plots of the sensor developments show. Going down in print size hits on resolution limits of inkjet papers and printers, not to mention the human eye. Anti-aliased detail in the A4-A3 print is more likely what most users observe than actual camera detail due to the (quality) pixels increase that is not 1:1 transferred to paper.

Medium format backs did a better job in art reproduction in the past, I think related to what I mentioned above. The Sony FF sensors score on the same level now, prime lenses are improved to meet that challenge. The MF backs relied on Schneider Digitars etc for a reason. It could be that the Sony APS sensors and A3+ printers (or Retina displays) set the ceiling for further developments but at some places photographers are harder to satisfy.

I expected a reference to the Zeiss optics of the ASML wafer steppers somewhere but you skipped that :-)


--
Met vriendelijke groet, Ernst

http://www.pigment-print.com/spectralplots/spectrumviz_1.htm
December 2012, 500+ inkjet media white spectral plots.

[EricV]

Very nice article.

One very small quibble:

DxOMark's Dynamic Range plot for these cameras shows that their Dynamic Range drops by almost one 1 EV each time the ISO is doubled. This resembles an ideal amplifier that amplifies the sensor’s signal and noise without adding noise of its own. That is impressive.

When DR drops by 1EV for every doubling of ISO, that is hardly impressive -- it is exactly what you would get by not changing ISO and simply underexposing more and more, then multiplying the raw image by factors of two as needed.  What is impressive about the amplifiers in cameras with very high DR is their extremely low intrinsic noise, compared to the sensor full well capacity.  The ability to effectively adjust ISO, in a way which improves image quality compared to underexposing, would show up as a DR curve which drops less than 1 EV per ISO doubling.

One suggestion/request:
Would it be possible to plot DxO score versus total sensor charge capacity (pixel count times pixel full well capacity)?  That might show a pretty high correlation.