Luminous Landscape Forum

Site & Board Matters => About This Site => Topic started by: Rory on December 15, 2012, 12:01:14 PM

Title: DxO marks
Post by: Rory on December 15, 2012, 12:01:14 PM
Great article.  A few minor things to correct:

Title: Re: DxO marks
Post by: svein on December 15, 2012, 02:43:09 PM
Great article.  A few minor things to correct:
Agree really interesting article, appreciate the time spend researching and writing it.

A couple of typos on sensor sizes:
X10 use a (so called) 2/3" sensor which is slightly bigger than the one in S110, and quite a bit bigger than the one in Pentax Q.
Title: Re: DxO marks
Post by: Peter van den Hamer on December 15, 2012, 03:30:52 PM
Thanks, Rory and Svein, for spotting these errors. I will see if I change the blue comma - it is indeed different on screen (on this site) than on paper.
Title: Re: DxO marks
Post by: DaveCurtis on December 15, 2012, 06:36:17 PM
Brilliant article.

Amazing to see how the sensors have evolved over the years.

I am now waiting for Canon to up the ante on DR front :)
Title: Re: DxO marks
Post by: Ray on December 16, 2012, 03:41:38 AM
Excellent article, but a bit daunting for someone who may be just trying to get beyond the overall scores which are weighted and sometimes a bit misleading.

I would take issue with the following comment;
Quote
Another way to look at the relevance of resolution: nowadays, unless you use expensive lenses on a relatively cheap camera, cameras tend to have enough resolution to handle what the lenses can project onto the sensor. And for most uses, 12-18 MPixels is more than enough anyway. So a properly designed noise benchmark can be used to predict image quality as long as you keep an eye on whether you have enough resolution for your needs.

I'm not happy with this concept that a camera can have enough resolution to handle what the lens can project onto the sensor.

The recorded resolution is always a product of both lens resolution and sensor resolution. Increase the resolution of either one, and the recorded resolution will also increase.

In effect, when one upgrades to a sensor with more megapixels, one automatically upgrades all one's lenses as a free bonus.

To take an example from DXOMark's lens database. The relatively cheap, standard Canon 50mm F1.4 on the 12.7mp 5D has a resolution of 55 lp/mm, according to DXO's methodology.

That same lens on a Canon 5D3 with less than double the the pixel count of the 5D, has a resolution of 63 lp/mm. If the 5D3 were to have the resolution of a D800, ie. another 50% increase in pixel count, that same lens would have a recorded image of perhaps around 66 lp/mm.

The point I would make is that probably no amount of money could buy a 50mm lens that could deliver 66 lp/mm or even 63 lp/mm on a 5D. Such a lens has not been produced.

The presence of more pixels on the sensor also has the effect of upgrading one's lenses in terms of the range of effective focal lengths in relation to the standards of a lower pixel-count sensor.

For example, supposing one were to consider that the 16mp of the Nikon D7000 is sufficient for one's purposes and that the 36.3mp of the D800 is far more than one would ever use because one never makes prints larger than A3+. How short-sighted would one be?

What the upgrade from a D7000 to a D800 does in effect, is not only upgrade the resolution of one's lenses, but also effectively converts all one's prime lenses into telephoto lenses in relation to those standards of the lower-pixel-count D7000, and extends the range of all one's zoom lenses in relation to the standards of the D7000. Such an upgrade is almost priceless.

For example, a 50mm, high quality prime on a D800 becomes in effect an extremely high quality 50-75mm DX zoom by D7000 standards.

That great lens, the Nikkor 14-24/2.8, on the D7000 becomes effectively a 21-36mm DX zoom. However, on the D800, by the DX standards of the 16mp D7000, it becomes a high quality 14-36/2.8 DX zoom. No such lens exists, and if it did, it would be hugely expensive at the same quality.

Title: Re: DxO marks
Post by: HSway on December 16, 2012, 06:07:45 AM
A nice Saturday’s read yesterday for me. It reminded me of a smell of academic periodicals' pages. I mean the physical smell, long before the internet. Actually, I realized, seldom I read on the web articles similar to these Saturday’s delights these 19th cent. established periodicals were bringing me.

The dxomark is really great and we are lucky to have its systematic data in this useful format at disposal. Thinking a moment about it, I, in fact, can’t imagine the digital photography community being without this well-arranged evaluative system. Many good points in the article, the dxomark is certainly not a complete answer to any question but is good to have it and I imagine the confusion, major errors and bias on the web would have much easier life without it.

Thanks for the article,

Hynek
Title: Re: DxO marks
Post by: AlanRussell on December 16, 2012, 08:37:29 AM
Peter- Many many thanks for a really superb article.  Best thing I've read on the best Photo Site in the Universe.  (So good that I am replying after many years of just lurking on LuLa.....).

The one thing my fuzzy little brain has trouble with is the rather huge Elephant in the Room with all of this (which, your article  correctly didn't cover- being a discussion of DxO marks specifically) -  i.e. diffraction softening - surely, in everyday practical terms, the biggest limitation to image quality, along with resolving power of the lenses.

Cramming more and smaller pitch sensels onto chips means that we will be limited to a very restricted range of apertures with these new high MP count dslrs such as the D800.

As a vague rule of thumb I've always worked on the 1/Pixel Pitch in Microns = smallest aperture worth using e.g. 8microns gives f8 (depends on how fussy you are with circles of confusion and all that of course).

The whole issue of how many MP we can cram on to a 35mm sensor, once discussed regularly seems to have been sidelined as SNR ratios have improved but surely we are close to some sort of nyquist limit already - as Scotty from Star Trek said "ye canny change the laws o physics, Jim".

I speak as a product photographer who needs both depth of field and the highest resolutions possible and who, being Scottish, would much prefer to spending £3K on a DSLR to £30K on a new back to achieve both decent Dof and Res.

I know very little of these laws o physics but I'd really appreciate your thoughts..

Yours aye,

Alan
Title: Re: DxO marks
Post by: ErikKaffehr on December 16, 2012, 08:53:07 AM
Hi,

You could check a few of my articles:

http://echophoto.dnsalias.net/ekr/index.php/photoarticles/68-effects-of-diffraction

http://echophoto.dnsalias.net/ekr/index.php/photoarticles/49-dof-in-digital-pictures?start=1

On the other hand, diffraction is very beneficial to sharpening. So optimal sharpening can recover a lot of contrast lost to diffraction.

Check this article by Tim Parkin: http://www.onlandscape.co.uk/2012/07/diffraction-limited/ he finds, among other things, that a D800 image at f/22 outresolves his Sony Alpha 900 at f/8, when both images are correctly sharpened.

By the way, going MFD will not help with diffraction. On MFD you need to stop down more DoF so you loose more to diffraction, it's a zero sum game. The best option you have is to use Scheimpflug or merge several images into one.

Both methods are covered here: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/29-handling-the-dof-trap

Best regards
Erik



Peter- Many many thanks for a really superb article.  Best thing I've read on the best Photo Site in the Universe.  (So good that I am replying after many years of just lurking on LuLa.....).

The one thing my fuzzy little brain has trouble with is the rather huge Elephant in the Room with all of this (which, your article  correctly didn't cover- being a discussion of DxO marks specifically) -  i.e. diffraction softening - surely, in everyday practical terms, the biggest limitation to image quality, along with resolving power of the lenses.

Cramming more and smaller pitch sensels onto chips means that we will be limited to a very restricted range of apertures with these new high MP count dslrs such as the D800.

As a vague rule of thumb I've always worked on the 1/Pixel Pitch in Microns = smallest aperture worth using e.g. 8microns gives f8 (depends on how fussy you are with circles of confusion and all that of course).

The whole issue of how many MP we can cram on to a 35mm sensor, once discussed regularly seems to have been sidelined as SNR ratios have improved but surely we are close to some sort of nyquist limit already - as Scotty from Star Trek said "ye canny change the laws o physics, Jim".

I speak as a product photographer who needs both depth of field and the highest resolutions possible and who, being Scottish, would much prefer to spending £3K on a DSLR to £30K on a new back to achieve both decent Dof and Res.

I know very little of these laws o physics but I'd really appreciate your thoughts..

Yours aye,

Alan
Title: Re: DxO marks
Post by: Peter van den Hamer on December 16, 2012, 09:06:45 AM
The recorded resolution is always a product of both lens resolution and sensor resolution. Increase the resolution of either one, and the recorded resolution will also increase.

In effect, when one upgrades to a sensor with more megapixels, one automatically upgrades all one's lenses as a free bonus.

Ray,

"Product of" would obviously be wrong, but I guess you meant "function of".
But that claim is like saying "for a given lens, increasing sensor resolution will not reduce overall image sharpness" - which is kind of hard to disagree with  :)

So, given that you recommend upgrading to a high res sensor to utilize the full capabilities of lenses (which I kind of did myself: 6 MPix APS-C 10D to 21 MPix FF 5D2), let's check how much this helps using your own example: a Canon 50mm/1.4 lens, used on full-frame cameras, and measuring the max/max/max resolution using DxOMark data.

This gives 55 line_pairs/mm on a 12.7 MPix Canon 5D. Assuming the lens could keep up with the increased resolution of the 5D2 (there is no DxO lens data for 5D3 yet), it would give 71 lp/mm (= 55 lp/mm * sqrt(21.1/12.7)).

Firstly, the highest resolution for any lens on any camera measured by DxOMark so far is only 67 lp/mm. So we can't expect 71 lp/mm for a lowly Canon 50/1.4 (successor is expected). Instead, the measured data for the 50mm/1.4 lens on the 5D2 is 63 lp/mm. This is a 15% increase compared to the three years older 5D design.

This confirms your (somewhat unrefutable) claim that higher resolution sensors contribute to higher resolution images. But it also shows that lenses don't really keep up with sensor resolution increases:

What the upgrade from a D7000 to a D800 does in effect, is not only upgrade the resolution of one's lenses, but also effectively converts all one's prime lenses into telephoto lenses in relation to those standards of the lower-pixel-count D7000, and extends the range of all one's zoom lenses in relation to the standards of the D7000. Such an upgrade is almost priceless.

The story that if you migrate from APS-C to full-frame, that you should increase resolution if you plan to crop back to APS-C makes a lot of sense, assuming that you owned an FX lens on a DX camera. Incidentally, the 36.6 MPixel D800(E) and the 16 MPixel D7000 both have a pixel pitch of 4.8-ish micrometer. The numbers are so alike, it may not be a coincidence (some product manager said "scale D7000 sensor to full-frame!"). The blue scaling line in Figure 6 shows that the D800 even performs pretty comparably to 2 (actually 2.25) D7000 sensors tiled side-by-side.

Peter
Title: Re: DxO marks
Post by: Peter van den Hamer on December 16, 2012, 09:14:35 AM
Actually, I realized, seldom I read on the web articles similar to these Saturday’s delights these 19th cent. established periodicals were bringing me.

I have written scientific articles in the past (not on image sensors, mind you). But the 19th century is even before my time. :P
Title: Re: DxO marks
Post by: Peter van den Hamer on December 16, 2012, 09:47:39 AM
huge Elephant in the Room with all of this (which, your article  correctly didn't cover- being a discussion of DxO marks specifically) -  i.e. diffraction softening - surely, in everyday practical terms, the biggest limitation to image quality, along with resolving power of the lenses.

The topic is indeed a bit out of scope but has actually been taken into account and is briefly mentioned twice in the article. So it is NOT an elephant in the room. I prefer cats anyway as they eat less and are cleaner.

1) See Erik Kaffehr's answer.

2) http://www.cambridgeincolour.com/tutorials/diffraction-photography.htm (http://www.cambridgeincolour.com/tutorials/diffraction-photography.htm) has a tutorial and an online calculator.

3) Note also Falk Lumo's site (quoted in the article). His equivalence criterion (summarized in text and a large table in my article) actually tells you what MF camera setup can be replaced by which full-frame configuration with the same depth of field and the same diffraction. It should help thrifty Scotchmen determine whether they can get away with a D800E instead of a MF camera. If you have a precise enough question ("am I right that X should match Y w.r.t. Z?"), chances are that Falk will help out. http://www.falklumo.com/lumolabs/index.html (http://www.falklumo.com/lumolabs/index.html)
Title: Re: DxO marks
Post by: Ray on December 16, 2012, 10:27:57 AM
Ray,

"Product of" would obviously be wrong, but I guess you meant "function of".

Peter,

Either product or function would be appropriate. No sensor, no recorded image. No lens, no recorded image. Let's not quibble.

Quote
But that claim is like saying "for a given lens, increasing sensor resolution will not reduce overall image sharpness" - which is kind of hard to disagree with

I'm saying more than that. However bad the lens may be, the fact that increasing sensor resolution will not reduce overall image sharpness, is a given. No need to raise the point.

What I'm saying is that 16 or 18mp, whether full frame 35mm or cropped format, is not sufficient to capture the full resolution projected by a good lens at its sharpest aperture and other apertures close to its sharpest aperture, such as F8 or F4, if F5.6 is the sharpest aperture.

In fact, my own tests comparing the 10mp Canon 40D (25.6mp full frame equivalent) with the 15mp Canon 50D (38mp full frame equivalent) show that even at F16 the 38mp image (equivalent) reveals more resolution than the 25.6mp image. At F8 and F5.6 there's no contest. Nikon can safely raise the pixel count of the D800 in future models to, say, 60mp without complaints that there is no resolution advantage.


Quote
This gives 55 line_pairs/mm on a 12.7 MPix Canon 5D. Assuming the lens could keep up with the increased resolution of the 5D2 (there is no DxO lens data for 5D3 yet), it would give 71 lp/mm (= 55 lp/mm * sqrt(21.1/12.7)).

You are correct that the figure I quoted of 63 lp/mm for the 5D3 is in fact for the 5D2. Typo error. However, the difference of 1mp between the 5D2 and 5D3 sensor is of no practical significance. One would expect the 5D3 to produce the same resolution of 63 lp/mm with that 50mm lens.

However, there is no reason to presume that any increase in resolution due to the increase in pixel count should be proportional to the square root of the increase. This would only occur if one had a perfect lens without any aberrations or diffraction effects. There is a law of diminishing returns at work. Each doubling of pixel count produces a progressively smaller increase in the resolution of the recorded image, using the same good lens at its sharpest aperture.

Of course, the sharpness of any lens at F22 and beyond is abysmal.  ;)

Title: Re: DxO marks
Post by: AlanRussell on December 16, 2012, 10:30:58 AM
"So it is NOT an elephant in the room. I prefer cats anyway as they eat less and are cleaner"

Thanks Peter - I apologise for my , ahem, terminological inexactitude re pachyderms, and I was NOT trying to disparage at all!

I agree that "cat in the room" might have been a better term (they have been adopted by builders of Passiv House Standard buildings as a useful unit of heating BTW)  - I WAS aware of your references within the article and thank you very much for reminding us of the detail and for pointing out the other sources of information.

It's interesting to postulate whether  the number of MP the manufacturers will be able to pack into a 36x24mm sensor size will be limited by diffraction softening / physical dificulties in making sensels increasingly small / lens resolution or marketing issues....

Once again thanks for the article and discussion.

Yours aye a Scotsman who quite likes Scotch (pedantic, moi?).

Alan
Title: Re: DxO marks
Post by: HSway on December 16, 2012, 10:34:59 AM
I have written scientific articles in the past (not on image sensors, mind you). But the 19th century is even before my time. :P


That sounds quite believable; it was more a time for our great-great-grandfathers to write. By the way, the Dxomark article looks to be written entirely in modern style  :)))
Checking it up I see the first issue of my favourite one came out in 1853, founded by Jan Evangelista Purkyně. The second one a while after in 1871.  I haven‘t seen one for years, world is changing, hopefully their level is still what it used to be.

Hynek

Title: Re: DxO marks
Post by: VidJa on December 16, 2012, 05:28:38 PM
Great article Peter, very clear and precise.

it shows exactly why I never minded upgrading to the latest of great cameras, unlike other members of my local photography club.

I used to have a D50 with only 6 MP. A few years a go I made the mistake to forget switching my camera back from 1.5 MP (which I used to take literally thousands of product photos for a website) to its full resolution. Of course I took a great picture at this 1.5 MP. A bit of scaling and  interpolation gave me a really acceptable 11"x16" print.

Unfortunately I broke the rugged D50 by dropping it on a concrete floor. A good excuse** to finally go for a new cam. The D3200 was a perfect excuse and with the difference between the D7000 I could actually ditch* my old 50mm, and 105 micro for new successors.

Despite all the little thingies you get with more expensive cameras I never felt unease with the manual mode ;-)***

*sell it to someone.......
** wife acceptance factor
*** The difference between 'taking a picture' or 'making a picture' is called Photography.
Title: Re: DxO marks
Post by: Peter van den Hamer on December 16, 2012, 06:20:09 PM
The D3200 was a perfect excuse and with the difference between the D7000 I could actually ditch* my old 50mm, and 105 micro for new successors.

Yup. At low ISO, the D3200 actually beats both the D3 and the Canon 5D3. At high ISO, the full frame cameras should win (with a suitable lens).

Peter
Title: Re: DxO marks
Post by: Dave Millier on December 17, 2012, 04:30:11 AM
The point about increased sensor resolution revealing lens aberrations more clearly is important.

The Sigma SD1 is a case in point.  Most commentators are reporting that the DP2m is superior to the SD1 despite the fact they use the same sensor.  The reason is the lens. The DP2m lens is excellent and well matched to the sensor.  Your standard back catalogue of Sigma lenses with a few exceptions really struggle with the SD1m.  The problem is that while most lenses are sharp in the centre, a lot of lenses are not so sharp at the edges.  The SD1 is revealing this amongst the Sigma range.  Extremely sharp and detailed centres coupled with fuzzy edges/corners look worse than the same lenses used with the older generation of cameras where the resolution is insufficient to really expose the difference between centres and edges. Softer all over seems better than sharp in one place and blurry elsewhere.  

David
Title: Re: DxO marks
Post by: bjanes on December 17, 2012, 09:00:33 AM
Peter,

Thank you for an excellent post. One area that was not covered is the difference between pixel binning in the sensor (hardware binning) and binning post capture by downsampling the image (software binning). At very low levels of illumination where read noise predominates over photon noise, software binning can not make a small pixel sensor perform as well as a larger pixel sensor with the same total sensor area.

Consider the IQ180 sensor with 80 MP. One could use software binning in Photoshop to yield a 20 MP image, combining 4 pixels into one. At low levels of illumination where the SNR is 1:1 and the noise is almost entirely read noise, one could use software 4:1 software binning to obtain a SNR of 2:1 where 4 pixels (each with a read noise component) are combined into one. On chip binning using the Sensor+ technology would yield a SNR of 4:1 since only one read noise component would be present (see this Phase one tutorial (http://www.phaseone.com/en/Camera-Systems/IQ-Series/IQ-Tutorials.aspx)--click on the Sensor+ page).

Regards,

Bill
Title: Re: DxO marks
Post by: Ernst Dinkla on December 17, 2012, 11:58:21 AM
Peter,

A lot of work to decide what your next camera will be ;-)   Good reading.

For the Color Depth at Low Iso measurements the label Portrait did not cover the content well, we all agree. But I considered it a good measurement for art reproduction jobs with ample (full spectrum) light allowing optimal settings for all components including a low Iso setting. The jobs where you would expect color calibration, profiling from camera to print and larger print sizes, the last revealing chroma noise when available which could influence a 1:1 reproduction of colors (with the original still there to check against). The dynamic range of a sensor will not be challenged when reproducing reflective art originals so is of less importance. Maybe "Portrait" could be replaced with "Still Life" or "Reproduction".

The Print "filter" on the data may not reflect daily practice for camera users that work with large format printers. I have not checked that again but I got the impression some years ago that the print filter could be too conservative in time. The filter brings sensor qualities much closer to one another than what large prints can reveal. Recent development in extrapolation and sharpening routines in the printer driver software etc should be counted too. The results more progressive than what your plots of the sensor developments show. Going down in print size hits on resolution limits of inkjet papers and printers, not to mention the human eye. Anti-aliased detail in the A4-A3 print is more likely what most users observe than actual camera detail due to the (quality) pixels increase that is not 1:1 transferred to paper.

Medium format backs did a better job in art reproduction in the past, I think related to what I mentioned above. The Sony FF sensors score on the same level now, prime lenses are improved to meet that challenge. The MF backs relied on Schneider Digitars etc for a reason. It could be that the Sony APS sensors and A3+ printers (or Retina displays) set the ceiling for further developments but at some places photographers are harder to satisfy.

I expected a reference to the Zeiss optics of the ASML wafer steppers somewhere but you skipped that :-)


--
Met vriendelijke groet, Ernst

http://www.pigment-print.com/spectralplots/spectrumviz_1.htm
December 2012, 500+ inkjet media white spectral plots.
Title: Re: DxO marks
Post by: EricV on December 17, 2012, 12:56:50 PM
Very nice article.

One very small quibble:
Quote
DxOMark's Dynamic Range plot for these cameras shows that their Dynamic Range drops by almost one 1 EV each time the ISO is doubled. This resembles an ideal amplifier that amplifies the sensor’s signal and noise without adding noise of its own. That is impressive.
When DR drops by 1EV for every doubling of ISO, that is hardly impressive -- it is exactly what you would get by not changing ISO and simply underexposing more and more, then multiplying the raw image by factors of two as needed.  What is impressive about the amplifiers in cameras with very high DR is their extremely low intrinsic noise, compared to the sensor full well capacity.  The ability to effectively adjust ISO, in a way which improves image quality compared to underexposing, would show up as a DR curve which drops less than 1 EV per ISO doubling.

One suggestion/request:
Would it be possible to plot DxO score versus total sensor charge capacity (pixel count times pixel full well capacity)?  That might show a pretty high correlation.
Title: Re: DxO marks
Post by: ErikKaffehr on December 17, 2012, 02:50:41 PM
Hi,

The "print" mode essentially does something pretty similar.

Best regards
Erik


Would it be possible to plot DxO score versus total sensor area (pixel count x pixel size)?  That might show a pretty high correlation.
Title: Re: DxO marks
Post by: EricV on December 17, 2012, 03:25:46 PM
Sorry, I meant sensor charge capacity, not area.  Charge capacity includes both pixel count and pixel full well depth (which is or course related to pixel area, but not quite the same), in the way which matters for noise in a final print.
Title: Re: DxO marks
Post by: Peter van den Hamer on December 17, 2012, 04:07:25 PM
Thank you for an excellent post. One area that was not covered is the difference between pixel binning in the sensor (hardware binning) and binning post capture by downsampling the image (software binning). At very low levels of illumination where read noise predominates over photon noise, software binning can not make a small pixel sensor perform as well as a larger pixel sensor with the same total sensor area.

Thanks. Actually there is mention (also in the older article) of this change binning technique. See endnotes 44 and 45 and associated body text:

Quote
"Note that although this scaling story holds for photon shot noise and dark current shot noise, other noise sources don’t necessarily scale the same way. In particular, some very high-end CCDs can use a special analog trick (“charge binning”) to sum the pixels, thus reducing the amount of times that a readout is required. This would reduce temporal noise by a further sqrt(N) where N is the number of pixels that are binned. Apart from the fact that only exotic sensors have this capability (Phase One’s Pixel+ technology), DxOMark’s data suggest that this extra improvement doesn’t play a significant role."
Title: Re: DxO marks
Post by: Peter van den Hamer on December 17, 2012, 04:47:14 PM
the label Portrait did not cover the content well, we all agree. But I considered it a good measurement for art reproduction jobs with ample (full spectrum) light allowing optimal settings for all components including a low Iso setting. The jobs where you would expect color calibration, profiling from camera to print and larger print sizes, the last revealing chroma noise when available which could influence a 1:1 reproduction of colors (with the original still there to check against). The dynamic range of a sensor will not be challenged when reproducing reflective art originals so is of less importance. Maybe "Portrait" could be replaced with "Still Life" or "Reproduction".

I agree that the Color Sensitivity benchmark should be very suitable for art reproduction work. But that is a niche for most of us. You might call it "studio". In the text I questioned whether people can actually see chroma noise under these conditions. I have no proof other than that people can hardly see the luminance noise at medium gray at 100 ISO, and that chroma noise at the pixel level should be even harder to see. Anybody feel like generating some test images (of gray patches) using MatLab to simulate what the latest cameras can achieve?

The Print "filter" on the data may not reflect daily practice for camera users that work with large format printers. I have not checked that again but I got the impression some years ago that the print filter could be too conservative in time. The filter brings sensor qualities much closer to one another than what large prints can reveal.

The print-mode data is just a way to normalize pixel-level noise to a common resolution. I agree that if you print large enough, you can do some serious pixel peeping - although that is easier and cheaper to do by clicking on "100%" screen viewing. I guess DxOMark considered pixel peeping more common on screens - even by non-photographers. Only some photographers (the nerdier ones?) inspect a print with their nose touching the print ;)

You probably already know this, but (to be on the safe side) this particular DxOMark benchmark data does not cover the contribution of sensor resolution to print quality. So if you pixel peep, this benchmark tells you what you will see when you are admiring the creamy richness of noise-free bokeh gradients :P
Title: Re: DxO marks
Post by: Peter van den Hamer on December 17, 2012, 05:23:05 PM
Would it be possible to plot DxO score versus total sensor charge capacity (pixel count times pixel full well capacity)?  That might show a pretty high correlation.

The total charge capacity should be a good indicator of signal to noise ratio in highlights (a somewhat academic number as we wouldn't really see that without extreme post-processing). The dynamic range and low-light ISO benchmarks also depend on noise in the lowlights/shadows.
Title: Re: DxO marks
Post by: Ray on December 17, 2012, 05:53:55 PM
I agree that the Color Sensitivity benchmark should be very suitable for art reproduction work. But that is a niche for most of us. You might call it "studio". In the text I questioned whether people can actually see chroma noise under these conditions. I have no proof other than that people can hardly see the luminance noise at medium gray at 100 ISO, and that chroma noise at the pixel level should be even harder to see. Anybody feel like generating some test images (of gray patches) using MatLab to simulate what the latest cameras can achieve?

The print-mode data is just a way to normalize pixel-level noise to a common resolution. I agree that if you print large enough, you can do some serious pixel peeping - although that is easier and cheaper to do by clicking on "100%" screen viewing. I guess DxOMark considered pixel peeping more common on screens - even by non-photographers. Only some photographers (the nerdier ones?) inspect a print with their nose touching the print ;)

You probably already know this, but (to be on the safe side) this particular DxOMark benchmark data does not cover the contribution of sensor resolution to print quality. So if you pixel peep, this benchmark tells you what you will see when you are admiring the creamy richness of noise-free bokeh gradients :P

Peter,
There are a few issues here which could do with more clarification.

Graphs are designed to highlight differences. That's their purpose. If one camera has just 0.1EV difference in DR compared with another camera, a graph is designed to clearly show it. That's fine. I have no objection.

However, it is important also to know the practical significance of such differences on the print or on the monitor at specific degrees of enlargement.

For example, the normalised print size of 8"x12" that DXO use is rather small, for good reasons no doubt, so no interpolation is required for the smallest resolution cameras that have been tested, such as the Canon 10D.

Now, we all know that different RAW converters produce slightly different results. One converter may apply greater default noise reduction which is beyond the user control, and another may produce slightly sharper default results but with greater noise.

Likewise, one particular interpolation algorithm may produce more detailed images than another.

The question that I have is this. Just how reliable are these 'normalised' results at 8"x10” when images are interpolated for the purpose of making much larger prints, using the same RAW converter?

I assume that differences due to the different handling of different brands of RAW images by the same converter will exist, but they may be negligible in practice. Would you agree?

Another issue which I think requires more clarifiction is the paractical significance of those value differences shown on the graphs, whether they be dB, bits or EV.

I gather from DXO’s articles that a difference of less than 1 bit in Color Sensitivity, and a difference of less than 0.5EV in DR may not be noticeable. We need to elaborate on such issues. How does print size affect such assessments as to what’s noticeable and/or significant?
Title: Re: DxO marks
Post by: Peter van den Hamer on December 17, 2012, 06:05:16 PM
One very small quibble:When DR drops by 1EV for every doubling of ISO, that is hardly impressive -- it is exactly what you would get by not changing ISO and simply underexposing more and more, then multiplying the raw image by factors of two as needed.

Changing the ISO is just a term for underexposing and compensating in the camera (via analog and sometimes digital scaling). Your "exactly what you would get" only applies to an ideal case. Approximating the ideal case is IMO pretty impressive.

The ability to effectively adjust ISO, in a way which improves image quality compared to underexposing, would show up as a DR curve which drops less than 1 EV per ISO doubling.

Sounds like you are asking for something physically impossible at high ISO? See Figure 7 for an example: the 1Dx at 51200 ISO does break the law, but cheats a bit (according to DxOMark) by starting to apply extra "smoothing" aka noise filtering at 51200 ISO. At low ISO, get less than a 1 EV increase per ISO halving, but this is because you are saturating against the maximum signal to noise ratio which the unamplified ("base iso") sensor can handle. So at low ISO - although you do drop less than 1 EV per ISO doubling - this leads to weaker low ISO DR.

So I think my original statement is quite accurate, although it might have alternatively phrased it as "1 EV DR increase per ISO halving" rather than "1 EV DR decrease per ISO doubling".
Title: Re: DxO marks
Post by: Fine_Art on December 17, 2012, 09:03:24 PM
Ray,

"Product of" would obviously be wrong, but I guess you meant "function of".
But that claim is like saying "for a given lens, increasing sensor resolution will not reduce overall image sharpness" - which is kind of hard to disagree with  :)

So, given that you recommend upgrading to a high res sensor to utilize the full capabilities of lenses (which I kind of did myself: 6 MPix APS-C 10D to 21 MPix FF 5D2), let's check how much this helps using your own example: a Canon 50mm/1.4 lens, used on full-frame cameras, and measuring the max/max/max resolution using DxOMark data.

This gives 55 line_pairs/mm on a 12.7 MPix Canon 5D. Assuming the lens could keep up with the increased resolution of the 5D2 (there is no DxO lens data for 5D3 yet), it would give 71 lp/mm (= 55 lp/mm * sqrt(21.1/12.7)).

Firstly, the highest resolution for any lens on any camera measured by DxOMark so far is only 67 lp/mm. So we can't expect 71 lp/mm for a lowly Canon 50/1.4 (successor is expected). Instead, the measured data for the 50mm/1.4 lens on the 5D2 is 63 lp/mm. This is a 15% increase compared to the three years older 5D design.

This confirms your (somewhat unrefutable) claim that higher resolution sensors contribute to higher resolution images. But it also shows that lenses don't really keep up with sensor resolution increases:
  • decreasing the pixel pitch by 30% (=increasing the MPixel count by 75%) only results in a 15% lineair resolution increase.
  • some of that 15% increase is probably due to unrelated camera improvements. Compare the pricey 85mm/1.4D on the Nikon D300s to the D700: both have a 12 Mpixel sensor resolution, but 8% lp/mm overall resolution difference (AA filter? crosstalk? processing?). So a modern 12 MPixel full-frame camera would presumably give better results than the old Canon 5D design. So the contribution of sensor resolution to the 15% measured overal resolution improvement might be below 10%. We could test this as soon as DxOMark tests the lens on the 5D3 (newer, roughly same resolution as 5D2).
  • the 5D had huge pixels for its time. The 8 MPixel 350D was already out. If you scale the full frame 5D down to 1.6x APS-C, you get a mere 5 MPixel camera. So the 5D had unusually low resolution for its time. A simplistic calculation for the Canon 5D says that it cannot outresolve 59 lp/mm (59 pairs/mm * 2 lines/pair * 24mm * 59 * 2 * 36mm = 12 MPixels).
  • the single figure resolution numbers provided by DxOMark are at each lens' best aperture, at the best zoom setting, and in the middle of the image. This max/max/max measurement obviously flatters the true capabilities of the lens.
  • we are doing the excercise at full-frame. Many people have smaller sensors. An APS-C sensor has 1.5 or 1.6 times smaller pixels than a full-frame sensor with the same resolution. This means that "increasing resolution to get the most out of your lenses" will probably give even less benefit for APS-C or smaller cameras. Arguably you need more than a 12 MPixel medium format camera, but these are not for sale, and users are less likely to crop there.

The story that if you migrate from APS-C to full-frame, that you should increase resolution if you plan to crop back to APS-C makes a lot of sense, assuming that you owned an FX lens on a DX camera. Incidentally, the 36.6 MPixel D800(E) and the 16 MPixel D7000 both have a pixel pitch of 4.8-ish micrometer. The numbers are so alike, it may not be a coincidence (some product manager said "scale D7000 sensor to full-frame!"). The blue scaling line in Figure 6 shows that the D800 even performs pretty comparably to 2 (actually 2.25) D7000 sensors tiled side-by-side.

Peter

Great article Peter.

I have to disagree with Ray. You can't just increase sensor resolution forever, expecting the lenses to handle it. The camera system will function as well as the weakest part. It so happens that lenses are still way ahead of FF sensors. All you have to do is look at the pixel size of the cheap compacts. Of course the lenses on our SLR systems are as good or better than the compacts. Ray is correct now, yes. The terminology assumed lenses can handle anything which is not right.

A Bayer system would hit the wall around 2x the red wavelength 720nm or .7 microns = 1.4 microns. Most DSLR sensors are 5 to 6 microns for FF or 4 to 5 microns for APS-C. There is still free resolution for now.
Title: Re: DxO marks
Post by: dreed on December 17, 2012, 11:44:44 PM
For further reading on the topic of sensor noise ...

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/
Title: Re: DxO marks
Post by: Ray on December 18, 2012, 12:31:26 AM
Great article Peter.

I have to disagree with Ray. You can't just increase sensor resolution forever, expecting the lenses to handle it. The camera system will function as well as the weakest part. It so happens that lenses are still way ahead of FF sensors. All you have to do is look at the pixel size of the cheap compacts. Of course the lenses on our SLR systems are as good or better than the compacts. Ray is correct now, yes. The terminology assumed lenses can handle anything which is not right.

A Bayer system would hit the wall around 2x the red wavelength 720nm or .7 microns = 1.4 microns. Most DSLR sensors are 5 to 6 microns for FF or 4 to 5 microns for APS-C. There is still free resolution for now.

I think you've got things back to front here when you write; "You can't just increase sensor resolution forever, expecting the lenses to handle it."

Lenses do not handle sensor resolution. It's the other way round. A lens will always project an image of a certain quality and resolution, depending on the aperture used, regardless of the quality of the sensor.

However, something is always lost as the sensor records the projected image from the lens, even when it's a low quality lens and high pixel-count sensor.

If the lens used is a high quality lens, then of course more is lost in the recording process than would be the case using a low quality lens with the same sensor.

However, setting aside semantics, you are correct that one cannot expect unlimited increases in sensor resolution to continue to provide increased resolution from the same lens. As I mentioned in my reply to Peter, there's a situation of diminishing returns that applies so that at some point any increase in the resolution of the recorded image due to increases in sensor resolution will be so small that it will be unnoticeable in practice.

You can test this for yourself if you've not discarded all your earlier, low resolution cameras, and you don't even have to acquire a low-resolution lens for the purpose. Any 35mm format lens can be considered low resolution from F11 to F32, if it stops down that far.
Title: Re: DxO marks
Post by: EricV on December 18, 2012, 12:39:37 AM
Changing the ISO is just a term for underexposing and compensating in the camera (via analog and sometimes digital scaling). Your "exactly what you would get" only applies to an ideal case. Approximating the ideal case is IMO pretty impressive.
Digital scaling, either in camera or in post-processing (multiplying raw image integer values on a computer long after the exposure), is exact and requires no impressive electronics.  The impressive electronics for ISO control is analog scaling which multiplies signal and shot noise more than readout noise, improving the slope of the curve at low ISO (where readout noise is significant).
Title: Re: DxO marks
Post by: ErikKaffehr on December 18, 2012, 12:42:59 AM
Hi,

It's different. Sensors with on chip converters seem to have very different characteristics compared with sensors with off chip converters.

Best regards
Erik

Digital scaling, either in camera or in post-processing (multiplying raw image integer values on a computer long after the exposure), is exact and requires no impressive electronics.  The impressive electronics for ISO control is analog scaling which multiplies signal and shot noise more than readout noise, improving the slope of the curve at low ISO (where readout noise is significant).
Title: Re: DxO marks
Post by: Fine_Art on December 18, 2012, 02:20:16 AM
I think you've got things back to front here when you write; "You can't just increase sensor resolution forever, expecting the lenses to handle it."

Lenses do not handle sensor resolution. It's the other way round. A lens will always project an image of a certain quality and resolution, depending on the aperture used, regardless of the quality of the sensor.

However, something is always lost as the sensor records the projected image from the lens, even when it's a low quality lens and high pixel-count sensor.

If the lens used is a high quality lens, then of course more is lost in the recording process than would be the case using a low quality lens with the same sensor.

However, setting aside semantics, you are correct that one cannot expect unlimited increases in sensor resolution to continue to provide increased resolution from the same lens. As I mentioned in my reply to Peter, there's a situation of diminishing returns that applies so that at some point any increase in the resolution of the recorded image due to increases in sensor resolution will be so small that it will be unnoticeable in practice.

You can test this for yourself if you've not discarded all your earlier, low resolution cameras, and you don't even have to acquire a low-resolution lens for the purpose. Any 35mm format lens can be considered low resolution from F11 to F32, if it stops down that far.


Ok. I was pretty sure you knew what you were talking about, i just thought the wording was not right.
Title: Re: DxO marks
Post by: BJL on December 18, 2012, 10:35:49 AM
The impressive electronics for ISO control is analog scaling which multiplies signal and shot noise more than readout noise, improving the slope of the curve at low ISO (where readout noise is significant).
That is only impressive as a solution to the problem of the part of the read noise that enters the signal after that analog amplification is applied, such as noise that enters during transportation of the analog signal across the edge of the sensor and then off the sensor to an off-board ADC. More impressive still is avoiding that part of the noise entirely by doing the ADC earlier, at the edge of the sensor, reducing sensor noise to mostly just the dark noise from within the photosites themselves. Some hallmarks of this avoidance of read noise from "downstream" sources are that the sensor noise is lower, and scales in the same way as signal and photon shot noise, so that SNR scales exactly with Exposure Index(*). These desirable characteristics are shown by many of the newer sensors used by Sony, Nikon, Olympus and Panasonic.


(*) A little rant on misuse of jargon: can we please stop using "ISO" as the name for multiple related but different measurements?

The "ISO" dial on a camera is used in adjusting exposure index, and is usually based on the ISO-defined "ISO Standard Output Sensitivity" Ssos or perhaps the "ISO Recommended Exposure Index" Srei, not other ISO defined quantities such as the rarely-used "ISO noise based speed" measure Sn40 and Sn10, or the "ISO saturation speed" Ssat (used by DxO). Most modern cameras should indicate in the EXIF which of Ssos or Srei is used.

See the CIPA document http://www.cipa.jp/english/hyoujunka/kikaku/pdf/DC-004_EN.pdf in which the ISO sensitivity standards for "Standard Output Sensitivity" and "Recommended Exposure Index" were originally described (and which is free, unlike the official ISO documents like 12232). Some readers might want to skip the technical details and look at the Overview on page (i) and the Explanation on page 20, where the disadvantages of using "ISO saturation speed" as a sensitivity measure are mentioned.
Title: Re: DxO marks
Post by: NikoJorj on December 18, 2012, 10:58:33 AM
First, many thanks to the author for this thorough insight into DxOMark!

(*) A little rant on misuse of jargon: can we please stop using ""ISO" as the name for multiple related but different measurements?
I completely agree : as many sensitivity measurements coexist in the same norm, it would be nice to know if one is looking at a saturation-based, SNR-based, gray level-based one or a REI (which is a cool acronym for a completely arbitrary number just made to sound good in tech specs ;D ).

Quote
[...] where the disadvantages of using "ISO saturation speed" as a sensitivity measure are mentioned.
The disadvantages are the same for all output-based methods, they take into account a rendered image, and are biased by rendering choices : tone curve, highlight reconstruction, and so on.
 
A saturation-based method makes much more sense to my eyes if it uses saturation of the raw file, as I understand DxOMark does (http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/ISO-sensitivity).
I'd even be tempted to think that it is the only unambiguous method for determining senstivity, not being influenced by rendering choices! But at least, it's the most relevant one for someone shooting raw, even if it's outside the ISO norm.
Title: Re: DxO marks
Post by: fike on December 18, 2012, 11:26:52 AM
The one aspect of the sensor quality race that I feel is missed is when you actually want to USE all those additional pixels on the new sensors.

I like to print LARGE!  Really LARGE.  Things like 72" x 48" are awesome, and it is what I do.  For me resolution matters...mostly in how many or few frames I need to include in my mosaic panoramic image to get somewhere between 240 DPI and 300 DPI where I like to print.  The normalization of resolution is necessary for comparison when folks won't use all that excess resolution.  For this reason, the screen measurements are slightly more helpful to me.

The other area where that resolution is NOT excess (and thus resolution normalization isn't helpful) is when you want to achieve a long reach with your gear...wildlife photography for example. This is the reason I have, for years, kept APS-C.  With the extra-dense pixel resolution I get additional reach out of my 400 mm lens.  In Nikon world, the D800 has made that reasoning mostly obsolete with a cropped sensor pixel pitch on a full-frame sensor. 

One of the key factors that the article clearly explains is that noise doesn't go up with resolution because it is represented as a ratio--signal to noise ratio.  When you downsample (normalize the resolution for comparison) generally the higher resolution cameras have came out marginally ahead when compared to prior sensor generations.  This normalization makes things easy and clear for noise comparison, but not when you really want all those pixels.  What you begin to realize is that with higher resolution sensors, you can no longer print noise-free at 240 DPI but instead have to move up to 300 DPI.  This means newer pixels aren't as useful at the per-pixel level even though in the end they do help me get to larger prints, just not in a linear fashion.
Title: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: BJL on December 18, 2012, 11:30:46 AM
... REI (which is a cool acronym for a completely arbitrary number just made to sound good in tech specs ;D ).
The disadvantages are the same for all output-based methods, they take into account a rendered image, and are biased by rendering choices : tone curve, highlight reconstruction, and so on.
 
A saturation-based method makes much more sense to my eyes if it uses saturation of the raw file, as I understand DxOMark does (http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/ISO-sensitivity).
I'd even be tempted to think that it is the only unambiguous method for determining sensitivity ...
The REI serves one purpose, probably of little interest to us though: getting out-of-the-camera JPEG's that look reasonable when using fancy multi-zone automated metering systems in casual point-and-shoot photography. My guess is that DSLR's and other system cameras use SOS, not REI.

Saturation-based makes sense for one purpose: wanting to know how much highlight headroom you have in the raw files if you expose "on meter", with no exposure compensation. In that case, the situation is roughly that
- if Ssat = SOS then raw files have a 1/2 stop of highlight headroom: parts of the scene at metered average luminosity are placed 1/2 stop below 18% of maximum raw level.
- if Ssat < SOS then you get more raw highlight headroom (with no exposure compensation). For example, if the difference is one stop, like the "ISO" dial giving SOS=400 but DxO giving Ssat=200, then you get  1 1/2 stops of raw highlight headroom with no exposure compensation.

On the other hand, when comparing noise level and DR at elevated exposure index, only SOS makes sense, because that compares at the same shutter speed when using the same aperture ratio in the same lighting. Pushing the DR and SNR 18% curves down to the left and thus down due to RAW files having more than 1/2 stop of highlight headroom (which DxO does) makes no sense when comparing low-light performance, so those graphs are best read by pushing the dots at ISO 200, 400, etc. back into alignment. Fortunately it seems that the "full SNR" curves as DxO are labelled with the camera's "ISO speed" settings or 200, 400, etc., so those can be compared without adjustment.
Title: Re: DxO marks
Post by: Fine_Art on December 18, 2012, 12:32:22 PM
Good timing on this new DxO report:

"The example above is based on data from DxOMark's database of test reults for more than 2,700 camera and lens combinations. These tests reveal that, on average, about 45% of the resolution is lost due to lens defects."

http://www.dpreview.com/news/2012/12/17/dxomark-introduces-perceptual-mpix-score-for-lenses (http://www.dpreview.com/news/2012/12/17/dxomark-introduces-perceptual-mpix-score-for-lenses)
Title: Re: DxO marks
Post by: Peter van den Hamer on December 18, 2012, 05:54:54 PM
I like to print LARGE!  Really LARGE.  Things like 72" x 48" are awesome, and it is what I do.  For me resolution matters...mostly in how many or few frames I need to include in my mosaic panoramic image to get somewhere between 240 DPI and 300 DPI where I like to print.  The normalization of resolution is necessary for comparison when folks won't use all that excess resolution.  For this reason, the screen measurements are slightly more helpful to me.

Ernst Dinkla said something similar, and he also has huge printers (A0, I guess).

Firstly, DxOMark Camera Sensor doesn't factor in sensor resolution. You are supposed to keep an eye on that requirement yourself.

In print mode, the normalization makes sure that you compare sensors with different resolutions by viewing them in the same way. For example, viewing the print (or a fixed section of the print) at a distance proportional to the size of the print: you view the same information at constant opening angle.

As mentioned in the article, it is easiest to explain when you downscale the higher res to the lower res. But I think it also applies if you would upscale if you algorithm tries hard to avoid uncorrelated noise in the "made up" pixels.

If you print higher resolution images at larger sizes (e.g. maintaining a constant DPI value) while viewing at a constant distance, screen mode is probably indeed more suitable.

Peter
Title: Re: DxO marks
Post by: qwz on December 20, 2012, 02:54:22 PM
Very detailed explanation, thanks.
But article says practically nothing about colour.
DXO gives some points with decimals (strangely called 'bits' - but term bit means binary digit!) to colour but it appear to be a simple measure of noise of three colour channels, nothing more.
But, in practical terms, its only relevant Blue channel, cause it less sensitive in all modern cameras.
Moreover, DXO calls this strange points meaningful for portraits! But no one shoots portraits for per-channel looking for RAW data.

DXO simply fails to say something useful about colour. And i wonder why. Cause people buy megapixels and noise levels?

Turning back to colour, its more interesting and much more critical to know, firstly the spectral transmission of the lens (because DXO is only company testing both lens and sensors), and secondly actual spectral sensitivity of sensels (and colour gamut of cameras).
It differs much between vendors and models (but not samples of this expensive stuff:-).

Software correction fundamentally cannot helps with this problem (until we convert photographer to painter;-) because theres many (infinite number, actually) of different stimuluses in real world which appears on three-chromatic cameras sensor like the same R-G-B values.
Of course, improving tonal resolution can help but not much (and in modern cameras with at least 12bit processing). It happens because spectral sensitivity of colour filter arrays on sensels is not ideal and each channel has some small (or big) differences with spectral sensitivities of human eye (and other sensors too).

Does DXO points says something about it? Nope. Pity.
Does we have many other ways to measure noise of digital cameras with enough practical accuracy? Yes we have it on many websites (and can shoot themselves or ask people to make samples and share RAW files).
And, finally, what so "exciting" in DXO points - simple website database interface giving us illusion of knowledge? Not much.
Title: Re: DxO marks
Post by: Peter van den Hamer on December 21, 2012, 07:21:46 PM
Very detailed explanation, thanks.
But article says practically nothing about colour.

Here (http://peter.vdhamer.com/about/comment-page-1/#comment-2156) is a response that I wrote to a similar remark.
If you do a text search on the article, please use "color".

Peter
Title: Re: DxO marks - 187x
Post by: Peter van den Hamer on December 22, 2012, 08:34:02 AM
Here are some graph updates to cover new DxOMark data for
You may need to log in to see the figure.
Title: Re: DxO marks - 187x
Post by: Peter van den Hamer on December 22, 2012, 10:33:47 AM
Here are some graph updates to cover new DxOMark data for
  • Sony Alpha 99 : 89 (;D)
  • Pentax K5 IIs : 82
  • Canon 6D : 82
  • Nikon 1 V2 : 50 (hmmm)
You may need to log in to see the figure.

Last one (can attach max 4 files per posting).
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: Ray on December 22, 2012, 06:41:05 PM

On the other hand, when comparing noise level and DR at elevated exposure index, only SOS makes sense, because that compares at the same shutter speed when using the same aperture ratio in the same lighting. Pushing the DR and SNR 18% curves down to the left and thus down due to RAW files having more than 1/2 stop of highlight headroom (which DxO does) makes no sense when comparing low-light performance, so those graphs are best read by pushing the dots at ISO 200, 400, etc. back into alignment. Fortunately it seems that the "full SNR" curves as DxO are labelled with the camera's "ISO speed" settings or 200, 400, etc., so those can be compared without adjustment.

Not necessarily, BJL. Not all lenses used at the same aperture let pass the same amount of light. There's almost always some degree of light transmission loss which can vary by as much as the variance between the manufacturer's nominated ISO and the DXO-tested ISO.

For the purpose of testing ISO, DXO do not use a lens, otherwise the results would be all over the place. This is why too much nit-picking attention directed to small differences in ISO serve little purpose for the practical photographer who has to use a lens.

For example, Camera A has 1/3rd of a stop less actual sensitivity than camera B, but the lens used with Camera A may have 1/3rd of a stop lower transmission loss than the lens used with Camera B, which effectively cancels out any ISO differences.

On the other hand, if the lens used with Camera A has 1/3rd of a stop greater transmission loss than the lens used with Camera B, then that difference in ISO sensitivity is effectively doubled, and becomes 2/3rds of a stop, which is significant.

This problem of differences between F/stop and T/stop first came to my attention when a I bought the Nikkor 14-24/F2.8 with Canon adapter. The wide-angle zoom I'd previously been using, mainly with my 5D, was the Sigma 15-30.

Naturally, the first thing I did when receiving the Nikkor lens was compare it with my Sigma 15-30, at equal FoV shooting equal scenes with equal lighting, and using the same camera body, the Canon 5D.

I was of course gratified to see that this copy of my Nikkor lens was indeed quite noticeably sharper than the Sigma 15-30. But what astounded me was the discovery that the same exposure at F2.8 with the Nikkor resulted in the same ETTR using my Sigma 15-30 at its maximum aperture of F3.5. There appeared to be as much as a whole stop of difference between the T/stop values of these two lenses.

Whether or not the magnitude of this difference is really so great, I'd like to find out, just out of curiosity. After all, I was using an adapter with the Nikkor lens which may have influenced the results. I've searched the internet for T/stop values for this lens, but cannot find any reference whatsoever. Nor can I find any test of this lens on the DXOMark website.

Maybe the T/stop is so bad there's a conspiracy to keep it a trade secret.  ;)

Of course, if I really wanted to waste my time exploring such issues, I could do more comparisons between the Nikkor zoom on my D700 and the Sigma zoom on my 5D, making allowances for the less-than-0.25 stop of difference in ISO sensitivity between the two cameras.

However, I see no practical purpose in such comparisons. The D700 has a base ISO of 200, at which ISO performance is at least equal to that of the 5D at ISO 100, and mostly better, especially DR which is a whole stop better. Problem solved! A difference of one whole stop of transmission loss is accommodated, and there's no way I would have gone back to using the Sigma zoom on my 5D in preference to the Nikkor zoom on my D700.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: Peter van den Hamer on December 23, 2012, 07:33:41 AM
For the purpose of testing ISO, DXO do not use a lens, otherwise the results would be all over the place. This is why too much nit-picking attention directed to small differences in ISO serve little purpose for the practical photographer who has to use a lens.

Yes, both nominal sensor ISO and lens f-stop values are routinely too optimistic if you care about say 0.5 stop differences. But you can test a sensor's sensitivity with a lens with known T-stop and vignetting characteristics. DxO measures both values in their lens tests. So I guess that DxO uses a lens in some form or other (commercial lens, collimator) to measure ISO sensitivity. This reduces the amount of stray light that bounces around the room and even bounces around inside the camera.

Although I am perfectly willing to accept that definition ISO metrands can be a fine art (with a rather select audience), I would simply use the measured values from DxO rather than the nominal values specified by the manufacturers (as suggested by BJL). Firstly because it takes care of any weird behavior (like 50 ISO and 100 ISO having the same sensitivity on the 1Dx, or manufacturers cheating with high ISO values). Secondly because typical usage involves comparing measurements from the same lab rather than comparing data from different labs. The latter is too hard because it essentially involves assessing the actual data measurment protocols themselves. Simply comparing DxO results for product A and B should be more than accurate enough for end users. The same applies to results from other serious labs.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: Ray on December 23, 2012, 09:38:12 AM
So I guess that DxO uses a lens in some form or other (commercial lens, collimator) to measure ISO sensitivity. This reduces the amount of stray light that bounces around the room and even bounces around inside the camera.

Peter,
I just checked the DXOMark protocols again to ensure my memory was correct. This is what they say about ISO testing.

Quote
Testing protocol for ISO sensitivity

The purpose of (saturation-based) ISO sensitivity measurement is to measure the exposure necessary to reach a given sensor's saturation point.

To measure the camera sensor’s ISO sensitivity, we set up the camera body alone (without a lens) on a stand to receive light from a controlled source.

The source is positioned far enough away from the camera sensor to ensure good light uniformity on the sensor plane. We then precisely measure the illuminance received by the sensor with a certified lux-meter.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: Peter van den Hamer on December 23, 2012, 01:10:29 PM
DxO says "To measure the camera sensor’s ISO sensitivity, we set up the camera body alone (without a lens) on a stand to receive light from a controlled source."

You are right. And my guess was only partly right: when the lens cannot be removed, they are forced to test through the lens (and include some compensation for T-stop and vignetting) as I had expected. Unfortunately if you cannot separate lens from body, there doesn't seem to be a way to distinguish between the efficiency of the sensor and the light losses within the lens.

Thanks.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: Ray on December 24, 2012, 04:05:07 AM
Unfortunately if you cannot separate lens from body, there doesn't seem to be a way to distinguish between the efficiency of the sensor and the light losses within the lens.

Hi Peter,
I'd wondered about that and also assumed that there'd be no way to distinguish between the efficiency of the sensor and the transmission efficiency of the lens if one couldn't separate lens from body.

However, what seems remarkable is that certain models of P&S cameras with fixed lenses seem to test exactly spot on with regard to ISO sensitivity according to DXO standards, which seems a bit of a coincidence.

For example, the nominated ISO values for the Canon Powershot S95 and the Panasonic FZ150 are exactly the same as the DXO tested values, with the occasional variance of just 1.

The Canon Powershot G15 goes against the trend and actually underestimates ISO sensitivity, according to DXO standards. It's nominated ISO of 100 is actually ISO 114, according to DXO tests.

Another issue I find puzzling is DXOMark's lack of any lens test results for the widely acclaimed Nikkor 14-24/F2.8. DXO have clearly tested this lens a long time ago, otherwise they wouldn't be able to provide the excellent 'volume anamorphosis' corrections for this lens in their RAW converter, and support for the lens with a number of camera bodies.

Why aren't they showing the test results? It's clearly a very popular and excellent lens.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: BartvanderWolf on December 24, 2012, 05:43:30 AM
Hi Peter,
I'd wondered about that and also assumed that there'd be no way to distinguish between the efficiency of the sensor and the transmission efficiency of the lens if one couldn't separate lens from body.

However, what seems remarkable is that certain models of P&S cameras with fixed lenses seem to test exactly spot on with regard to ISO sensitivity according to DXO standards, which seems a bit of a coincidence.

Hi Ray,

Perhaps it is because with a fixed lens, the ISO is already compensated for known lens transmission losses. We've also seen such under-the-hood gain adjustment behavior (http://www.luminous-landscape.com/forum/index.php?topic=47800.msg398677#msg398677) for DSLRs when the Aperture value gets wider than f/2, which is when a small compensating amplification boost of the signal is detectable.

Cheers,
Bart
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: Ray on December 24, 2012, 08:54:34 AM
Hi Ray,

Perhaps it is because with a fixed lens, the ISO is already compensated for known lens transmission losses. We've also seen such under-the-hood gain adjustment behavior (http://www.luminous-landscape.com/forum/index.php?topic=47800.msg398677#msg398677) for DSLRs when the Aperture value gets wider than f/2, which is when a small compensating amplification boost of the signal is detectable.

Cheers,
Bart

Hi Bart,
If this is the case, then one wonders why the manufacturers of more serious cameras with detachable lenses, knowing that all their lenses have T/stop values which are numerically greater to some degree than the F/stop value, do not at least attempt to compensate for such transmission loss by understating the nominated ISO values instead of overstating them.

If one wishes to argue that such manufacturers of DSLRs are not overstating their nominated ISO values but correctly conforming to the ISO standard, perhaps in order to allow headroom for in-camera jpeg conversion, then why would the same manufacturer produce a camera like the S95, which is more likely to be used in jpeg mode, that conforms exactly to the DXO sensor-saturation method of measuring ISO?

Merry Christmas   ;D
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: BartvanderWolf on December 24, 2012, 10:05:33 AM
Hi Bart,
If this is the case, then one wonders why the manufacturers of more serious cameras with detachable lenses, knowing that all their lenses have T/stop values which are numerically greater to some degree than the F/stop value, do not at least attempt to compensate for such transmission loss by understating the nominated ISO values instead of overstating them.

Hi Ray,

So to avoid confusion, they just leave them as they are, and DxO states them as they find them, no alterations to hide what has or has not been done. DxO do not need to normalize the readings at this stage, it would only remove information.

Cheers, and a merry Christmas to all,
Bart
Title: Re: DxO marks
Post by: qwz on December 25, 2012, 11:15:27 AM
Peter van den Hamer
Quote
Here is a response that I wrote to a similar remark.
If you do a text search on the article, please use "color".
Yes, thank you.
There i found this:
Quote
The ability to differentiate colors is called “Color sensitivity” in the article (see Figure 8). The full DxOMark measurements include much more detailed measurements of how the camera handles color. This includes the color gamut which the sensor can handle, the metamerism index, and the spectral behavior of the color filters.
It doesn't explain much, except simple counting of terms. Of course i want to believe that DXO uses most complicated and scientific bunch of methods, but how it exactly affect this strange so-called bits rating???
Moreover, now its only meaning that this MARKS says nothing practical about color except some relative things about unknown difference between cameras and sensors.
For example, previous generation full-frame cameras has drastically different colour gamut and fidelity (i mean D700, 5DII, A900) but it has less than half of so-called "Portrait (color depth) bits" difference. IF DXO measures only per-channel colour noise, why it is so close (cause in reality this three cameras handles noise completely different)??
Or DXO use in this calculations other things too. But this Rating says nothing about very perceivable differences between this cams (and others too).
And so on, and so on.

Maybe more honest to call article on such reputable resource as LuLa "DxOMark Camera Sensor Noise Explained" and tell readers, that third line of this Rating is meaningless?

Title: Re: DxO marks
Post by: Peter van den Hamer on December 25, 2012, 08:11:15 PM
Maybe more honest to call article on such reputable resource as LuLa "DxOMark Camera Sensor Noise Explained" and tell readers, that third line of this Rating is meaningless?

Actually, the article says
Quote
DxOMark Camera Sensor ratings essentially measure image noise and dynamic range.
and
Quote
Another reason why the term “Sensor” in the name of the benchmark can be a bit confusing is that the benchmark only covers the noise performance of the camera sensor.

The original article indeed had a subtitle "A noise benchmark of 183 digital cameras", but I dropped because LuLa templates don't really support subtitles.

Note that DxOMark contains extensive measurement information of the type you are interested in: select a camera -> Measurements -> Color Response (see also "Full CS" if you want to see gamut-related info). So DxOMark provides the data you are probably looking for - but not as part of their benchmark scores. I expect this is because color response or color accuracy is something that can fixed by software (e.g. Lightroom, by loading a "Camera Calibration profile" for your camera type or even your personal camera). So you can make colors as accurate or inaccurate as you wish (although it admit that good color control is tough).

Similarly, I doubt that characterizing the camera gamut more precisely than the  ISO standard 17321 color matrix and metamerism index is helpful. The topics are much more complicated than the stuff that I covered. And the impact on actual photography is again too dependent on software settings.

In other words, statements like "camera A has much more accurate or better-looking colors than camera B" simply may not be meaningful: it is fine if somebody prefers a particular set of color deviations (e.g. the look of Kodachrome or Velvia) but this can be achieved with appropriate software settings. If, on the other hand, somebody wants natural colors, this can be achieved by calibrating with e.g. XRite Color Checker Passport or searching for camera ICC profiles that you prefer.

tell readers, that third line of this Rating is meaningless?

DxOMark defines what it means. And it means something (chroma noise related) that is less fixable than color accuracy. Color accuracy can be fixed pretty well by calibration. Serious noise converters contain calibrations for major cameras. You can provide your own calibration if you want. Chroma noise, on the other hand, harms a (high ISO) image in ways that cannot be fixed without creating other problems.
Title: Re: DxO marks
Post by: qwz on December 25, 2012, 11:30:35 PM
Peter van der Hamer
OK
Title: Re: DxO marks
Post by: ErikKaffehr on December 26, 2012, 02:57:07 AM
Hi,

Just commenting a bit on the issue:

1) DxO mark is really about noise.

2) There is something called metamerism index in the DxO test, which describes how well the sensor can match the 16 color fields of the X-rite color checker.

If this is meaningful info or not can be discussed. Tim Parkin found the index coherent with his own observatiosn: http://www.onlandscape.co.uk/2012/02/the-myth-of-universal-colour/

Best regards
Erik


Very detailed explanation, thanks.
But article says practically nothing about colour.
DXO gives some points with decimals (strangely called 'bits' - but term bit means binary digit!) to colour but it appear to be a simple measure of noise of three colour channels, nothing more.
But, in practical terms, its only relevant Blue channel, cause it less sensitive in all modern cameras.
Moreover, DXO calls this strange points meaningful for portraits! But no one shoots portraits for per-channel looking for RAW data.

DXO simply fails to say something useful about colour. And i wonder why. Cause people buy megapixels and noise levels?

Turning back to colour, its more interesting and much more critical to know, firstly the spectral transmission of the lens (because DXO is only company testing both lens and sensors), and secondly actual spectral sensitivity of sensels (and colour gamut of cameras).
It differs much between vendors and models (but not samples of this expensive stuff:-).

Software correction fundamentally cannot helps with this problem (until we convert photographer to painter;-) because theres many (infinite number, actually) of different stimuluses in real world which appears on three-chromatic cameras sensor like the same R-G-B values.
Of course, improving tonal resolution can help but not much (and in modern cameras with at least 12bit processing). It happens because spectral sensitivity of colour filter arrays on sensels is not ideal and each channel has some small (or big) differences with spectral sensitivities of human eye (and other sensors too).

Does DXO points says something about it? Nope. Pity.
Does we have many other ways to measure noise of digital cameras with enough practical accuracy? Yes we have it on many websites (and can shoot themselves or ask people to make samples and share RAW files).
And, finally, what so "exciting" in DXO points - simple website database interface giving us illusion of knowledge? Not much.

Title: Re: DxO marks
Post by: Peter van den Hamer on December 26, 2012, 06:35:46 AM
..metamerism index in the DxO test, which describes how well the sensor can match the 16 color fields of the X-rite color checker.

Actually, metamerism represents to what degree colors with different spectra may map to the same RGB (etc) value. Once such "metameric failure" happens, no amount of calibration can distinguish the original two colors.

There are other metrics for how well colors represent say the 16 color patches in the Gretag-Macbeth chart (see delta-E in http://en.wikipedia.org/wiki/Color_difference (http://en.wikipedia.org/wiki/Color_difference)).

This is roughly the level where my understanding of color management stops.
Title: Re: DxO marks
Post by: ErikKaffehr on December 26, 2012, 07:29:46 AM
Update: formula for SMI enclosed.

Hi,

This is what DxO-marks says about SMI: http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/Color-sensitivity

SMI: Sensitivity Metamerism Index
The sensitivity metamerism index (SMI) is defined in the ISO standard 17321 and describes the ability of a camera to reproduce accurate colors. Digital processing permits changing color rendering at will, but whether the camera can or cannot exactly and accurately reproduce the scene colors is intrinsic to the sensor response and independent of the raw converter.
The underlying physics is that a sensor can distinguish exactly the same colors as the average human eye, if and only if the spectral responses of the sensor can be obtained by a linear combination of the eye cone responses. These conditions are called Luther-Ives conditions, and in practice, these never occur. There are objects that a sensor sees as having certain colors, while the eye sees the same objects differently, and the reverse is also true.
SMI is an index quantifying this property, and is represented by a number lower than 100 (negative values are possible). A value equal to 100 is perfect color accuracy, and is only attained when Luther-Ives conditions hold (which, as previously stated, never happens in practice). A value of 50 is the difference in color between a daylight illuminant and an illuminant generated by fluorescent tubes, which is considered a moderate error.
More precisely, SMI is defined as
 ,
where  is the average CIELAB error observed on a set of various colors. In our experiments, we used the 18 colored patches of a Gretag McBeth color checker, as ISO 17321 recommends. The SMI varies depending on the illuminant.
In practice, the SMI for DSLRs ranges between 75 and 85, and is not very discriminating. It is different for low-end cameras (such as camera phones), which typically have a SMI of about 40. For this reason, we give this measurement as an indication but do not integrate it in DxO Mark.


Best regards
Erik
Title: Re: DxO marks
Post by: qwz on December 26, 2012, 11:10:35 AM
ErikKaffehr
Yes, Erik, thank you. Its exactly about i worried.
But two-digit index about conformity of only 24 color patches.
(Test target is printed and has much more narrower gamut).
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: BJL on December 26, 2012, 04:10:45 PM
Not necessarily, BJL. Not all lenses used at the same aperture let pass the same amount of light.
An issue that is irrelevant when using TTL light-metering. The only worry that I would have about the overstatement of a camera's sensitivity settings on a camera is if with comparable lenses and the same aperture ratios and apertures and lighting conditions, one camera uses a significantly longer exposure time than others when the cameras are at the same "ISO" sensitivity setting. The DxO measurements of raw level placement tell me nothing about that problem.

However, if you know the T-stops of your lenses and use an external non-TTL lightmeter and wish to use that information to place the midtones a certain number of stops below the maximum raw level, then I can see how the DxO saturation based sensitivity measurements could be useful to you.

Meanwhile, I am happy to agree with the CPIA and ISO that the measurements based on the Standard Output Sensitivity as defined by ISO standard 12232, and as used by most or all cameras makers, are valid and useful, especially for those of us who use TTL light metering most or all the time.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: bjanes on December 26, 2012, 05:04:46 PM
Meanwhile, I am happy to agree with the CPIA and ISO that the measurements based on the Standard Output Sensitivity as defined by ISO standard 12232, and as used by most or all cameras makers, are valid and useful, especially for those of us who use TTL light metering most or all the time.

One should not assume that the SOS is used with his camera. The Nikon D800e uses REI (recommended exposure index). My 800e places the metered tone at 14% saturation, 2.82 stops below clipping. This is close to the 12.7% saturation of the old saturation standard and is in line with the DXO measured ISO of 73 for the nominal camera setting of 100. This value is 2.82 stops below saturation allows about 0.34 EV of headroom for the highlights. The saturation standard gives a saturation of 12.7%, allowing 0.5 EV of headroom for the highlights.

Regards,

Bill
Title: Re: DxO marks
Post by: joofa on December 26, 2012, 10:09:45 PM

The underlying physics is that a sensor can distinguish exactly the same colors as the average human eye, if and only if the spectral responses of the sensor can be obtained by a linear combination of the eye cone responses. These conditions are called Luther-Ives conditions, and in practice, these never occur. There are objects that a sensor sees as having certain colors, while the eye sees the same objects differently, and the reverse is also true.


I guess the easy way to visualize the above is as in the graphic below. The recording of color in a device (or human eye) can be modeled as a projection as illustrated below. Different spaces means different "angles" of projection resulting in metamerism and metameric failures. Saying that "Luther-Ives is not satisfied" is equivalent to saying that the two spaces, say sensor and eye, are two different 3D spaces embedded in a larger, higher dimensional space.

(http://djjoofa.com/data/images/metamerism_illustration.jpg)
Title: Re: DxO marks
Post by: Peter van den Hamer on December 27, 2012, 07:18:35 AM
I guess the easy way to visualize the above [being metameric failures in color representation] is as in the graphic below.
(http://djjoofa.com/data/images/metamerism_illustration.jpg)

Cool, Joofa! I recognize the loss of information when going from a spectrum to 3 color channels (RGB).

I am struggling whether this loss of information is equivalent to a lineair projection. So far, I guess you are right: each color channel "sees" the integral of (the light spectrum falling on the sensor x the sensor channel's spectral sensitivity). The spectrum can be approximated as a large number of individual measurements (think: bar chart) that are subject to a lineair transformation aka projection.

Question: when people calibrate colors, they are essentially (non-lineairly) un-distorting a 3 dimensional color space. Why is that needed?
Let's assume, for simplicity, we only want to make the camera see things the exact same way our eyes do (read: let's not get into trying to simulate D50 with tunsten, etc).

Is that because the "projection" from a continous/high-dimensional spectrum to RGB somehow gets distorted? Or is this un-distorting just a hack to compensate for effects of the projection (on "important" colors, like Gretag-Macbeth patches)? In other words, if had a true Luther-Yves conformant sensor, would I still need non-lineair color calibration software (e.g. burried deep down in LR)?
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: BJL on December 27, 2012, 01:09:25 PM
One should not assume that the SOS is used with his camera. The Nikon D800e uses REI (recommended exposure index). My 800e places the metered tone at 14% saturation, 2.82 stops below clipping. This is close to the 12.7% saturation of the old saturation standard and is in line with the DXO measured ISO of 73 for the nominal camera setting of 100. This value is 2.82 stops below saturation allows about 0.34 EV of headroom for the highlights. The saturation standard gives a saturation of 12.7%, allowing 0.5 EV of headroom for the highlights.
The SOS standard allows for some rounding, to accomodate the conventional 1/3 stop increments, and the gap between 12.7% and 14% is within that tolerance. But I agree that in principle it is allowed by CIPA rules for a CIPA member like Nikon to use REI instead of SOS.

How are you measuring this 14%? SOS and REI are defined in terms of output in JPEG or such, not raw files.

P.S. Here are the definitions again, in the original CIPA standards that became ISO standards: http://www.cipa.jp/english/hyoujunka/kikaku/pdf/DC-004_EN.pdf
Title: Re: DxO marks
Post by: joofa on December 27, 2012, 04:04:31 PM

I am struggling whether this loss of information is equivalent to a lineair projection.

Yes, it is a linear projection.

Quote

Question: when people calibrate colors, they are essentially (non-lineairly) un-distorting a 3 dimensional color space. Why is that needed?

Because, by using a nonlinear transformation it is potentially more likely to obtain a mapping between desired and measured responses that satisfies the maximal number of points.

Quote

In other words, if had a true Luther-Yves conformant sensor, would I still need non-lineair color calibration software (e.g. burried deep down in LR)?

Luther Ives works only if there are no noise concerns. Therefore, even if Luther Ives is satisfied, noise can make one use nonlinear techniques to get a better response.
Title: Re: DxO mark: the uses of the different ISO-defined sensitivity measures
Post by: bjanes on December 27, 2012, 06:06:40 PM
The SOS standard allows for some rounding, to accomodate the conventional 1/3 stop increments, and the gap between 12.7% and 14% is within that tolerance. But I agree that in principle it is allowed by CIPA rules for a CIPA member like Nikon to use REI instead of SOS.

How are you measuring this 14%? SOS and REI are defined in terms of output in JPEG or such, not raw files.

P.S. Here are the definitions again, in the original CIPA standards that became ISO standards: http://www.cipa.jp/english/hyoujunka/kikaku/pdf/DC-004_EN.pdf

BJL,

Thanks for your comments. Since I use raw files for my work, neither the REI or SOS methods are applicable, so I use the saturation standard (as does DXO). Their method is shown here (http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/ISO-sensitivity).

I repeated my measurements and used aperture priority for determining the sensor saturation at the nominal light meter reading. This metering mode uses continuously variable shutter speeds and avoids the granularity of the manual shutter speed settings. The sensor saturates at about 15778 in the green channels. At the metered exposure, the raw pixel value in the green channels was 1829, giving a saturation of 12.17%. This allows 0.5 EV headroom for the highlights.

To calculate the ISO, I took shots of my computer screen which has a calibrated luminance of 120 cd/m^2. The sensor saturated with an exposure of 0.85 seconds at f/8. Plugging these values into the DXO formula, I get an ISO of 69, which is close to the DXO measurement of 73.
 
Regards,

Bill
 
Title: DxO mark: is anyone calibrating shutter speed selection and thus EI?
Post by: BJL on December 28, 2012, 04:34:25 PM
Since I use raw files for my work, neither the REI or SOS methods are applicable, so I use the saturation standard (as does DXO). Their method is shown here (http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/ISO-sensitivity).
Thanks for the details Bill.

As noted somewhere, when working with raw (and thus linear) data, the saturation measure is essentially equivalent to a mid-tone placement measure akin to SOS, where one for example determines the exposure that places an 18% gray card image at 1/2 stop below 18% of maximum raw level, or about the 12.7% that you mentioned. So there is probably no major difference on that count.

Unfortunately, what none of these approaches measures is something that I would like calibrated, which is exposure index: do the combinations of aperture and shutter speed recommended by the in-camera light meter under given lighting conditions match what you would get from an external light meter set to the same ISO sensitivity as the camera? (Preferably with adjustment for the discrepancy between f-stops and T-stops due to the lens transmission factor as all the ISO and CIPA sandards specify, to keep Ray happy.) Is any camera testing or review site making that sort of check? The REI standard is meant to allow camera makers to approximate this, but with adjustments when using fancier pattern metering systems and algorithms instead of simple center-weighted or spot metering.


P. S. what will or should DxO do if an "ISO-free" sensor comes along, meaning one that does not benefit at from variable analog gain, and a camera with such a sensor uses a fixed analog gain, handling exposure index settings in the digital domain, during raw-to-JPEG conversion, so that raw files reflect variations in EI solely by metadata? In fact, do not some older DMF backs do exactly this?

As far as I can tell, this would lead to the DxO-style raw-based sensitivity readings being the same regardless of the "ISO" setting on the camera. More realistically, it could make sense for analog gain on many modern sensors to top out quite low, between 200 and 800, with all higher EI settings handled digitally. Instead it seems that some cameras do use purely digital gain only beond some level, but apply it to raw files with bit-shifting of the ADC output before recording it in raw files, pointlessly sacrificing highlight headroom.
Title: Re: DxO marks
Post by: Ernst Dinkla on December 29, 2012, 07:11:39 AM
Update: formula for SMI enclosed.

Hi,

This is what DxO-marks says about SMI: http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/Color-sensitivity

Best regards
Erik

What is the light source used in the test, the ISO prescription and the one actually used?  Or is the test done over a range of color temperatures or light sources?  I recall something like Canon sensors being more color accurate near 4000K light than at 5000K. Which probably does not give a good SMI quote. DxO  did not test Sigma sensors but it would be interesting to see what their SMI result is.

Luther-Ives condition can not be met by RGB filtered sensors but another eye. Would multi-spectral imaging meet that condition?
There has been the Sony RGBE sensor: RD1http://en.wikipedia.org/wiki/RGBE_filter
For art reproduction multi-spectral imaging is already done.
The HP G4010 desktop scanner uses another approach by scanning twice with different light sources and stacking the samples with adequate algorithms.
The women that have tetrachromacy set another condition too :-)

The Portrait score as a reference to art reproduction like I thought and SMI not counting in "Portrait" scoring seems more or less contradictive. Better keep it at Portrait score if the main ingredient is color noise.

--
Met vriendelijke groet, Ernst

http://www.pigment-print.com/spectralplots/spectrumviz_1.htm
December 2012, 500+ inkjet media white spectral plots.
Title: Re: DxO marks
Post by: hjulenissen on December 29, 2012, 07:59:50 AM
ErikKaffehr
Yes, Erik, thank you. Its exactly about i worried.
But two-digit index about conformity of only 24 color patches.
(Test target is printed and has much more narrower gamut).

You want something like this:?
(http://www.onlandscape.co.uk/wp-content/uploads/2012/02/d700-vs-40d.jpg)

Title: Re: DxO marks
Post by: ErikKaffehr on December 29, 2012, 08:40:58 AM
Hi,


I am somewhat skeptical about DxO and color. The description on their web site is not very complete.

I wrote down a bit about color here, with links to other articles, but I think that color is highly subjective: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/71-mf-digital-myths-or-facts?start=9

Best regards
Erik


What is the light source used in the test, the ISO prescription and the one actually used?  Or is the test done over a range of color temperatures or light sources?  I recall something like Canon sensors being more color accurate near 4000K light than at 5000K. Which probably does not give a good SMI quote. DxO  did not test Sigma sensors but it would be interesting to see what their SMI result is.

Luther-Ives condition can not be met by RGB filtered sensors but another eye. Would multi-spectral imaging meet that condition?
There has been the Sony RGBE sensor: RD1http://en.wikipedia.org/wiki/RGBE_filter
For art reproduction multi-spectral imaging is already done.
The HP G4010 desktop scanner uses another approach by scanning twice with different light sources and stacking the samples with adequate algorithms.
The women that have tetrachromacy set another condition too :-)

The Portrait score as a reference to art reproduction like I thought and SMI not counting in "Portrait" scoring seems more or less contradictive. Better keep it at Portrait score if the main ingredient is color noise.

--
Met vriendelijke groet, Ernst

http://www.pigment-print.com/spectralplots/spectrumviz_1.htm
December 2012, 500+ inkjet media white spectral plots.
Title: Re: DxO marks
Post by: qwz on December 30, 2012, 06:08:21 AM
hjulenissen
Yes, but MaxMax.com uses JPEG - and this method completely wrong.
Title: Re: DxO marks
Post by: qwz on January 09, 2013, 03:44:46 AM
And about DXOs new results.

А99 - 1555 Low-Light ISO
RX1 - 2534 Low-Light ISO (pretty similar to the same sensor in D600)

As we know, a99 mirror takes around 1/4 of light.
How exactly DXO measured such big difference???
Title: Re: DxO marks
Post by: hjulenissen on January 09, 2013, 04:52:25 AM
hjulenissen
Yes, but MaxMax.com uses JPEG - and this method completely wrong.
Do you have a reference for this?

I did not know that they used JPEG (I had a look at their website and the text seemed to suggest that they used raw, though they did use imprecise language). I can not imagine how one would use JPEG for this, as WB and color correction should make any measurement strange and meaningless.

-h
Title: Re: DxO marks
Post by: dreed on January 09, 2013, 05:08:06 AM
And about DXOs new results.

А99 - 1555 Low-Light ISO
RX1 - 2534 Low-Light ISO (pretty similar to the same sensor in D600)

As we know, a99 mirror takes around 1/4 of light.
How exactly DXO measured such big difference???

Maybe the impact of the mirror on low-light ISO performance is not linear?
Title: DxO sensitivity is effected by analog gain, not just signal at photosites
Post by: BJL on January 09, 2013, 11:36:37 AM
And about DXOs new results.

А99 - 1555 Low-Light ISO
RX1 - 2534 Low-Light ISO (pretty similar to the same sensor in D600)

As we know, a99 mirror takes around 1/4 of light.
How exactly DXO measured such big difference???
Because DxO is not measuring the signal in photoelectrons counted by the sensor: it is looking at raw output levels produced at the various ISO exposure index settings on the cameras, after analog gain and analog-to-digital conversion. Thus, a camera maker's decision to apply a greater or lesser amount of analog gain at a given ISO setting will give a higher or lower DxO sensitivity "score", even from the same sensor tested with the same exposure H in lux-seconds (e.g same intensity of sensor illumination L in lux and same exposure time t in seconds; H = L*t, ISO defined exposure index EI = 10/H.)

Let me explain once again that DxO is _not_ measuring a pure sensor (photosite) characteristic, but a "processing pipe-line" characteristic including the effects of gain applied after the signal in photoelectrons is moved from the photo-sites.

By the way, my analysis of the DxO results for some different cameras with the same sensor confirm that the horizontal scales on its SNR and DR curves do not compare sensors at equal exposure in lux-seconds, so are misleading in comparing performance in low light conditions. You get far closer to comparison at equal exposure (i. e. at equal exposure index) if you move each dot on those curves horizontally back to the values 100, 200, 400 etc. indicated by the cameras' ISO sensitivity settings. That is, compare at the ISO exposure index values used by the camera, not the DxO raw file level saturation measurements.
Title: Re: DxO sensitivity is effected by analog gain, not just signal at photosites
Post by: Vladimirovich on January 09, 2013, 12:10:53 PM
By the way, my analysis of the DxO results for some different cameras with the same sensor confirm that the horizontal scales on its SNR and DR curves do not compare sensors at equal exposure in lux-seconds, so are misleading in comparing performance in low light conditions. You get far closer to comparison at equal exposure (i. e. at equal exposure index) if you move each dot on those curves horizontally back to the values 100, 200, 400 etc. indicated by the cameras' ISO sensitivity settings. That is, compare at the ISO exposure index values used by the camera, not the DxO raw file level saturation measurements.

can you show us that analysis please ? you probably had 2 raw files and we can see raw histogram from rawdigger to illustrate the saturation, right ?

Title: Re: DxO sensitivity is effected by analog gain, not just signal at photosites
Post by: Ray on January 09, 2013, 08:31:01 PM
You get far closer to comparison at equal exposure (i. e. at equal exposure index) if you move each dot on those curves horizontally back to the values 100, 200, 400 etc. indicated by the cameras' ISO sensitivity settings. That is, compare at the ISO exposure index values used by the camera, not the DxO raw file level saturation measurements.

I'm not sure what point you are trying to make here, BJL, unless your point is that the noise performance of two cameras used at equal shutter speed can vary to an even greater extent than it would if different shutter speeds were used to compensate for different ISO sensitivies.

If two cameras have the same fundamental performance in terms of SNR and DR etc, but one of the cameras has a third of a stop lower sensitivity, as measured by DXO, then that camera with the lower sensitivity will require a third of a stop slower shutter speed to achieve an equally correct or full exposure, exhibiting the equal SNR and DR that it is potentially capable of.

However, graphs do have a vertical and horizontal axis, so it should be no problem to guess what the difference in SNR and DR will be when two cameras are used at the same shutter speed. This gives one an appreciation of just how significant in practice, the different 'real' sensitivies of sensors may be.

For example, the DXO graphs for SNR, comparing the Sony RX-1 with the A99, indicate that using the same shutter speed with the RX-1 and A99, at at their base ISOs of 100 for the RX-1(actually 81), and ISO 50 for the A99 (actually 48), would produce at least a 2dB worse SNR on the A99. One can guess this value simply by moving one's eye vertically down from the best reading for the RX-1 (at ISO 81) to an imagined ISO 81 on the A99 graph.

However, if one uses whatever shutter speed is required to produce a full exposure with each camera at its base ISO, the differences are less. SNR at 18% is a negligible 0.2dB down for the A99, but DR is 1/3rd of a stop down, which is getting close to significant.

These results are roughly consistent with the fact that the A99 has a fixed, semi-transparent mirror which unavoidably redirects a small portion of the light. Consequently, at all manufacturer-nominated ISOs, one would expect the A99 to require a longer exposure to achieve, hopefully, the same or at least nearly the same performance.

The interesting point is, even with the longer exposure, the noise and DR performance of the A99 is not quite as good as one might expect. The third of a stop lower DR performance at the different base ISOs, is perhaps not significant, but the 2/3rds of a stop difference at the manufacturer-nominated ISOs of 800, is significant. In other words, after giving the A99 a good 1/3rd of a stop more exposure than the RX-1 at ISO 800, the RX-1 still retains a 2/3rds of a stop DR advantage.

If one were to use the same shutter speeds with both cameras at the manufacturer-nominated ISOs of 800 (and same FL of lens at same aperture and same T/stop of course), the RX-1 would have a full stop DR advantage, and as much as 3dB SNR-at-18% advantage.

Such differences can be of practical significance when shooting in full manual mode, selecting a specific shutter speed required to freeze movement, and a specific F/stop for DoF and/or sharpness purposes. The RX-1 in these circumstances, according to DXO's measurements, should produce approximately (or as much as) a whole stop better DR and about 2/3rds of a stop better SNR, assuming lenses with the same T/stop are used. That's quite a significant improvement.

Tonal Range and Color Sensitivity will also be noticeably better in the RX-1 at equal shutter speeds and F/stops.

There's also another issue which still puzzles me a bit, which Bart has not yet clarified  ;)  .

The RX-1 has a fixed lens, and therefore any ISO sensitivity measurements by DXO must include the lens which will have a specific T/stop factor unavoidably included in the ISO sensitivity results. The T/stop factor is a transmission loss due to the opacity of the lens elements, which effectively reduces the size of the aperture regards exposure requirements. The lens on the RX-1 at F2 might effectively be F2.5 regards shutter speed requirements. I'm not aware of any lens that has a T/stop rating which is numerically smaller than the actual F/stop rating. That would imply a light transmission gain, instead of a loss.

When DXO test the A99 for ISO sensitivity, they do not use a lens. However, the photographer who uses the A99 has to use a lens, and that lens will inevitably introduce another factor which effectively reduces the sensitivity of the sensor, by some degree, whether by 0.1 of a stop, O.25 of a stop, 0.67 of a stop, or sometimes even by a full stop.

So those figures I've extracted from the DXO graphs (and not from my arse, in case you are wondering), are probably very conservative. The best case scenario of a 0.2dB loss in SNR, and a 0.32EV loss in DR, comparing base ISOs at the appropriate shutter speed for an ETTR exposure, could only be reached if the lens used with the A99 had a T/stop equal to its F/stop, which is perhaps unrealistic.

Agreed, Bart?  :)


Title: Getting correct levels in the final (JPEG?) output regardless of raw placement
Post by: BJL on January 10, 2013, 11:34:26 AM
Ray,

    If at the same ISO setting and corresponding exposure level (aperture-shutter speed combination or to be more precise, lux-seconds), two cameras give different level placements in raw files, I see no evidence of a need to adjust exposure level. One can instead achieve correct levels in the final displayed output (JPEG or whatever) by appropriate raw conversion. For example, if the midtone placement is at the Ssat "half stop down" level, then then a default raw conversion with flat tone curve and normal contrast will need to apply a half stop adjustment, multiplying the linear levels by 1.4 at some stage. If instead the camera maker has decided to provide a full stop of highlight headroom, the factor is 2 instead of 1.4; if 1 1/2 stops, then 2.8.

And indeed, raw conversion software generally knows about the different midtone placement in different raw files (because there is no industry standard recommending or requiring any particular raw level placement, and as DxO has shows there is not a hint of uniformity in this respect) and so default conversions seem to give a reasonable midtone placement in JPEG (around 116-118).

At higher exposure index levels (say 800 and up for the EM5) where noise is the greatest concern, cameras are amplifying the dark noise up to well above the quantization noise level of the ADC, so quantization noise is insignificant and different raw placements have no significant effect on the S/N ratios of the data recorded in the raw file. Thus after the appropriate level scaling in conversion from raw, these different choices about raw level placement have no significant effect on noise levels in the final displayed image.

On the other hand, if for whatever reason your goal is to have midtones placed in raw files exactly a half stop below 18% of maximum level, you still do not need to change exposure levels (aperture/shutter speed). Taking the simplified example of a one stop difference (a camera than places metered levels 1 1/2 stop below 18% of maximum raw level), you could
- use the same exposure levels (for example with EI of ISO 800)
- set the camera's ISO one stop higher (1600 in this example)
With in-camera metering, this corresponds to setting a +1 exposure compensation;
with an external meter, it corresponds to setting that meter to an EI of ISO 800.


Do not worry; this does _not_ give SNR at the level indicated by the DxO measurements of 18% SNR for the higher ISO=1600, because the one stop "overexposure" is placing metered light levels twice as high as those used to get the 18% SNR measurement for ISO 1600 setting, and that increase in illumination improves SNR by a factor of 1.4, or 3dB higher: almost exactly the same as the values at ISO 800 setting! No surprise, because SNR 18% is dominated by photon shot noise (until you reach very high exposure index) and so is determined mainly by the count of photons detected by the photosites, very little effected by in-camera noise sources, and for a particular sensor, the photon count is determined by the exposure (lux-sec) delivered to the sensor: that is, by the chosen exposure index, not the ISO setting in the camera or the DxO SSat value.


As an example, I compared two recent cameras with the same sensor (and I have verified equal quantum efficiency): the Olympus E-M5 and E-PL5: see E-M5 and E-PL5 at DxO (http://www.dxomark.com/index.php/Cameras/Compare-Camera-Sensors/Compare-cameras-side-by-side/(appareil1)/839|0/(brand)/Olympus/(appareil2)/793|0/(brand2)/Olympus)

Analyzing the SNR graphs, it is clear that the values on the SNR and DR curves at equal camera ISO settings are in fact at equal exposure levels (shown by near-equal 18% SNR values) so the only difference is that as ISO setting in increased, the gain applied between photosites and raw levels differs, increasingly higher for the E-PL5 than for the E-M5. DxO uses a horizontal axis of DxO SSat values, pushing the dots on the E-PL5 curve to the right and therefore moving the SNR and DR curves "higher", but if instead a graph used exposure levels (equal aperture/shutter speed combinations under equal lighting conditions), the curves would basically agree, with some small fluctuations. This agreement shows that if one proceeds to expose at exposure index equal to camera ISO setting, you get the same SNR at sensor exposure levels all the way down to seven stops below those for 18% at base ISO speed (which is what the 18% SNR value for ISO 25600 setting measures, indirectly.)

That is, in this comparison, variations in the raw level placements used with the same sensor has no significant effect on noise levels in the images produced with equal exposure (lux-seconds), contrary to what is suggested by the DxO SNR graphs, and in the difference in DxO "sport" score between these two cameras. The lower raw placement of the E-M5 does not at all hurt its IQ at equal exposure level.
Title: using DxO to measure base-ISO speed and noise-based sensitivity limits
Post by: BJL on January 10, 2013, 12:38:20 PM
Ray,

    I basically agree with what you say about using DxO data to assess base-ISO sensitivity. The DxO measurements at a camera's minimum ISO setting (and thus minimum gain between photosites and raw level) is a useful measure of the maximum exposure level (minimum exposure index) than can be used and still have a standard "safe" amount of highlight headroom before some highlight photosites get saturated. (Note the fun fact that the ISO 50 setting on the RX1 has exactly the same DxO SSat of 81 as its ISO 100 setting, so the latter is probably already sending full well signal to maximum raw level, and ISO 50 is a one stop "overexposure" with the same gain from photsites to raw levels as ISO 100.)

Also, with the DxO data showing that the A99 has about 1dB worse SNR at equal exposure levels, and so needs about 1/3 stop more exposure to get equal SNR, then this indicates a true sensitivity difference of about 1/3 stop, and I agree that this is probably attributable to the partial mirror of the A99. More generally, DxO SNR 18% graphs can be used to get relevant comparisons of noise levels at equal exposure level, if one reads the graphs correctly.

Indeed, reading where the three horizontal lines on the DxO SNR 18% graphs hit the curve gives measures akin the the ISO 12232 noise-based sensitivities, except that DxO has chosen to mark SNR levels of 38:1, 30:1 and 20:1 rather than the more traditional 40:1 ("excellent image") and 10:1 ("acceptable image") mentioned in ISO 12232 and in this Kodak document: http://www.kodak.com/global/plugins/acrobat/en/business/ISS/supportdocs/ISOMeasurements.pdf

To get noise-based upper limits on usable EI levels, choose a SNR level like the green 38:1 line, read across till it hits the SNR curve, find the dot closest to that value, and read of the exposure level that achieved this SNR. To be more precise, find the two dots on ether side and interpolate the exposure levels. Of course the best estimate of exposure level available from DxO is the manufacturer's ISO, not the DxO measurement of raw level placement, for reasons that I have explained repeatedly.


Apart from this special case where a partial mirror reduces the effective sensitivity of the sensor (maybe DxO could break the A99's mirror off to give us true raw sensor performance measures!) have you found examples where different cameras using the same sensor give significantly different SNR 18% at equal ISO settings?
Title: Re: Getting correct levels in the final (JPEG?) output regardless of raw placement
Post by: Vladimirovich on January 10, 2013, 12:59:32 PM
The lower raw placement of the E-M5 does not at all hurt its IQ at equal exposure level.
now imagine that you do raw+jpg shooting... with the approach when camera undersaturates you either have a good raw (and too bright JPG) or good jpg (and undersaturated raw)... that is not a problem for JPG shooters (if they like in camera JPG from a particular camera) or for raw shooters as they saturate as they want (at the expenses of usable preview).
Title: Re: Getting correct levels in the final (JPEG?) output regardless of raw placement
Post by: BJL on January 10, 2013, 03:29:59 PM
now imagine that you do raw+jpg shooting... with the approach when camera undersaturates you either have a good raw (and too bright JPG) or good jpg (and undersaturated raw).
I do not need to imagine, as I do use RAW+JPEG with the E-M5, and do not see any problem with either raw or JPEG files. The in-camera JPEGs have appropriate levels (not to bright or too dark), and there is no detectable problem with the raw files, with the default conversions by various raw convertors giving JPEGs similar to the in-camera ones default. And as I have tried to explain and illustrate, the lower raw level placement does no measurable harm to noise levels for the E-M5, which is near the extreme for low raw level placement.

But perhaps by "undersaturated raw" you simply mean a raw file with the metered exposure levels placed at less than 12.8% of the maximum level (three stops below maximum), so that highlights can be more than three stops above metered level before they suffer clipping (which at higher than base ISO speed would be due to excessive amplification not over-full wells). My recurring point, as illustrated with the above comparison of the E-M5 to the E-PL5, is that within the range of placements used by various cameras, this seems to cause no detectable harm to the final image quality; it just requires multiplying the raw levels by a factor different from 1.4 at some stage of the raw conversion process.

So what exactly is the disadvantage you see in placing the metered exposure level at less than 12.8%?

I suspect that some people are still confusing two things:
These are quite different at higher ISO settings: for example, at exposure level of ISO 1600, midtone photosites might have 2% of full-well capacity, while even if highlights range up to four stops brighter, the brightest are still at only 32% of capacity. But if raw conversion then places the midtones at 12.8%, those highlights are blown: in fact the signal from any photosite receiving more than 16% of capacity gets "clipped" in the amplification and ADC process. In that situation, the photosites are all under-saturated, but the ADC output is "over-saturated".
Title: Re: Getting correct levels in the final (JPEG?) output regardless of raw placement
Post by: Vladimirovich on January 10, 2013, 03:46:13 PM
I do not need to imagine, as I do use RAW+JPEG with the E-M5, and do not see any problem with either raw or JPEG files: the in-camera JPEGs have appropriate levels (not to bright or too dark), and there is no detectable problem with the raw files

open in rawdigger and see the raw histogram... but I believe that you will call that "non detectable" :-)

, which various raw convertors give similar JPEGs with default settings.

you mean converters like ACR/LR that apply hidden exposure correction (convert to DNG and see baseline exposure tag value)

And as I have tried to explain, the lower raw level placement does no measurable harm to noise levels.

that is not the issue... the effect of difference in gain itself provided that you deliver the same amount of light to the sensor is not that big (provided that you are not clipping by applying a bigger gain for example, etc) - but the issue is the one that I indicated above... if you JPG is "normal" you typically have a lot of room in all raw channels to expose more, but you can't (if you need in camera JPG and/or useable preview)... but I understand that you have no issue to leave a stop or more on the table even if you can have more light to the sensor, it is a personal call.


But perhaps by "undersaturated raw" you mean a raw file

I mean a raw file with >> 1/3 EV to clipping in all raw channels for non spec. highlights... camera meters to produce JPG and if "nominal" ISO is 1 stop ahead of "measured" then manufacturer sacrifices possibly better SNR for extra safety (highlights) for JPG shooters... not a problem if you shoot raw, if you understand what camera does, what raw converters like LR/ACR do (hidden expocorrection) and do not care about previews... then you can meter properly, dial in positive correction and pull back in raw conversion... but otherwise again you leave a lot of light on the table in the most situations... and that was happening with sensors before EM5 where sensors performance was worse...

Title: maximizing raw levels is nothing to do with maximizing sensor exposure
Post by: BJL on January 10, 2013, 04:49:11 PM
open in rawdigger and see the raw histogram... but I believe that you will call that "non detectable" :-)
I only care about differences that are detectable in the final displayed prints, possibly measured by SNR measurements when the sensors receives equal exposure level. I do not care about a mere moving to the left or right of the numerical levels in the raw histogram since my end goal in to look at pictures, not histograms.

Quote
you mean converters like ACR/LR that apply hidden exposure correction (convert to DNG and see baseline exposure tag value)
Again: I do not care about how the converter gets there, I only care about how the results look. I think that baseline exposure tag is there exactly to tell the raw converter where the midtones have been placed, because (despite some misunderstanding of the intent of the descriptive ISO standards) there is no prescriptive industry standard for where levels _should_ be placed in raw files, and camera makers are free to make their own choices. So this tag probably helps raw converters to produce a good default conversion.

By the way, almost any raw conversion involves an "exposure correction", since raw files almost always place midtones half a stop or more below 18%, so that a default raw conversion (flat tone curve of contrast adjustment) must be scaled up by a half stop or more. My recurring question is when and why different levels of scaling up are better or worse. And you seem to agree that it does not matter much:
Quote
that is not the issue... the effect of difference in gain itself provided that you deliver the same amount of light to the sensor is not that big (provided that you are not clipping by applying a bigger gain for example, etc)
Agreed: that has been my main point all along!

Quote
I understand that you have no issue to leave a stop or more on the table even if you can have more light to the sensor, it is a personal call.
You completely misunderstand. Please read the end of my previous post again about the difference between "ETTR" (which is what you are talking about when you mention "more light to the sensor") and the subsequent degree of amplification of that raw signal reflected in ADC output:
None of these differences in raw levels are about giving more or less exposure to the sensor;
they are about how much amplification is applied to that output when the sensor has been given equal exposure level,
and thus where the numerical levels fall in the raw files.


To repeat, the SNR curves for the E-M5 and E-PL5 show that the same ISO settings on those cameras produce the same exposure levels and the same photo-electron counts in the photosites (as shown by equality of the SNR 18% values). The differences of up to 10% in their DxO "saturation" ISO values is not a difference in "light to the sensor": it is in "ADU levels per photo-electron".
Title: Re: maximizing raw levels is nothing to do with maximizing sensor exposure
Post by: Vladimirovich on January 10, 2013, 05:32:25 PM
I only care about differences that are detectable in the final displayed prints, possibly measured by SNR measurements when the sensors receives equal exposure level.

we are talking about your camera (not 2 different cameras) typically receiving less light (less lux-seconds) than possible if you try to keep in camera JPG and/or preview normal (camera metering is tuned for in camera JPGs, so you can't just dial expocorrection to get more light for better raw data w/o risk to get unusable JPG and/or preview) in a situation when "nominal" ISO is significantly higher (1 stop) than "measured" (DxO) ISO... not about just changing gain while keeping aperture and exposure time constant...

I do not care about a mere moving to the left or right of the numerical levels in the raw histogram since my end goal in to look at pictures, not histograms.

and I am not talking about  that - it is you trying to turn the table to the gain game, instead of considering inconvenience of "nominal" ISO being significantly higher (1 stop) than "measured" (DxO) ISO for a raw shooter who wants for whatever reason raw metering tailored to the best raw data, not extra safe guaring of highlights for JPG shooters (albeit they are THE market)
Title: Re: maximizing raw levels is nothing to do with maximizing sensor exposure
Post by: Vladimirovich on January 10, 2013, 05:39:17 PM
and camera makers are free to make their own choices.

they are, nobody is stating that it is against any standards, just that "measured" (DxO) ISO (and "nominal" = "measured") is better for a raw shooter.

So this tag probably helps raw converters to produce a good default conversion.

except you think that receiving a stop less exposure (not gain mind you - but actual light) is somehow does not make any difference...
Title: Re: maximizing raw levels is nothing to do with maximizing sensor exposure
Post by: Vladimirovich on January 10, 2013, 05:45:25 PM
By the way, almost any raw conversion involves an "exposure correction", since raw files almost always place midtones half a stop or more below 18%, so that a default raw conversion (flat tone curve of contrast adjustment) must be scaled up by a half stop or more.

it depends on how do you meter to expose and which your raw converter you use (and how)... for example (in Adobe's realm, default settings, what Adobe DNG converter will write to DNG during raw -> DNG conversion, that is what ACR/LR are using w/ raws /unless overwritten in a custom camera profile/ - conversion just the way to check) baseline exposure for GH2 = -0.5, baseline exposure for GH3 = 0, baseline exposure for EM5 = +0.5...  
Title: Re: maximizing raw levels is nothing to do with maximizing sensor exposure
Post by: Vladimirovich on January 10, 2013, 06:00:36 PM
My recurring question is when and why different levels of scaling up are better or worse. And you seem to agree that it does not matter much....that has been my main point all along!
indeed that I agree - if you receive the same amount of light (aperture and exposure time) and you do not incur any clipping in raw or any ill effects (alleged non linearities, blooming, whatever) near well saturation point then changing just the gain may or may not be beneficial depending on your sensor and if there are benefits they might not be serious (depending on sensor)... for Canon-like sensors there might be some gains in shadows (while gain is applied to a signal, not to digital data), for Sony-like sensors there might be no practical sense, for Panasonic-like sensors (they are closer to Sony than to Canon) - you decide... and for cameras where gain is neither analog nor digital but merely a tag value in raw files it is even not a question... but I was trying to turn the tables  8) myself towards other aspects of the situation...
Title: Re: DxO marks
Post by: Ray on January 10, 2013, 08:51:32 PM
BJL,
I don't believe I've ever read so much complication over what is for me an essentially simple matter. I imagine that novice photographers reading your posts, who are thinking about getting more serious about photography, would not know whether they are coming or going.

I could be wrong, but my impression is that it has been established several years ago that anyone who wants to capture the most detail, and the widest dynamic range with the lowest noise that their camera is capable of, is advised to shoot in RAW mode.

Furthermore, when shooting in RAW mode, one need not be concerned about 'placement of midtones' which you mention so frequently.The concern is, at least my concern is, that I give the maximum exposure to the scene whilst still retaining full detail in any highlight areas that I consider are important to the composition.

The process is known as Expose To The Right (ETTR) which I know you are quite familiar with, so I think it would help in such duscussions if you were to state from the outset that you are proposing an alternative method to ETTR.

My impression is, that you are proposing a method which involves as little post-processing as possible of scenes that generally do not have a high contrast ratio. In fact, I recall in a previous post, you made the comment that you believe most photographs taken by the public at large are not of high-contrast scenes and do not require cameras with a high DR capability, which is no doubt true.

The problem here seems to be that we are both involved in quite different approaches to the art (or craft) of photography. I'll tell you what mine is, then you can tell me what yours is.

For me, when I take a photograph, I generally consider it as a form of extremely detailed note-taking of a scene which I find interesting for any reason. Whether or not I can reproduce that initial interest, or emotional impact, or even enhance it, during the post-processing of the image or print, is the challenge.

Sometimes, first attempts fall flat, and I put the image aside and perhaps return to it months or even years later, with the benefit of greater experience, greater skill (hopefully) and improved processing programs.

The essential point I'm making is that my style of photography is more of a peripatetic, opportunistic and unplanned style, as opposed to the photographer who largely creates the scene he's about to photograph, using various props, and/or controls the lighting, has a client, a time frame and a schedule he has to meet for business purposes.

I can understand that certain professional photographers may find that the jpegs out-of-the-camera are sufficient for their purposes when time constraints are important and getting the image to the client as quickly as possible is a priority.

This is not my situation. Maybe it's yours. So please tell us, BJL, what your situation is, regarding your photographic style.
Title: Re: DxO marks
Post by: Charles Johnson on January 11, 2013, 11:00:08 AM
As far as I can tell, the concept of "camera equivalence" by Falk Lumo is identical to my "equivalent image" that was published on luminous-landscape in 2007.  My essay had the title, "Why Is My 50mm Lens Equivalent to 80mm on a 35mm Camera and Why Is There More Depth-of-Field?"  It appeared in my book, SCP, as Chapter 11 with the title, "What is an Equivalent Image?"  I explained what parameters have to be scaled when the sensor size changed in order to get an image that appears to be the same in every way. However, I did not consider the pixel count or noise floor.  SCP = "Science for the Curious Photographer." 
Title: DxO marks in the low light case (sensor saturation not possible)
Post by: BJL on January 11, 2013, 03:53:49 PM
Ray and Vladimirovich,
    I suspect that we are talking at cross purposes, due to thinking about different situations. My comments are addressed at the low light case where it is not possible to get enough exposure to saturate the sensor: no photosite is close to full.

Then the exposure strategy is to still to expose as far to the right as possible (maximum exposure level, minimum exposure index), but then there is no fixed relationship between how much signal the photosites get and what the raw file numerical levels are: the relationship between the two depends on factors like the analog gain applied, which varies both with the camera's ISO sensitivity setting and with the manufacturer's decisions.

Also I agree with Ray that it is simpler to think only about raw files, not in-camera JPEGs (when worrying about optimum exposure, I bracket, so one exposure might give the best raw while another exposure level gives the best in-camera JPEG.)

As I said before, the DxO SSat is a useful measure at minimum ISO setting, for ETTR purposes.
Title: different sensors can react very differently to analog gain
Post by: BJL on January 12, 2013, 11:51:20 AM
My thanks to Jack Hogan in this post (http://www.luminous-landscape.com/forum/index.php?topic=68334.msg589296#msg589296) for pointing to Bill Claff's excellent Photographic Dynamic Rage comparison chart tool (http://home.comcast.net/~nikond70/Charts/PDR.htm), which provides a nice way to see how different sensors react differently to extra analog gain in low light situations, when no photosite is getting close to full-well exposure. In particular, consider some quite different sensors, the Canon 5D Mk 3, Olympus E-M5, and Nikon D800e.

As far as I can tell from these graphs and other sources:
(Note: with low sensor exposure levels, meaning high exposure index, the histogram of actual photosite signal levels as a fraction of well capacity is far to the left, but the closest we get to seeing that is by setting the camera at its minimum ISO speed setting.)
Title: Re: DxO marks
Post by: Peter van den Hamer on January 22, 2013, 02:29:59 AM
As far as I can tell, the concept of "camera equivalence" by Falk Lumo is identical to my "equivalent image" that was published on luminous-landscape in 2007.

I agree. I actually ran into both articles over time. Due to their similarity, I actually got confused at some point whether they were written by the same author. So when I decided I needed a reference for the topic, I simply found Falk's version first.

Strictly speaking I could have referenced both, or Falk could have referenced you as related prior work (assuming you were first and he knew about your article). I can still add a note to the Luminous Landscape version of the article. Let me know if you want this. There is also a version of the article at DxO's site, but this is harder for me to change (and gets less attention).
Title: Re: DxO marks
Post by: fike on January 22, 2013, 11:08:04 AM
Shifting gears here:

How can it be that an APS-C from Nikon (the D5200) outscores the full-frame 5DIII--84 to 81?  Is this score a true reflection of IQ reality?  Does this undersell the Canon camera, or is Canon THAT FAR behind?

I am really beginning to pine for sites that just show you the reference images and let you draw your own conclusions.  This putting a number to it seems to inflate pretty minor differences and obscure the things that might matter to you...those things you see with your own eyes.
Title: Re: DxO marks
Post by: bjanes on January 22, 2013, 02:24:22 PM
Shifting gears here:

How can it be that an APS-C from Nikon (the D5200) outscores the full-frame 5DIII--84 to 81?  Is this score a true reflection of IQ reality?  Does this undersell the Canon camera, or is Canon THAT FAR behind?

I am really beginning to pine for sites that just show you the reference images and let you draw your own conclusions.  This putting a number to it seems to inflate pretty minor differences and obscure the things that might matter to you...those things you see with your own eyes.

I would take the overall score with a grain of salt. Afterall, the Nikon D800e outscores the Phase One IQ180 back. The weighting of the various tests might not be how you would weight them. In the case of the D5200 and Canon 5DIII, the dynamic range at low ISO is what gives the Nikon a higher score. The SNR at 18% and other measures are better for the Canon. The Canon sensors are limited by old technology with a high read noise at base ISO. However, the larger sensor gives it an advantage when shot noise comes into play. The Nikon and Canon are evenly matched in terms of resolution, but the APS sensor places more demands on the lens which is not taken into account in the DXO testing.

Posting reference images would be a good idea, but then the lens and demosaicing software or hardware also comes into play.

Regards,

Bill
Title: Re: DxO marks
Post by: Vladimirovich on January 22, 2013, 07:21:57 PM
but the APS sensor places more demands on the lens which is not taken into account in the DXO testing.
but then you shall use lenses designed for smaller circle, smaller sized optical elements (diameter of elements) shall have lesser issues when produced even vs FF lenses with only center of the frame used on those,  isn't it so ?
Title: Re: DxO marks
Post by: ErikKaffehr on January 23, 2013, 12:51:03 AM
Hi,

That is much about DR at minimum ISO playing a major role in the DxO weighting.

On the other hand, I have a very high ranking camera the Alpha 99 and also a camera that ranks close to the 5DII the Alpha 900. The Alpha 99 has about two stops more DR, but it took me several months to find an image that would demonstrate it, that was duping a high contrast Velvia slide in a dark room eliminating all stray light. Im real world situation I'd say the DR advantage may be masked by lens flare.

I don't have Canons so I cannot do a direct comparison.

Best regards
Erik


Shifting gears here:

How can it be that an APS-C from Nikon (the D5200) outscores the full-frame 5DIII--84 to 81?  Is this score a true reflection of IQ reality?  Does this undersell the Canon camera, or is Canon THAT FAR behind?

I am really beginning to pine for sites that just show you the reference images and let you draw your own conclusions.  This putting a number to it seems to inflate pretty minor differences and obscure the things that might matter to you...those things you see with your own eyes.
Title: Re: DxO marks
Post by: bjanes on January 23, 2013, 08:48:05 AM
but then you shall use lenses designed for smaller circle, smaller sized optical elements (diameter of elements) shall have lesser issues when produced even vs FF lenses with only center of the frame used on those,  isn't it so ?

Yes and no. I have personal experience only with Nikon. Their crop frame lenses are termed DX and these are represented by midrange zooms and a few primes, all consumer grade. Professional grade lenses are found in the FX line and these are often used on DX cameras when no equivalent DX lens is available. Examples are the 70-200 f/2.8 VR1 and VR2 and the entire Zeiss line.

Your comment about using only the center of the image applies to the 70-200 f/2.8 VR1. That lens had a very good reputation when Nikon had only crop frame sensors, but with the advent of full frame sensors it became apparent that edge resolution was wanting.

Regards,

Bill
Title: Re: DxO marks
Post by: risedal on January 29, 2013, 05:36:09 PM
Shifting gears here:

How can it be that an APS-C from Nikon (the D5200) outscores the full-frame 5DIII--84 to 81?  Is this score a true reflection of IQ reality?  Does this undersell the Canon camera, or is Canon THAT FAR behind?

I am really beginning to pine for sites that just show you the reference images and let you draw your own conclusions.  This putting a number to it seems to inflate pretty minor differences and obscure the things that might matter to you...those things you see with your own eyes.

Hello. this will be my first answer here  (you may have read answers from me at dpreview, Canon Rumors, etc.)

The APS sensor from Toshiba/Nikon and Sony/Nikon with raw-vise   ADC on the sensor edge  has both a high QE  and very low read  out noise regarding electrons which  Canon can not compete with  today because of theirs older  sensor technology= long analog signal path way and theirs late amplifications stages.
This means that Canons DR will be inferior at base iso compare to Nikons sensors. Regarding picture quality and especially above 550 iso the Canon 24x36mm sensor will  be better due the  larger sensor size and the demands of the lenses are smaller with a 24x36mm sensor compared to the pixel density in a APS size sensor and area (contrast, resolution) What we can hope for is that Canon can reduce the  high read out noise, banding  and increase the QE

(my choice of sentence structure and word choice can be a little Swe / english)

 
Title: Re: DxO marks
Post by: ErikKaffehr on January 29, 2013, 06:41:02 PM
Hi,

Canon cannot achieve the high DR at low ISO the latest generation of Nikon, Pentax and Sony can. DxO mark is only about DR and noise and handling of color, it doesn't take resolution into account.

My personal opinion is that DR may be overvalued, but it is certainly nice to have.

At high ISO the Canons catch up.

Best regards
Erik

Hello. this will be my first answer here  (you may have read answers from me at dpreview, Canon Rumors, etc.)

The APS sensor from Toshiba/Nikon and Sony/Nikon with raw-vise   ADC on the sensor edge  has both a high QE  and very low read  out noise regarding electrons which  Canon can not compete with  today because of theirs older  sensor technology= long analog signal path way and theirs late amplifications stages.
This means that Canons DR will be inferior at base iso compare to Nikons sensors. Regarding picture quality and especially above 550 iso the Canon 24x36mm sensor will  be better due the  larger sensor size and the demands of the lenses are smaller with a 24x36mm sensor compared to the pixel density in a APS size sensor and area (contrast, resolution) What we can hope for is that Canon can reduce the  high read out noise, banding  and increase the QE

(my choice of sentence structure and word choice can be a little Swe / english)

 
Title: Re: DxO marks
Post by: risedal on January 29, 2013, 08:54:30 PM
My personal opinion is that a large DR=14 stop , mainly because of low read out noise gives us a freedom to under expose a contrasty motive with several stops, bring in high lights who are fare above 4 stops  of middle grey and then adjust the rest of motive which means that you got a larger visual DR with the  Nikons sensors compared to for example  Canon who has 12 times higher read noise  and  also banding in theirs 11.5 stops of DR ,how much banding  is depending on the model.6D has today the best sensor data of all Canon including 1dx regarding QE and less banding than 5dmk2 mk3