Luminous Landscape Forum

Equipment & Techniques => Cameras, Lenses and Shooting gear => Topic started by: marcmccalmont on April 04, 2012, 06:50:51 am

Title: DxO Mark interpretation and application
Post by: marcmccalmont on April 04, 2012, 06:50:51 am
It seems there is a lot of confusion on how to interpret (and apply) DxO Mark results both sensor and lens. Perhaps one of the smart guys in this forum could write a quick essay on how to properly interpret these metrics and how to apply them to real world photography and camera/lens purchases. Some people seem to misinterpret a score of 95 as saying the D800 is the best camera in the world and the other extreme is because a DSLR has a higher score than the MFDB the rating is hogwash. I think most of us are somewhere in between these 2 extremes. A clearly written essay would be invaluable for most of us.
Marc
Title: Re: DxO Mark interpretation and application
Post by: Tony Jay on April 04, 2012, 07:04:03 am
+1
Title: Re: DxO Mark interpretation and application
Post by: torger on April 04, 2012, 07:21:14 am
What one needs to know is what each individual measurement means, and then decide for yourself what aspects of performance that is important to you. (Information about what the measurements show can be found on the dxomark web site.) For example if 80 megapixels is of key importance than it does not matter how well the D800 36 megapixels perform.

The dxomark single score rating is not so useful because then dxo has itself chosen how important each aspect is, which may not match you own needs. They don't put any value in resolution for sensors for example, which is a great disadvantage for MF sensors. I'd say that today the MF advantage is 90% about resolution.

For lenses it may for some be important that it performs well at f/2.8, and for others f/8, so again their single number metric does not work.

But the individual measurements are valuable, one of the best available. Unfortunate that they have not tested much MF gear though, would love to see tests of Rodenstock and Schneider lenses... the MF world is much about "test for yourself at your dealer" though, so how MF actually performs as a system in controlled scientific test is not so easy to find out. It is good to test for yourself, but having well-made unbiased expert tests as a guide for which things to test and which products that are promising etc is truly helpful I think.
Title: Re: DxO Mark interpretation and application
Post by: Christoph C. Feldhaim on April 04, 2012, 08:41:02 am
I'm not in the market for a digital camera in the moment, but this may change in the future and I'm watching losely what comes up. For me it is a sort of finding out about the development of cost per technical performance level. But is totally clear for me, if I ever buy a high end digital system, I will rent at my favourite shop first (the give rent money back if you buy) and test for myself. Not to confirm the technical awesomeness I already know from DXO, but to experience the handling of the system and first to actually see the beauty or non beauty of the rendering of the world by the system as a whole.
Title: Re: DxO Mark interpretation and application
Post by: Ray on April 04, 2012, 10:10:21 am
For me, the great value of the DXOMark test results is not their over all scores but the side-by-side comparisons one is able to make amongst 3 different models of cameras, of specific sensor performance characteristics presented on graphs on the same page, at various measured ISO settings which are sometimes quite at variance with the manufacturers' claimed ISOs.

I imagine it might be offputting for some people looking at such graphs if they are not familiar with the units of measurement used with the graphs, so it might help to translate a few of the units, although I'm not necessarily the most qualified person to do this, but I'll try.  ;D

We all know the effects on noise of underexposing. It's why the ETTR concept has  been the subject of so many threads on the forum.  Underexpose just one stop (that is, give one stop less exposure than one needs to, without adverse consequences such as camera or subject movement) and one will usually notice an increase in noise somewhere in the image. Skin tones might not be as smooth as they otherwise would be and, if the image has deep shadows, there may be unacceptable noise there, causing one to blacken the shadows more than one might otherwise do during post processing, or use heavier noise reduction which tends to reduce resolution.

Under the DXOMark Measurements tab, there are a number of headings; ISO Sensitivity, SNR at 18% Grey, Dynamic Range, Tonal Range, and Color Sensitivity.

The unit of measurement used for SNR (Signal-to-noise ratio) is the decibel. As I understand, at the luminosity level of 18% grey, a difference of 3dB respresents a difference of one F\stop in exposure. In other words, if Camera A has 3dB better SNR at 18% grey than Camera B, it means one could underexpose an image from camera A by one full stop and still get skin tones (or other smooth surfaces) with about the same amount of noise as Camera B.

The unit of measurement for Dynamic Range is the Exposure Value, or EV. One EV is equivalent to one F/stop. As I understand, DR refers to noise in the deep shadows and should be distinguished from noise at 18% grey. If Camera A has 2 EV greater DR than Camera B, but only 3dB higher SNR at 18% grey, and one underexposes an image from Camera A by two stops, and compares with the image from Camera B which has been correctly exposed, then the noise in the deep shadows should be about the same in both images, but the noise at 18% grey, generally representative of skin tones, should actually be one stop better in the image from Camera B.

A difference on the graphs of less than 0.5EV in DR is of little consequence and probably not noticeable except at extreme pixel-peeping levels.

Tonal Range refers to the number of distinguishable grey levels the sensor can record, and the units used refer to the number of bits required to encode those levels. I believe a difference of less than 1 bit would not (or hardly) be noticeable.

Likewise, Color Sensitivity is also measured in bits, and the number of bits indicates what is required to encode all the subtle nuances of different shades of color the sensor is capable of capturing. Again, any differences in the test results smaller than 1 bit are of little consequence.

The above comments refer to comparisons of normalised or equal-size images. Whilst the 'print' mode on DXO graphs represents a particularly small size of 8"x12" at 300ppi, to avoid having to uprez images from the cameras with a low pixel count, which introduces other problems, the results for comparison purposes apply to whatever normalized size one may choose which doesn't involve interpolation. Whilst the absolute values may change with changing image size, the relative values won't.

In other words, if one compares the D800 with the Phase IQ180, the relative differences shown on the DXO graphs at 8"x12" and 300ppi would also apply if the D800 image were displayed or printed at 16"x24" and 300ppi, and the IQ180 image downsized to 16"x24" at 300ppi.

Now I can't guarantee that everything I've written here is absolutely correct, but it's as I understand it.


Title: Re: DxO Mark interpretation and application
Post by: theguywitha645d on April 04, 2012, 10:41:13 am
The overall score does not mean much. The other data are interesting. I only use it to compare cameras I am interested in purchasing. And only compare sensors with the same format. And like Ray, slight differences in performance are not important. I use this as one data point in my selection. There are other factors that might be more important for me than simply sensor signal output. I like DxO scores, but I am not sure I would buy a camera based solely on the numbers--there is more to photography than that. I think the DxO scores will indicate a poor sensor, not validate a good one--and within a particular format.

But if I were choosing sides, then DxO scores would be either be the be-all and end-all evaluation or a bunch of rubbish depending if my favorite camera was winning or losing. Just imagine the embarrassment if another photographer saw me on the street with an inferior camera, especially if he had a camera with a better score. I would look such a fool.
Title: Re: DxO Mark interpretation and application
Post by: marcmccalmont on April 05, 2012, 06:18:08 pm
Ray
Thanks
Marc
Title: Re: DxO Mark interpretation and application
Post by: Ray on April 05, 2012, 10:14:45 pm
Expanding on what I've written above, I think it might be helpful to explain why the DXOMark over all scores can sometimes appear to be outrageously wrong, causing some people to dismiss the results and give them no credence.

The recent example of the Nikon D800 achieving a higher over all score than the Phase One IQ180 would be a good case to examine.

First we note that the over all score is based upon performance in just 3 categories of camera usage; Portraiture, Landscape, and Sports.

Now these categories may cover most shooting situations, but not all. What about Macro Photography? If this category were included, I think most P&S cameras would get a significantly higher score (in this category) than any MFDB due to the generally extremely close focussing capability of the P&S, sometimes as close as 1cm from the subject, and due to the good DoF at relatively wide apertures which allow the use of a fast shutter speed to freeze movement.

On the other hand, if the over all score were to include a 'Maximum Print Size' category, perhaps alternatively referred to as sensor resolution, then the IQ180 with over twice the number of pixels would undoubtedly get a much higher score than the D800, in this category, which would most certainly push the IQ180 ahead in the over all score.

Similarly, if one were to remove a particular category from the over all score, such as low-light performance for sports, which MFDBs in general are not suited for, then the over all score would likely favour the IQ180.

In the 3 categories shown, that comprise the over all score, the IQ180 has an advantage over the D800 in portraiture as a result of its better redition of subtle shades of color, and I presumes it's lower SNR at 18% grey which is over 1 stop better than that of the D800. However, this 'Portrait' advantage of the IQ180 is offset by the D800's better dynamic range for Landscape purposes, which is 0.8 stops better.

Neither advantage is particularly significant in my opinion, but should represent a noticeable difference. The point is, within the over all score these two factors cancel each other out, leaving the sole reason for the D800's higher over all score being its better low-light performance for sports purposes.

Now I'm sure that most owners of an IQ180 DB did not buy their back for sports coverage, so there should be no need to get upset at the DXOMark results.  ;D

http://www.dxomark.com/index.php/Cameras/Compare-Camera-Sensors/Compare-cameras-side-by-side/(appareil1)/792%7C0/(brand)/Nikon/(appareil2)/746%7C0/(brand2)/Phase%20One
Title: Re: DxO Mark interpretation and application
Post by: ejmartin on April 05, 2012, 10:53:16 pm
This may be helpful:

http://www.luminous-landscape.com/forum/index.php?topic=42158.0

http://www.luminous-landscape.com/forum/index.php?topic=50778.msg420276#msg420276
Title: Re: DxO Mark interpretation and application
Post by: ErikKaffehr on April 06, 2012, 03:06:25 am
Hi,

My interpretation of DxO data is that the measurements are mostly about noise related data. So parameters like resolution and OLP filtering are not taken into account. The measurements are probably reliable. The DxO figure of merit mark is a compound of three parameters:

1) DR at minimum ISO  (Landscape)
2) High ISO performance  (Sports)
3) Color depth (Portrait)

So DxO mark leaves lens aside.

If we look at DR, it is defined essentially as log2(Full Well Capacity / readout noise). Full Well Capacity is the number of electrons each pixel can hold. FWC defines the saturation level in the sensor. Readout noise is measured in electron charges (most conveniently).

So what DR actually tells us is how noisy the readout channel is at minimum exposure. The best sensors today have readout noise of a couple of electrons.

The FWC is related to size of the pixels but also on process technology used. It seems that it is pretty constant function of the pixel area.

Most of the noise we see is what is called shot noise, simply a statistical variation of the number of photons captured. Shot noise is presumed to be Poisson distributed, which means (among other things) that noise = sqrt(signal).

It doesn't really matter for shot noise how many pixels are collecting the data if we split a big pixel into four smaller ones they will still collect the same amount of photons (if fill factor is constant). Each small pixel would be twice as noisy, but if we looked at the image at 50% size the four pixels would appear to be binned, so pixel noise would be the same as in the image with the bigger pixels seen at 100% size.

En extension of this is that shot noise is determined by sensor area, or more correctly by number of pixels times FWC (per pixel).

The "tonal range" measurement in DxO mark measures essentially shot noise. The curves are normally quite close when seen in print mode. If a curve is "higher" at base ISO it would really indicate that FWC was pushed in process technology. If the curves are shifted to the right it indicates that photon capture is more efficient. Better "quantum efficiency", better microlenses or a weaker (less orthogonal) color grid array.

On many older cameras DR is essentially flat up till say 800 ISO after which it starts to drop in the normal fashion. That would indicate a very clean sensor signal but noisy readout circuitry.

So, what does this mean in practice?

Having an extended DR essentially means that we can extract more shadow detail in an image that is correctly exposed to the right. This comes into play mostly at base ISO. On MFD or Sony Exmoor type CMOS sensors. We need to keep in mind that readout noise is ugly noise (salt and pepper type impulse noise) while shot noise is smoother noise. So if we want to extract dark detail we would certainly want to keep read noise down.

We need also keep in mind that DxO mark (sensor) does not measure lens MTF. A very good lens on an MF camera will probably transfer much more contrast on a single structure like a straw of grass than a possibly less good lens on a DSLR. The main reason is that we would use a longer lens on an MF camera that would make for a larger image on the sensor.

On most DSLRs the image is also somewhat diffused by an OLP-filter, mainly intended to suppress color moiré. Having more pixels is essentially almost always advantageous. Bigger sensor with longer lens for equivalent field of view will always yield more detail for the plane in focus. Except for diffraction. But, diffraction is benign to sharpening.

Another factor with DR is that in practice is always limited by lens flare. That is the reason that the Arriflex test target DxO uses has just a few small holes. Better lenses with better coatings, fewer glass air surface, blackened edges on lens elements and good internal baffling will give better shadow detail than complex lenses built with less care.

This may be interesting: http://www.arri.com/fileadmin/media/arri.com/downloads/Camera/Camera_Technologies/2011_05-13_Dynamic_Range_Test_Charts_Brochure.pdf

Best regards
Erik




It seems there is a lot of confusion on how to interpret (and apply) DxO Mark results both sensor and lens. Perhaps one of the smart guys in this forum could write a quick essay on how to properly interpret these metrics and how to apply them to real world photography and camera/lens purchases. Some people seem to misinterpret a score of 95 as saying the D800 is the best camera in the world and the other extreme is because a DSLR has a higher score than the MFDB the rating is hogwash. I think most of us are somewhere in between these 2 extremes. A clearly written essay would be invaluable for most of us.
Marc
Title: Re: DxO Mark interpretation and application
Post by: marcmccalmont on April 06, 2012, 06:23:36 pm
Erik
Thanks
Marc
Title: DxO Mark interpretation: use of "true ISO" in graphs is dodgy
Post by: BJL on April 13, 2012, 09:50:40 pm
I have just one disagreement with Dx0, apart from the consensus view that the individual measurements are far more objective and useful than the aggregate score. This is the use of a sensitivity ("ISO") measurement based on highlight headroom to calibrate comparisons of noise levels and dynamic range. The only valid correction to cameras' stated ISO speed in such comparisons would be to compensate for different cameras giving different shutter speeds under the same lighting, aperture and ISO speed settings, so that the curves line up vertically for equal shutter speeds. What Dx0 instead does is report a lower ISO speed even if a camera gives the same or higher shutter speed, simply because it positions the midtones at a lower output levels, which leaves more highlight headroom.

It would be great if Dx0, or somebody, actually measured sensitivity with the noise-based measures also defined in the ISO12232 standard and called S40 and S10, based respectively on getting SNR of 40:1 or 10:1 for midtones. That would be closer to what a film's ISO speed measures.
Title: Re: DxO Mark interpretation: use of "true ISO" in graphs is dodgy
Post by: Ray on April 14, 2012, 03:21:34 am
I have just one disagreement with Dx0, apart from the consensus view that the individual measurements are far more objective and useful than the aggregate score. This is the use of a sensitivity ("ISO") measurement based on highlight headroom to calibrate comparisons of noise levels and dynamic range. The only valid correction to cameras' stated ISO speed in such comparisons would be to compensate for different cameras giving different shutter speeds under the same lighting, aperture and ISO speed settings, so that the curves line up vertically for equal shutter speeds. What Dx0 instead does is report a lower ISO speed even if a camera gives the same or higher shutter speed, simply because it positions the midtones at a lower output levels, which leaves more highlight headroom.

It would be great if Dx0, or somebody, actually measured sensitivity with the noise-based measures also defined in the ISO12232 standard and called S40 and S10, based respectively on getting SNR of 40:1 or 10:1 for midtones. That would be closer to what a film's ISO speed measures.


Sorry! Can't make head nor tail of this suggestion of yours, BJL. When shutter speed is not an issue to freeze subject movement (or camera shake in the absence of a tripod), maximum image quality is captured with an ETTR exposure in relation to the specific areas of the scene one does not want blown.

Bernard in the following thread at: http://www.luminous-landscape.com/forum/index.php?topic=65415.0  has provided an excellent example at: http://www.flickr.com/photos/bernardlanguillier/6929054304/sizes/o/in/photostream/  of a stitched jpeg from the D800 showing remarkable clarity and detail, and clean shadows, but a totally blown sky.

If the sky is bland and considered to be unimportant within the composition, then it's quite understandable that cleaner shadows take priority.

Exposure in relation to a standard for noise in the midtones might sometimes be in conflict with exposure to gain maximum detail in the highlights. An ISO rating in relation to a sensor's full-well capacity is preferred, in my view. But perhaps I've misunderstood your point.

Title: Base ISO speed is about highlights, elevated exposure index is about shadows
Post by: BJL on April 14, 2012, 08:54:05 am
When shutter speed is not an issue to freeze subject movement (or camera shake in the absence of a tripod), maximum image quality is captured with an ETTR exposure ...
Ray, that is all fine with respect to base ISO speed, and exposing to as to make most use of the full well capacity of the sensors. Indeed, as far as I can tell, the ISO definition of sensitivity based on placement of midtones relative to maximum signal (which is what Dx0 uses) is intended primarily or exclusively for detemnining the base ISO speed: the minimum before you get into the realm of cramped highlights, such as happens with extended low settings like 50.

I am instead complaining about the opposite case, when shutter speed is an issue to freeze subject movement or such. Then, the only comparisons that make sense are of how good or bad the performance is (for noise, DR, etc.) at the shutter speed needed, like comparisons of two cameras both at 1/1000, then both at 1/2000, and so on. So the calibration should be, in essence, so that the horizontal axis is based on shutter speed.
And indeed, in tests at equal f-stop and equal ISO speed setting with the same subject and same ilumination, some cameras give a lower shutter speed than others, which is "cheating" because they can gather more light but will blur the action more at the same "ISO" setting.

The "ISO" calibration that Dx0 does is nothing to do with the cameras' choices of shutter speeds, but only on where midtones are placed in the JPEGs.

For low light/high shutter speed performace, I would prefer tests using manual mode, with all cameras tested at the same sequence of shutter speeds on the same subject with the same illumination.


P. S. I will leave aside for now whether apertures should be chosen for equal aperture ratio or for equal effective aperture diameter and thus for equal DOF! Ideally, I would like to see both kinds of tests: I like the fact that some of the sample scene comparisions at DPReview are done with apertures chosen to give roughly equal DOF, so as to compare equal compositions.


Title: Re: Base ISO speed is about highlights, elevated exposure index is about shadows
Post by: Ray on April 14, 2012, 10:00:46 am
Ray, that is all fine with respect to base ISO speed, and exposing to as to make most use of the full well capacity of the sensors. Indeed, as far as I can tell, the ISO definition of sensitivity based on placement of midtones relative to maximum signal (which is what Dx0 uses) is intended primarily or exclusively for detemnining the base ISO speed: the minimum before you get into the realm of cramped highlights, such as happens with extended low settings like 50.

I am instead complaining about the opposite case, when shutter speed is an issue to freeze subject movement or such. Then, the only comparisons that make sense are of how good or bad the performance is (for noise, DR, etc.) at the shutter speed needed, like comparisons of two cameras both at 1/1000, then both at 1/2000, and so on. So the calibration should be, in essence, so that the horizontal axis is based on shutter speed.
And indeed, in tests at equal f-stop and equal ISO speed setting with the same subject and same ilumination, some cameras give a lower shutter speed than others, which is "cheating" because they can gather more light but will blur the action more at the same "ISO" setting.

The "ISO" calibration that Dx0 does is nothing to do with the cameras' choices of shutter speeds, but only on where midtones are placed in the JPEGs.

For low light/high shutter speed performace, I would prefer tests using manual mode, with all cameras tested at the same sequence of shutter speeds on the same subject with the same illumination.


P. S. I will leave aside for now whether apertures should be chosen for equal aperture ratio or for equal effective aperture diameter and thus for equal DOF! Ideally, I would like to see both kinds of tests: I like the fact that some of the sample scene comparisions at DPReview are done with apertures chosen to give roughly equal DOF, so as to compare equal compositions.


Sorry! Still can't see the problem. All the dots on the graphs are joined with a continuous line. Whilst it's true that the dots are positioned on the graphs in relation to the actual, measured ISO, and don't line up vertically, it's quite easy to make a reasonably accurate guess as to, for example, the DR at equal shutter speeds.

To take a specific example from the following DXOMark graph. By placing the cursor over the dots, the D800 is shown as having a DR of 12.83EV at the manufacturer's nominated ISO of 400, and the D7000 as having a DR of 11.97EV at the manufacturer's nominated ISO of 400, the difference being 0.86EV.

However, the dots don't line up vertically because they are positioned according to the actual, measured ISOs of 297 and 328 respectively.

Surely this is not a problem. It's easy to see that the DR of the D7000 at a position on the graph which vertically lines up with the D800 reading, would be about 12.2EV instead of 11.97, plus or minus a small error which is really irrelevant, and that the difference in DR at the same shutter speed, at the true ISO of 297, is approximately 0.6EV in favour of the D800.

Does one need to be more precise than that?

http://www.dxomark.com/index.php/Cameras/Compare-Camera-Sensors/Compare-cameras-side-by-side/(appareil1)/792%7C0/(brand)/Nikon/(appareil2)/680%7C0/(brand2)/Nikon/(appareil3)/483%7C0/(brand3)/Canon
Title: Re: DxO Mark interpretation and application
Post by: BJL on April 14, 2012, 10:53:02 am
Ray,
    I agree that the main part of the problem can be fixed by careful readers in the way you say: by comparing the values at the dots, ignoring the curves that are potentially shifted horizontally in a poorly justified way. My issue there is just to warn people that the graphs are potentially misleading, especially to the many forum commentators who trust the Dx0 ISO speed measurements and thus also criticise some cameras for "overstating" sensitivity ("ISO speed") when in fact they might simply be allowing a little more highlight headroom, and it is the cameras that go in the other direction (Dx0 measured ISO speed values higher than stated by the camera maker) that are doing something dodgy, by allowing less highlight headroom than the ISO standard requires, and so risking blown highlights. (Blown not because the wells are full, but because the highlight signal is amplified until it hits the ceiling of the amplifier, or of the ADC, or otherwise gets pushed beyond maximum digital output level.)

My other, secondary, issue is a complaint about all low-light performance comparisons I have seen, not specifically of Dx0: some cameras (Canon is often accused, for example) give lower shutter speeds than others at the same "ISO speed" setting. This is hard to explain as an error, and so sounds very much like deliberate gaming of high ISO noise level testing, by giving the sensor a bit more light than the competition. Real world comparisons with moving subjects might be a good way to expose this dodge! If ISO speeds were corrected in that way, there would be a different horizontal movement of the curves.

As I will keep saying, it mystifies me why the very useful ISO standard definitions for noise based sensitivity, S40 and S10, are so rarely measured, published or discussed, despite the almost morbid obsession of so many camera forum participants with noise comparisons.

Some references on ISO12232:2006 noise-based speed measures:
http://en.wikipedia.org/wiki/Film_speed#Noise-based_speed
and pages 18 and 19 of this useful slideshow overview of many aspects of photography related standards and measurement methods from an ISO participant.
http://www.rps-isg.org/DF2008/DigitalPhotographyStandards.pdf
Title: Re: DxO Mark interpretation and application
Post by: Ray on April 14, 2012, 10:45:26 pm
As I will keep saying, it mystifies me why the very useful ISO standard definitions for noise based sensitivity, S40 and S10, are so rarely measured, published or discussed, despite the almost morbid obsession of so many camera forum participants with noise comparisons.

I suspect it's because of the ETTR concept which didn't apply in the days of film. I still can't grasp the purpose of your suggested reorganization along the lines of noise results in relation to specific shutter speeds.

In the case of certain Canon DSLRs, we know that an ETTR exposure at ISO 200, or 400, or 800 etc, has lower noise than the same exposure at ISO 100. I can't see the point in providing noise comparisons at specific shutter speeds that are not related to specific ISO settings.

Furthermore, with certain Canon models, this improvement in noise that can result from the ISO setting, in addition to the choice of exposure, applies only to the one-stop increments of ISO. With some models, an ETTR at ISO 125 produces the same noise characteristics as a shot underexposed by 2/3rds of a stop at ISO 200, and an ETTR at ISO 160 produces the same noise as a shot underexposed by 1/3rd of a stop at ISO 200.

I'm sorry to say, BJL, if DXO were to adopt your suggestion, I think confusion would prevail.

However, if all cameras were like the Nikon D7000, your suggestion might be workable. Unfortunately, not everyone is so privileged as to own a D7000.  ;D

Checking the DXO graphs again, for the D7000, I see that SNR at ISO 100 is 41.1dB. 5 stops down at ISO 3200, SNR is 26.7dB. At 18% grey, a 3dB difference in SNR is equivalent to one stop of exposure. 5 stops' lower exposure should result in SNR at ISO 3200 being 15dB down.

In fact, it's 14.4dB down; a trivial improvement of 0.6dB if one uses ISO 3200. A similar situation applies to DR, except there's no difference at all.

For this reason, the only purpose I see in using a higher-than-base ISO on the D7000, is so I can review the image on the camera's LCD screen, or show it off to someone else.
Title: Elevated ISO speed is for attaining a needed shutter speed
Post by: BJL on April 14, 2012, 11:48:18 pm
I suspect it's because of the ETTR concept which didn't apply in the days of film. I still can't grasp the purpose of your suggested reorganization along the lines of noise results in relation to specific shutter speeds.
...
I can't see the point in providing noise comparisons at specific shutter speeds that are not related to specific ISO settings.


1. ETTR is mostly irrelevant to high ISO settings! It is about making the most of the full well capacity, which is not close to happening a the elevated ISO speeds that I am addressing.

2. The main reason that elevated ISO speeds are used is to get high enough shutter speeds, and in that situation, the only noise comparisons that matter are at equal shutter speed. I am not saying that ISO setting is irrelevant, only that I am interested in noise levels at equal shutter speeds. Of course, it is often best to then choose an appropriate ISO speed, like the one that places the mid-tones at a suitable level.

Let me offer a somewhat extreme hypothetical example:
- We wish to photograph a certain scene at 1/1600s to freeze motion, with f/2.8 lenses, wide open.
- According to a light meter, the Exposure Index (ISO speed setting) should be 1600 (E.g. the illumination is about 5 stops dimmer than midday sun.)
- At EI 1600, camera A indeed gives shutter speed 1/1600s, but camera B gives shutter speed 1/800s, and needs EI=3200 to get the needed shutter speed of 1/1600s.
- At EI 1600 on both cameras, the results from camera A are a bit noisier than those from camera B,
  but
- Once we get 1/1600s on camera B (say by raising the ISO speed to 3200, or by underexposing by one stop at EI=1600 and then adjusting levels up in post), the results from camera B are noisier.

If one does the common thing of comparing noise levels at equal ISO speed of 1600, camera B seems better, but when one needs to photograph such a scene at f/2.8 and 1/1600s, camera A has less noise.

That full one stop difference is a bit of an exaggeration to simplify the numbers, but there seem to be examples of different cameras that vary by up to 2/3 stop in the shutter speed given at equal EI setting.
Title: Re: Elevated ISO speed is for attaining a needed shutter speed
Post by: Ray on April 15, 2012, 06:40:26 am

1. ETTR is mostly irrelevant to high ISO settings! It is about making the most of the full well capacity, which is not close to happening a the elevated ISO speeds that I am addressing.


Sorry to disagree again. ETTR is also about making the most of the analogue amplification of the signal in accordance with a particular ISO setting (instruction). In fact, with most Canon cameras, up to ISO 1600 and certainly up to ISO 800, dynamic range falls only marginally each time one halves exposure and increases ISO. It falls only 1/2 a stop between ISO 100 and ISO 800 on the 5D2. Beyond ISO 800 DR begins to fall more rapidly, falling 0.61EV between ISO 800 and ISO 1600. Above ISO 1600 there is little benefit. One might as well underexpose at ISO 1600.

With such cameras, noise readings which are related only to shutter speed could be confusing and misleading. There's a huge difference in DR (or shadow noise) between an ETTR exposure at ISO 800 and the same exposure at ISO 100, on most Canon DSLRs.


Quote
Let me offer a somewhat extreme hypothetical example:
- We wish to photograph a certain scene at 1/1600s to freeze motion, with f/2.8 lenses, wide open.
- According to a light meter, the Exposure Index (ISO speed setting) should be 1600 (E.g. the illumination is about 5 stops dimmer than midday sun.)
- At EI 1600, camera A indeed gives shutter speed 1/1600s, but camera B gives shutter speed 1/800s, and needs EI=3200 to get the needed shutter speed of 1/1600s.
- At EI 1600 on both cameras, the results from camera A are a bit noisier than those from camera B,
but
- Once we get 1/1600s on camera B (say by raising the ISO speed to 3200, or by underexposing by one stop at EI=1600 and then adjusting levels up in post), the results from camera B are noisier.

At ISOs above the manufacturer's nominated 1600, I get the impression there is generally little benefit to increasing ISO, and that seems to be true for both Canon and Nikon cameras. In other words, the same exposure at either ISO 1600 or ISO 3200 will produce images with about the same amount of noise and DR. For this reason it's probably advisable to play it safe and use ISO 1600, to avoid the risk of blowing out highlights at ISO 3200.

However, there are exceptions. The Nikon D3s appears to be the star performer at high ISO, so one might expect there would be an advantage using an ETTR at ISO 3200 instead of the same exposure at ISO 1600.

Referring to DXOMark again, we find this is indeed the case. At half the ISO 1600 exposure, used at ISO 3200, we get a drop in DR of only 0.57EV, ie, half a stop as opposed to a predicted full stop.

DXOMark provides us with the SNR and DR test results in relation to the real or actual or measured ISO ratings. The position of the dots on the graph is in accordance with the actual, tested ISO values, which is as it should be, in my view. The manufacturer's nominated ISO is also mentioned, but this has no bearing on the position of the dots.

I presume if one were to find two cameras with identical ISO ratings, say ISO 297 instead of the manufacturer's ISO 400, then any exposure which was an ETTR with one camera would also be an ETTR with the other camera, assuming identical scenes and lighting conditions, and assuming equal T-stops for the lenses used.

Variations in lens T-stop values is another reason why noise in relation to exposure would not be meaningful. Not only would the noise and DR values at a given exposure change according to the selection of ISO, but would also change according to the T-Stop value at the particular F/stop used.


Title: Re: Elevated ISO speed is for attaining a needed shutter speed
Post by: Bart_van_der_Wolf on April 15, 2012, 08:09:38 am
There's a huge difference in DR (or shadow noise) between an ETTR exposure at ISO 800 and the same exposure at ISO 100, on most Canon DSLRs.

Underexposing by 3 stops at ISO 100 can hardly be called ETTR. You're arguing for argument's sake.

ETTR is all about optimizing (= maximizing without compromising required highlight data) the number of recorded photons. This is most effective at base ISO where the DR (engineering definition) is the largest. Increasing ISO has nothing to do with that, although it may have other useful effects.

Cheers,
Bart
Title: Re: Elevated ISO speed is for attaining a needed shutter speed
Post by: Ray on April 15, 2012, 09:48:17 am
Underexposing by 3 stops at ISO 100 can hardly be called ETTR. You're arguing for argument's sake.

Cheers,
Bart

Bart,
I never claimed it was. My use of ETTR with my Canon cameras was to fully expose to the right of the histogram whatever the ISO chosen. This is not an argument. It's a statement of fact. My argument would relate to the reasons for doing this. And the reasons are, when the required or desired shutter speed is too fast to enable an ETTR at base ISO, the best results are obtained at the highest ISO which is consistent with an ETTR exposure at that higher ISO, up to ISO 1600 at least.

However, this does not apply when using cameras such as the Nikon D7000 and Pentax K5. The reasons for choosing a higher ISO with such cameras would be so one can review the shot on the camera's LCD screen.

Quote
ETTR is all about optimizing (= maximizing without compromising required highlight data) the number of recorded photons. This is most effective at base ISO where the DR (engineering definition) is the largest. Increasing ISO has nothing to do with that, although it may have other useful effects.

It's certainly true that the lowest noise and highest DR is achieved by recording the greatest number of photons, which is only possible at base ISO. But consider this. Would you agree that an exposure which is one stop underexposed at ISO 100 (in relation to a correct ETTR exposure) will have one stop lower DR as a result of that underexposure? Likewise, would you agree that an exposure at ISO 100 that is 2 stops underexposed will have 2 stops (or 2EV) lower DR than the fully exposed ETTR at ISO 100?

If you check the DXOMark DR figures for the Canon 5D2, you will find that at ISO 100 DR is 11.86EV. At ISO 200 it is insignificantly lower at 11.82EV, which is virtually the same. At ISO 400 is down just a little at 11.62EV, which is just 1/4th of a stop less.

It is clear from these figures that in circumstances where a shutter speed is required that would result in a 2 stop underexposure at ISO 100, the same exposure at ISO 400, which would be an ETTR exposure at ISO 400, will result in a significantly higher DR by about 1.75EV. Can't you see this?  ;D
Title: Re: DxO Mark interpretation and application
Post by: BJL on April 15, 2012, 11:00:19 am
Ray,
    to address your new-found concern about T-stop variations:
Please reconsider my example with "equal T-stop" in place of "equal f-stop". This could be achieved by using the same lens in all tests via mount adaptors. However, for all practical purposes, witu good prime lenses there will be negligible variation in transmission efficiency and thus a negligible discrepancy in T-stop at equal f-stop. Oh, and maybe change the EI values up a few stops, to say 6400 and 3200, because that what I say should be clear even with your examples of those unfortunate old-tech Canon sensors that need to use analogue amplification carefully in order to control noise sources later in the analogue signal path.

In that equal T-stop scenario, can you truly not see why people interested in low-light, high shutter speed performance want to compare images taken at equal shutter speed? (Along of course with optimal choice of exposure index setting and analogue gain levels, to the extent that this helps.)


P.S. When the Dx0 measurement of exposure index at higher "ISO" settings disagrees with that of the camera, it is more likely that the camera maker is conforming to the relevant definition of sensitivity amd exposure index in ISO standard ISO12232:2006, meaning one of the two measurements that the Japanese industry association CIPA requires its members to use:
Standard Output Sensitivity (SOS) or
Recommended Exposure Index (REI).
These are both based on getting appropriate levels in the final (JPEG) output when the exposure level (combination of shutter speed and aperture ratio) is as stanard light metering indicates. DX0 instead uses another measurement, based on so-called saturation based sensitivity, which is _not_ recomended for use by CIPA. We could debate why CIPA rejects that saturation-based measure, but to me it is clear: that measure is what used to be called base ISO speed, and is intended primarily for measuring the minimum safely usable EI setting (maximum amount of exposure sent to the sensor), not at all for calibration of the higher EI settings.

Why might that be? At high enough EI levels, quantization noise from the ADC is overwhelmed by dark noise and photon shot noise, so worrying about placing the signal as close as possible to maximum raw output levels is misguided. Maybe with Canon's technology, "high enough" is not till about EI 1600, but with newer sensor technologies as used by Sony and Nikon (and maybe Panasonic and Olympus), this is true from about 400 and up, and with some MF cameras with their combination of somewhat noisy CCDs, no possibility for on-chip gain, and excellent ADCs, this is possibly true at all EI levels, so that there might be no point in vatying the analogue gain at all.

That is, analogue amplification for the sake of place the midtones as far to the right as is safe in the raw files (this seems to be roughly what you are meaning by "ETTR") is of little of no relevance at sufficiently high EI, and the final tonal placement in output files (JPEG, TIFF or whatever) might best be done largely or completely in the digital domain, with litle or no variation in analogue gain.

At that point, Dx0 style callibration based on level placement in raw files is a red-herring: it is a measurement part-way through the process, which I consider half-baked. By "the process", I mean getting from a required high shutter speed to a final image (JPEG, TIFF or whatever) with correct placement of mid-tones and such and as little noise as possible.
Title: Re: Elevated ISO speed is for attaining a needed shutter speed
Post by: Bart_van_der_Wolf on April 15, 2012, 11:57:12 am
It's certainly true that the lowest noise and highest DR is achieved by recording the greatest number of photons, which is only possible at base ISO. But consider this. Would you agree that an exposure which is one stop underexposed at ISO 100 (in relation to a correct ETTR exposure) will have one stop lower DR as a result of that underexposure? Likewise, would you agree that an exposure at ISO 100 that is 2 stops underexposed will have 2 stops (or 2EV) lower DR than the fully exposed ETTR at ISO 100?

No, not necessarily on all cameras.

Quote
If you check the DXOMark DR figures for the Canon 5D2, you will find that at ISO 100 DR is 11.86EV. At ISO 200 it is insignificantly lower at 11.82EV, which is virtually the same. At ISO 400 is down just a little at 11.62EV, which is just 1/4th of a stop less.

I find it more instructive to view the ratings for the sensor itself than for a downsampled print. But indeed, ISO 100=11.16, ISO 200=11.12, and ISO 400=10.92, showing a similar pattern. The penalty for underexposing and ramping up the gain to boost the signal level is mild in this camera. It has to do with the actual gain factor and the number of electrons per DN (data number, also referred to as ADU) at a given ISO, amongst other things. So by all means use it if a faster shutterspeed or a wider aperture are more important than the modest loss of DR. Of course one probably also boosts the visibility of pattern noise, but photography is also about making compromises. TANSTAAFL.

But the principle remains that one still needs to optimize the exposure to get as many photons in as possible without clipping required highlights. Comparing a 2 stop underexposed ISO 100 versus a correctly exposed ISO 400 makes no ETTR sense.

Cheers,
Bart
Title: Re: DxO Mark interpretation and application
Post by: bjanes on April 15, 2012, 12:37:34 pm

My other, secondary, issue is a complaint about all low-light performance comparisons I have seen, not specifically of Dx0: some cameras (Canon is often accused, for example) give lower shutter speeds than others at the same "ISO speed" setting. This is hard to explain as an error, and so sounds very much like deliberate gaming of high ISO noise level testing, by giving the sensor a bit more light than the competition. Real world comparisons with moving subjects might be a good way to expose this dodge! If ISO speeds were corrected in that way, there would be a different horizontal movement of the curves.

As I will keep saying, it mystifies me why the very useful ISO standard definitions for noise based sensitivity, S40 and S10, are so rarely measured, published or discussed, despite the almost morbid obsession of so many camera forum participants with noise comparisons.

Some references on ISO12232:2006 noise-based speed measures:
http://en.wikipedia.org/wiki/Film_speed#Noise-based_speed
and pages 18 and 19 of this useful slideshow overview of many aspects of photography related standards and measurement methods from an ISO participant.
http://www.rps-isg.org/DF2008/DigitalPhotographyStandards.pdf

DXO (http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/ISO-sensitivity) uses the ISO12232:1998 saturation standard for the ISO exposure index where Ssat = 78/Hsat. Hsat is the exposure in lux.s needed to reach sensor saturation. This means that if one exposes a gray/white target according to the reading of a standard light meter, the sensor will be at 12.8% saturation and there will be a half stop of highlight headroom. This is what I find with my Nikon dSRLs.

This brings up the ISO 2721 standard for calibration of light meters, which dates back to film days. In effect, the light meter is calibrated to an effective subject reflectance of 12.8% (see Thom Hogan (http://www.bythom.com/graycards.htm)). Digital camera makers differ in the amount of highlight headroom that they allow for their sensors. If they choose to allow more than the standard 0.5 EV of headroom, they have two choices. They can adjust the calibration of the meter so that less exposure would be given than according to ISO 2721. However, in this case a photographer using a hand held meter would have to use this different calibration, which would be confusing. The other alternative would be to leave the meter calibration unchanged so that there would be concordance between an external meter and the camera meter and then assign a higher ISO to the sensor than would be warranted according to ISO 12232.  In the case of the Phase One IQ180, a large amount of highlight headroom is allowed. For a nominal ISO of 100, DXO measures a value of 29.

Regards,

Bill
Title: Re: Elevated ISO speed is for attaining a needed shutter speed
Post by: Ray on April 15, 2012, 01:00:12 pm
But the principle remains that one still needs to optimize the exposure to get as many photons in as possible without clipping required highlights. Comparing a 2 stop underexposed ISO 100 versus a correctly exposed ISO 400 makes no ETTR sense.

Cheers,

Bart

It makes perfect sense to me, Bart. ETTR means Expose To The Right, doesn't it? 'The Right' refers to the right of the histogram, doesn't it? Setting aside the unavoidable inaccuracies of a histogram based on a jpeg conversion, the histogram and warning flashing is still a guide for correct exposure. Whatever the ISO setting, if one exposes beyond the right of the histogram, one will unavoidably blow some highlights. If the exposure is such that the histogram doesn't extend fully to the right, the image will be noisier than it otherwise might have been.

The principle of ETTR applies to all ISOs, even those which of themselves don't provide any image quality advantage, such as ISO 3200 on a 5D. If I choose ISO 3200 on a 5D and use a 1/2000th exposure when I could have used a 1/1600th sec without blowing highlights, I will have a lower quality image, noise-wise, because I failed to expose to the right.
Title: Re: DxO Mark interpretation and application
Post by: BJL on April 15, 2012, 01:06:31 pm
Bill, that is basically right, except that Dx0 is clearly not using _sensor_ saturation, which would give a value independent of the EI setting or any amplification applied between photosites and ADC. Instead, Dx0 is using "ADC" saturation, requiring that the levels in the raw file are at a standard placement, with mid-tones about three stops below maximum ADC output level. The 2006 standard is perhaps ambiguous about whether it must be applied to maximum signal in photosites or can alsomapply to maximum signal after analogue amplification and ADC comversion but before any further levels adjustment in conversion to JPEG.

My feeling of course, is that this measurement mid-wy through the process is neither what the ISO standards originally intended nor or much practical relevance to assessing the low light performance of a camera.

For one thing, once at high enough EI that the dynamic range of the signal is several stops less than the bit depth of the ADC, there are several stops of placement in ADC levels that give equally good handling of noise levels, and then a lower than standard placement improves highlight handling without making noise worse, and so could be a good thing. Yet the Dx0 measurement "punishes" that approach by declaring the true sensitivity to be less and shifting the noise and DR graphs to the right.

In an exteme case, a future sensor with column parallel ADC or even with ADC done right at ach photosite (such designs do exist) might have no reason to apply variable analogue gain at all, with the effect of each doubling of the EI setting being simply to double the shutter speed (at equal f-stop and lighting), halving the raw level placement of a mid-tone part of the scene, and then shift one bit to the left in conversion from raw to JPEG (or flagging this intent in raw files). In this scenario, DX0 would declare the "true ISO" to be the same regardless of the EI setting, and produce a graph with all measurements stacked vertically at the left!
Title: Re: DxO Mark interpretation and application
Post by: Ray on April 15, 2012, 01:10:57 pm
Ray,
    to address your new-found concern about T-stop variations:
Please reconsider my example with "equal T-stop" in place of "equal f-stop".


I have and it's not practical. There are no T-stop markings on any of my lenses.

 
Quote
This could be achieved by using the same lens in all tests via mount adaptors.

That's even less practical.

Quote
However, for all practical purposes, witu good prime lenses there will be negligible variation in transmission efficiency and thus a negligible discrepancy in T-stop at equal f-stop.

Most people use zoom lenses. My best zoom lens, the Nikkor 14-24/2.8 seems to have T-stops that differ quite significantly from the F/stops, according to my own comparisons with other lenses

 
Quote
Oh, and maybe change the EI values up a few stops, to say 6400 and 3200, because that what I say should be clear even with your examples of those unfortunate old-tech Canon sensors that need to use analogue amplification carefully in order to control noise sources later in the analogue signal path.

Didn't I mention? A number of Nikon cameras behave in a similar way, including the D3, D3s, D700. Also, if the graphs presented on this forum showing the DR of the new 5D3 are reasonably accurate, then the 5D3 has similar characteristics in this respect.

Quote
In that equal T-stop scenario, can you truly not see why people interested in low-light, high shutter speed performance want to compare images taken at equal shutter speed?

Of course I can, and that's precisely what I'm able to determine from the results that DXO already provide. I can't understand why you have a problem in this respect.
Consider a comparison between the IQ180 and the D800. The DXO Measurements tell us that at a base ISO of 74 the D800 can provide a DR of 14.33EV.

At a base ISO of 29, the IQ180 can provide a slightly lower DR of 13.56, which is still quite good, but at ISO 29 the shutter speed needs to be less than half what the D800 needs for an ETTR at its base ISO of 74.

If I want to determine the noise and DR at equal shutter speeds, the graphs provide a fairly accurate indication, plus or minus say 1/10th of a stop. To use the same shutter speed with the IQ180 as I would use with the D800 at its base ISO of 74, I would need to either increase the ISO of the IQ180 by 1 & 1/3 stops to reach approximately ISO 74, or underexpose 1 & 1/3rd stops at ISO 29. Either way, DR is going to be about 12.2 EV, significantly worse than that of the D800 at the same shutter speed.

However, SNR at 18% grey will be about the same at 45dB, for both cameras. Same for tonal Range and Color Sensitivity.What's the problem? All this is quite apparent from the existing graphs.


Title: Re: DxO Mark interpretation and application
Post by: BJL on April 15, 2012, 01:22:11 pm
Ray, we seem to have wandered off onto a different topic, and you for some reason keep quoting data about Canon sensors at lowish EI settings like 100, 200, and 400, even though I keep pointing out that my concern is comparisons of performance at high EI, like 1600 and up.

So let me ask again a simple question:

Why should I care about comparison of noise levels in low light situations where one camera is using a longer exposure time than the other while being at equal f-stop (or equal T-stop if you insist), regardless of whether either the camera maker or DX0 declare the cameras to be at the same ISO sensitivity?

I ask because the main thing that I care about in that situation is performance when I a m forced to use a high shutter speed by subject movement or such, and I will then need to use that _same_ shutter speed with any camera.
Title: Re: DxO Mark interpretation and application
Post by: BJL on April 15, 2012, 01:36:46 pm
Of course I can, and that's precisely what I'm able to determine from the results that DXO already provide. I can't understand why you have a problem in this respect.
Ray, your T-stop issue is a red herring: I was talking about professional testing of cameras such as that done by Dx0, where it is perfectly practical to use prime lenses for which f-stop is a thoroughly accurate measure of the intensity of illumination reaching the sensor.

Why on earth do you think that the Dx0 measurement based on highlight headroom in raw files is making the correct comparison even when it mismatches shutter speeds? As I indicated in my previous post, a camera with no significant noise introduced beyond the photosite could quite reasonably have the same Dx0 "ISO" rating at every setting, correcting levels for final display digitally instead of with analogue gain. More modestly, different cameras using the same sensor can choose to have different raw level placements with no significant affect on SNR or on noise levels in the final displayed images, so that the only difference is that the camera choosing the lower placement would have better handling of extreme highlights, and yet Dx0 would rate it as having worse noise, due to pushing its noise vs "ISO" curves to the left. (In fact, Olympus and Panasonic do seem to differ in their raw level placement when using the same sensors, and I believe that Pentax, Nikon and Sony also vary in their raw placements when using the same Sony sensors.)
Title: Re: DxO Mark interpretation and application
Post by: bjanes on April 15, 2012, 03:42:05 pm
Bill, that is basically right, except that Dx0 is clearly not using _sensor_ saturation, which would give a value independent of the EI setting or any amplification applied between photosites and ADC. Instead, Dx0 is using "ADC" saturation, requiring that the levels in the raw file are at a standard placement, with mid-tones about three stops below maximum ADC output level. The 2006 standard is perhaps ambiguous about whether it must be applied to maximum signal in photosites or can alsomapply to maximum signal after analogue amplification and ADC comversion but before any further levels adjustment in conversion to JPEG.

In an exteme case, a future sensor with column parallel ADC or even with ADC done right at ach photosite (such designs do exist) might have no reason to apply variable analogue gain at all, with the effect of each doubling of the EI setting being simply to double the shutter speed (at equal f-stop and lighting), halving the raw level placement of a mid-tone part of the scene, and then shift one bit to the left in conversion from raw to JPEG (or flagging this intent in raw files). In this scenario, DX0 would declare the "true ISO" to be the same regardless of the EI setting, and produce a graph with all measurements stacked vertically at the left!

You make a good point: the saturation standard applies only when the sensor reaches saturation at base ISO. The increased amplification used for ISOs above base do not increase the sensitivity of the sensor but merely scale the signal so that it occupies the full range of the ADC. I understand that some recent cameras (e.g. the Nikon D7000) have already reached the ISO less stage that you mention.

Regards,

Bill
Title: Could future raw values be essentially photo-electron counts, independent of EI?
Post by: BJL on April 15, 2012, 05:12:54 pm
Bill,
I just thought of another simpler way to state this: one ideal, that we might not be so far away from, is that the raw output could be essentially a count of the photo-electrons accumulated in each photosite. Subsequent transformations in the digital domain could take care of appropriate "tonal placement". Or perhaps (for mere convenience) the raw values could be modified from these electron counts by a suitable scaling factor, to reflect an intended tonal placement.
Title: Re: DxO Mark interpretation and application
Post by: Ray on April 17, 2012, 02:28:46 am
Ray, your T-stop issue is a red herring: I was talking about professional testing of cameras such as that done by Dx0, where it is perfectly practical to use prime lenses for which f-stop is a thoroughly accurate measure of the intensity of illumination reaching the sensor.

You seem to have misunderstood my point, BJL. Of course DXO is quite capable of either finding a prime lens with a T-stop rating similar to the F/stop, or making some adjustment to the measurements to compensate for any discrepancies. That's their business.

The issue is not that your idea cannot be implemented, but rather, how useful the data about sensor performance in relation to shutter speed might be for the camera-buying public who are mostly unaware of the T-stop values of their lenses?

DXO seem to be one of the few lens-review companies who test the transmission quality of lenses, if not the only one. For example, I can't find any T-stop values for my Nikkor 14-24/2.8. DXO haven't posted any test results yet for this lens, as they haven't for many other popular lenses.

However, during my own comparisons of this lens with the Sigma 15-30, both used on the Canon 5D with an adapter for the Nikkor lens, I was surprised to find some significant variation in shutter speed requirements, at the same focal length and F/stop, shooting the same scene with same lighting. As I recall, the differences sometimes seemed to be as great as 1 full stop. In other words, for an ETTR exposure on the 5D, I needed to use F5.6 with the Nikkor lens, at 24mm, but F8 on the Sigma lens, with the same shutter speed.

I wish I could find some verification for this observation, but I can find no T-stop results for either of these lenses on the internet.

However, searching the lens data base of DXOMark, I can find T-stop results for some of the lenses I already own and use, and the results are surprising.

For example, the Canon 24-105/F4 that I have used a lot, has a T-stop rating that differs by 0.8EV, at 24mm and full aperture. That's the equivalent of almost one full F/stop in exposure.

The Nikkor equivalent of this lens, the AF-S 24-120/F4 VR which I also use a lot, has T-stop values which are much closer to the F/stop ratings. The difference is only 0.2EV at 24mm.

In other words, when these two lenses are used at 24mm and the same f/stop, either on the same camera through use of an adaptor, or on the appropriate camera brand, both of which happen to have the same ISO sensitivity (let's assume), the Canon system will require almost 2/3rds of a stop more exposure to compensate for that additional transmission loss.

Now it may be the case that prime lenses tend to have less transmission loss, on average, than zoom lenses. But such transmission loss can still make the difference between something that would not be noticeable, and something that suddenly becomes noticeable through choice of lens.

DXO state that a difference in DR of less than 1/2 a stop is probably not going to be noticeable (depending on degree of pixel-peeping). However, a difference of 1/3rd of a stop in noise and DR, coupled with another difference of 1/3rd of a stop due to T-stop differences, could result in an over all 2/3rds of a stop noise difference at the same shutter speed, which could be noticeable and significant, depending on the nature of the scene and the degree of print enlargement.

Checking DXO's lens database, I notice there are a number of prime lenses with a T-stop rating that differs by 0.4EV. The Nikkor AF-S 24/1.4 differs by 0.6EV.

Why you think this is a red herring, beats me.  ;D

Another good reason for DXO not adopting your suggestion of relating sensor performance to shutter speed, is the fact that they have already tested hundreds of camera models using a consistent methodology and consistent format for the presentaion of the results, which allows for easy comparison between the latest Nikon model, say the D800, and the earliest Canon model, say the 6mp 10D.

Quote
Why on earth do you think that the Dx0 measurement based on highlight headroom in raw files is making the correct comparison even when it mismatches shutter speeds?


DXO doesn't even mention shutter speeds in its results. How can it mismatch them?

As regards the correctness of their comparisons, my confidence is based upon two factors.

(1) My own tests of sensor performance in different models of cameras, some of which were carried out before DXO published their test results, concur very closely with the DXO results.

(2) I've never seen any published tests from other parties which demonstrate inaccuracies in the DXO results. But I've seen lots of 'blah, blah, blah' which has not been backed up by any testing employing rigorous methodology.

I always remember the occasion in Bangkok when I compared the high-ISO performance of the Nikon D3 with my Canon 5D, before the DXOMark site existed. I couldn't hire or borrow a D3 because they were in short supply at the time, a bit like the current D800. But I found a camera shop that allowed me to use their demonstration model inside the shop, for comparison purposes.

The reason I took the trouble to do the comparison was because the claimed improvement in high-ISO performance of the D3, up to 2 stops better than any other existing DSLR, just seemed too good to be true. If they were true, I wanted the camera, since I often find myself in situations where flash and/or tripod are not appropriate.

However, I suspected that most reviewers of the D3 at the time, had been duped by the fact that the D3 had a couple of stops higher ISO settings than, for example the 5D. I recall statements in some reviews along the lines, 'We weren't able to compare performance at ISO 12,800 with the 5D because the 5D doesn't have such a setting'. I guess they just weren't smart enough.

My procedure was to shoot the same dark scene in the corner of the shop with both cameras on a tripod, using a number of different exposures varying by 1/3rd of a stop, underexposing the 5D at ISO 3200 to simulate ISO 6400 and 12800, and using the same aperture and focussing, and the same FL of lens, but different lenses.

Now, at the time I didn't even consider differences in T-stop ratings, didn't even think about them, but I did notice something which, at the time, I incorrectly attributed to differences in ISO sensitivies. The D3 seemed at about 1/3rd or perhaps even 1/2 a stop more sensitive. Looking now, at the DXO T-stop ratings for the two lenses that I used, the Nikkor 24-70/2.8 on the D3 and the Canon 24-105/F4 on the 5D, I see that the Nikkor lens has about 1/2 a stop lower T-stop rating than the Canon lens I used. (0.8Ev for the Canon as opposed to 0.3EV for the Nikkor).

That means that the Nikkor lens lets pass about 1/2 a stop more light than the Canon lens at any given exposure of the same scene at the same aperture.

The fact that I incorrectly attributed such differences to ISO sensitivity instead of T-stop differences, should not affect the results provided I took into consideration such differences during my comparisons, which I did.

In fact, according to DXO results, the ISO sensitivities of these two cameras is about the same, the D3 being only about 1/5th of a stop at the most, less sensitive. Offsetting the fact that the Canon glass was 1/2 a stop less sensitive (ie, more opaque), we get a nett effect of the D3 appearing to be about 1/3rd of a stop more sensitive.

Making adjustments for such apparent differences, I was able to determine that the D3 had a high-ISO advantage of the order of 1/3rd of a stop. Since 1/3rd of a stop was the smallest increment I could use, and since I was relying upon the impartiality of the ACR RAW converter I was using, it was impossible to be absolutely precise about such differences. With more precise testing using more sophisticated procedures and equipment, I would have accepted any differences of values between 1/4 of a stop and 1/2 a stop as being valid and consistent with my own testing.

Checking the DXOMark results which were published much later, and allowing for that slight difference of around 1/5th of a stop (or less) in ISO sensitivities, the DR of the D3 at ISO 3200 (actually 2512 as opposed to 2710 for the 5D) is shown as being about 0.4EV better, or slightly less than 1/2 a stop better.

This is one of the reasons why I have confidence in the accuracy of DXOMark test results. I've also carried out other comparisons of other cameras, which have also concurred with DXO results, such as comparing the DR of my D7000 with that of my 5D.

The crucial point when making such comparisons is that one must always start off from a comparison of equal exposures in terms of equal sensor saturation, such as ETTR, so that one can make allowances for any apparent differences in ISO sensitivities and/or T-stop differences.