Ray, your T-stop issue is a red herring: I was talking about professional testing of cameras such as that done by Dx0, where it is perfectly practical to use prime lenses for which f-stop is a thoroughly accurate measure of the intensity of illumination reaching the sensor.
You seem to have misunderstood my point, BJL. Of course DXO is quite capable of either finding a prime lens with a T-stop rating similar to the F/stop, or making some adjustment to the measurements to compensate for any discrepancies. That's their business.
The issue is not that your idea cannot be implemented, but rather, how useful the data about sensor performance in relation to shutter speed might be for the camera-buying public who are mostly unaware of the T-stop values of their lenses?
DXO seem to be one of the few lens-review companies who test the transmission quality of lenses, if not the only one. For example, I can't find any T-stop values for my Nikkor 14-24/2.8. DXO haven't posted any test results yet for this lens, as they haven't for many other popular lenses.
However, during my own comparisons of this lens with the Sigma 15-30, both used on the Canon 5D with an adapter for the Nikkor lens, I was surprised to find some significant variation in shutter speed requirements, at the same focal length and F/stop, shooting the same scene with same lighting. As I recall, the differences sometimes seemed to be as great as 1 full stop. In other words, for an ETTR exposure on the 5D, I needed to use F5.6 with the Nikkor lens, at 24mm, but F8 on the Sigma lens, with the same shutter speed.
I wish I could find some verification for this observation, but I can find no T-stop results for either of these lenses on the internet.
However, searching the lens data base of DXOMark, I can find T-stop results for some of the lenses I already own and use, and the results are surprising.
For example, the Canon 24-105/F4 that I have used a lot, has a T-stop rating that differs by 0.8EV, at 24mm and full aperture. That's the equivalent of almost one full F/stop in exposure.
The Nikkor equivalent of this lens, the AF-S 24-120/F4 VR which I also use a lot, has T-stop values which are much closer to the F/stop ratings. The difference is only 0.2EV at 24mm.
In other words, when these two lenses are used at 24mm and the same f/stop, either on the same camera through use of an adaptor, or on the appropriate camera brand, both of which happen to have the same ISO sensitivity (let's assume), the Canon system will require almost 2/3rds of a stop more exposure to compensate for that additional transmission loss.
Now it may be the case that prime lenses tend to have less transmission loss, on average, than zoom lenses. But such transmission loss can still make the difference between something that would not be noticeable, and something that suddenly becomes noticeable through choice of lens.
DXO state that a difference in DR of less than 1/2 a stop is probably not going to be noticeable (depending on degree of pixel-peeping). However, a difference of 1/3rd of a stop in noise and DR,
coupled with another difference of 1/3rd of a stop due to T-stop differences, could result in an over all 2/3rds of a stop noise difference at the same shutter speed, which could be noticeable and significant, depending on the nature of the scene and the degree of print enlargement.
Checking DXO's lens database, I notice there are a number of prime lenses with a T-stop rating that differs by 0.4EV. The Nikkor AF-S 24/1.4 differs by 0.6EV.
Why you think this is a red herring, beats me.
Another good reason for DXO not adopting your suggestion of relating sensor performance to shutter speed, is the fact that they have already tested hundreds of camera models using a consistent methodology and consistent format for the presentaion of the results, which allows for easy comparison between the latest Nikon model, say the D800, and the earliest Canon model, say the 6mp 10D.
Why on earth do you think that the Dx0 measurement based on highlight headroom in raw files is making the correct comparison even when it mismatches shutter speeds?
DXO doesn't even mention shutter speeds in its results. How can it mismatch them?
As regards the correctness of their comparisons, my confidence is based upon two factors.
(1) My own tests of sensor performance in different models of cameras, some of which were carried out before DXO published their test results, concur very closely with the DXO results.
(2) I've never seen any published tests from other parties which demonstrate inaccuracies in the DXO results. But I've seen lots of 'blah, blah, blah' which has not been backed up by any testing employing rigorous methodology.
I always remember the occasion in Bangkok when I compared the high-ISO performance of the Nikon D3 with my Canon 5D, before the DXOMark site existed. I couldn't hire or borrow a D3 because they were in short supply at the time, a bit like the current D800. But I found a camera shop that allowed me to use their demonstration model inside the shop, for comparison purposes.
The reason I took the trouble to do the comparison was because the claimed improvement in high-ISO performance of the D3, up to 2 stops better than any other existing DSLR, just seemed too good to be true. If they were true, I wanted the camera, since I often find myself in situations where flash and/or tripod are not appropriate.
However, I suspected that most reviewers of the D3 at the time, had been duped by the fact that the D3 had a couple of stops higher ISO settings than, for example the 5D. I recall statements in some reviews along the lines, 'We weren't able to compare performance at ISO 12,800 with the 5D because the 5D doesn't have such a setting'. I guess they just weren't smart enough.
My procedure was to shoot the same dark scene in the corner of the shop with both cameras on a tripod, using a number of different exposures varying by 1/3rd of a stop, underexposing the 5D at ISO 3200 to simulate ISO 6400 and 12800, and using the same aperture and focussing, and the same FL of lens, but different lenses.
Now, at the time I didn't even consider differences in T-stop ratings, didn't even think about them, but I did notice something which, at the time, I incorrectly attributed to differences in ISO sensitivies. The D3 seemed at about 1/3rd or perhaps even 1/2 a stop more sensitive. Looking now, at the DXO T-stop ratings for the two lenses that I used, the Nikkor 24-70/2.8 on the D3 and the Canon 24-105/F4 on the 5D, I see that the Nikkor lens has about 1/2 a stop lower T-stop rating than the Canon lens I used. (0.8Ev for the Canon as opposed to 0.3EV for the Nikkor).
That means that the Nikkor lens lets pass about 1/2 a stop more light than the Canon lens at any given exposure of the same scene at the same aperture.
The fact that I incorrectly attributed such differences to ISO sensitivity instead of T-stop differences, should not affect the results provided I took into consideration such differences during my comparisons, which I did.
In fact, according to DXO results, the ISO sensitivities of these two cameras is about the same, the D3 being only about 1/5th of a stop at the most, less sensitive. Offsetting the fact that the Canon glass was 1/2 a stop less sensitive (ie, more opaque), we get a nett effect of the D3 appearing to be about 1/3rd of a stop more sensitive.
Making adjustments for such apparent differences, I was able to determine that the D3 had a high-ISO advantage of the order of 1/3rd of a stop. Since 1/3rd of a stop was the smallest increment I could use, and since I was relying upon the impartiality of the ACR RAW converter I was using, it was impossible to be absolutely precise about such differences. With more precise testing using more sophisticated procedures and equipment, I would have accepted any differences of values between 1/4 of a stop and 1/2 a stop as being valid and consistent with my own testing.
Checking the DXOMark results which were published much later, and allowing for that slight difference of around 1/5th of a stop (or less) in ISO sensitivities, the DR of the D3 at ISO 3200 (actually 2512 as opposed to 2710 for the 5D) is shown as being about 0.4EV better, or slightly less than 1/2 a stop better.
This is one of the reasons why I have confidence in the accuracy of DXOMark test results. I've also carried out other comparisons of other cameras, which have also concurred with DXO results, such as comparing the DR of my D7000 with that of my 5D.
The crucial point when making such comparisons is that one must always start off from a comparison of equal exposures in terms of equal sensor saturation, such as ETTR, so that one can make allowances for any apparent differences in ISO sensitivities and/or T-stop differences.
