DXO tests

[FMueller]

Seriously you guys...

+1

[Ken R]

Seriously you guys...

+2

Usual suspects always appear and post the same images over and over and try to hammer in their same viewpoints over and over and argue with the same people...over and over... This happens in all forums but in this one it seems it is only a handful of individuals...

Maybe there should be a camera engineering forum...

[Joe Towner]

LensRentals got a couple of really fancy camera testers - http://www.lensrentals.com/blog/2014/05/introducing-the-optical-bench

I would love to see a number of MF lenses pass through this, as it takes the sensor out of the testing process (unlike DXO which is dependent on the camera).  I think given a proper sample size, we all could learn a lot about the optics and how well certain lenses work.

[ErikKaffehr]

Hi,

I agree with what you say. Hasselblad and Rollei publishes MTF tests of all their lenses, Schneider and Rodenstock publishes MTF data, too.

Unfortunately, different firms use different presentation of data, so they can be hard to compare. Schneider often has 15, 30 and 60 lp/mm while Hasselblad uses 10, 20 and 40 lp/mm.

Best regards
Erik

LensRentals got a couple of really fancy camera testers - http://www.lensrentals.com/blog/2014/05/introducing-the-optical-bench

I would love to see a number of MF lenses pass through this, as it takes the sensor out of the testing process (unlike DXO which is dependent on the camera).  I think given a proper sample size, we all could learn a lot about the optics and how well certain lenses work.

[jerome_m]

Erik, for example: you use both a MF camera and a 24x36 A99. Are you really sure that you never noticed any other effect beyond increased dynamic range and lower noise?

That is an interesting question to which I have no answer. You sometimes made the point that the way I work I equalise things, and that is quite true. I normally shoot medium apertures, and for limited DoF I rather use a long telephoto on the DSLR where I can focus using magnified live view. I see some disadvantage in DR to MFD but I don't see it as very limiting. Colour rendition is pretty close after I made my own DNG profiles.

So: what about trying to shoot differently for a month or so? It will neither kill you nor break the camera if you use different apertures, let the exposure wander a bit to the left or drop the color chart. You may learn a thing or two.

[ErikKaffehr]

Hi,

You started a technical thread about DxO. I don't think my choice of subject or reluctance to intentional underexposure belong to the thread. You asked me some question, and I did answer those questions as correctly and politely as I could.

Best regards
Erik

So: what about trying to shoot differently for a month or so? It will neither kill you nor break the camera if you use different apertures, let the exposure wander a bit to the left or drop the color chart. You may learn a thing or two.

[jerome_m]

You started a technical thread about DxO. I don't think my choice of subject or reluctance to intentional underexposure belong to the thread. You asked me some question, and I did answer those questions as correctly and politely as I could.

Are you implying that my last question is incorrect or impolite?

[hjulenissen]

Photographically usable dynamic range in a real-world workflow matters a LOT to many types of photographers. dXo does not test for this. They test DR in an engineering sense.

Flash card manufacturers tests write speeds in an engineering sense (lets hope that it is not marketing guys). Thankfully, we photographers do not live in an alternate dimension, and the results of those engineering measurements can be directly related to concepts that matters to many photographers.

The idea that art and engineering live in unconnected spheres seems counterproductive to both art and engineering. It is also ignorant of the brilliant people and work that has resulted somewhere in-between.

The effect of raw processing (quite important given the manufacturer software often gets the best result out of their file), aesthetics, tonal smoothness, quality (not just quantity) and look of grain/noise, and color accuracy in deep shadow transitions are all ignored.

Many photographers use a 3rd party raw processor that will be remain constant*) no matter what camera they use.

Image quality cannot be totally described by 3 or 4 parameters. Thus, there may probably be visible differences (for some scenes) even for two cameras with similar dxo-type ratings. It seems that camera sensors are in some ways a lot simpler than some people make them out to be, especially if one leaves out the subjectivity present in raw development. Camera sensors (to my knowledge) does not have less accurate colors in the shadows. Raw converters may, if that is chosen by the implementer, possibly to work around some limitation in the sensor.

That will not change my opinion that the best way to see how a camera handles in the real world, is to shoot it in the real world, and not in a lab setting.

Of course. I would not spend $100.000 (or my career) on a piece of equipment without spending some time to know if the investement was good. That includes actually using the thing extensively.

Thing is, I have a finite amount of time and a finite ability to do precise comparisions. If there is a flaw in a camera that will only show up once every 10000 images, chances are that I won't figure out until I make the purchase. If someone are willing to do some relevant preliminary testing of a large number of cameras, I will try to comprehend their results and incorporate their conclusions into my own process. Sticking my fingers in my ears (eyes?) and singing "lala" does seem ignorant. Offering well thought-out critique of DXO (and everything else) is a good thing, the kind of thing that has helped us progress.

-h

[ErikKaffehr]

Hi Jerome,

No, I don't imply that your question was impolite. What I said that I tried to answer your question (about my perception of the P45+ vs. the Sony) as accurately and politely as I could. I put some effort in both accuracy (finding and checking references) and wording.

On the other hand, I would also say that suggesting that I would spend time to different type shooting than I do, is quite a bit irrelevant on a thread discussing DxO tests, which incidentally was started by yourself.

Best regards
Erik

Are you implying that my last question is incorrect or impolite?

[jerome_m]

On the other hand, I would also say that suggesting that I would spend time to different type shooting than I do, is quite a bit irrelevant on a thread discussing DxO tests, which incidentally was started by yourself.

I did not suggest that you change your practice for ever, I only suggested that as a test.

As to relevance, I can say the following: when I started this thread, I argued that some differences due to sensor size are not tested by DXO, because of the particular assumptions they made for their test. The way you work just happens to correspond to these assumptions and, consequently, you do not see differences that are equalised by both your practice and the DXO test suite. I suggested that you try a different practice a few times to be able to see these differences.

As a side note, the reason why these discussions go nowhere is that you and a few others present that particular way of measuring and testing as an absolute metric. It is not. I, for example, have little use for dynamic range (it is generally sufficient for me in modern cameras), shadow noise level or colour fidelity. I rarely have use for optical sharpness, as measured by flat resolution charts at 1-4m distance. I may have use for optical sharpness, as measured at infinite distance (and I presented some tests of mine for that).

For example, I have use for rendering of out of focus subjects, homogeneity of the rendering across the field, consistency of colour under different illuminants and plenty of other factors which are generally absent from DXO tests.

[ErikKaffehr]

Hi Jerome,

DxO lens test show homogenity of rendering across the field, and they publish SMI for two illuminants, so they actually cover that ground quite a bit.

They don't show bokeh, the Photozone tests cover that area a bit more. DxO doesn't evaluate sensor colour reproduction i spiky spectra.

I will do some more lens testing as I added another Zeiss lens to my Hasselblad kit, the Planar 100/3.5 which is a famously sharp lens. Plan to do something similar to your shots. As a side note: i have posted sample images from all my Hasselblad (V-series) lenses here: http://echophoto.dnsalias.net/ekr/Articles/Shoots/BernardSamples/

The screenshots below are from DxO reports on one of the Hasselbalds (50 MP back) and comparing Zeiss 24-70/2.8 and Sigma 24-105/4 Art. I am a bit interested in buying the Sigma 24-105/4 as the Zeiss 24-70/2.8 is lacking a bit at the long end.

Best regards
Erik

For example, I have use for rendering of out of focus subjects, homogeneity of the rendering across the field, consistency of colour under different illuminants and plenty of other factors which are generally absent from DXO tests.

[jerome_m]

DxO lens test show homogeneity of rendering across the field

That data would only be useful for me if I reproduced paintings. I am not interested in homogeneity a 1-4m, which is what you get from test charts. I am interested about much larger distances, and the results are different. I am also interested in homogeneity of the bokeh, which they do not test at all (e.g. swirly bokeh or cateye effect).

They publish SMI for two illuminants, so they actually cover that ground quite a bit.

Not at all. All cameras work reasonably well under CIE-D50 and CIE-A. I would like to know what happens under non-standard yet common illuminants: overcast daylight, fluorescent or led lightning for example. I know the results will be poor for fluorescent, but I also know that some camera fare much better than others.

I am a bit interested in buying the Sigma 24-105/4 as the Zeiss 24-70/2.8 is lacking a bit at the long end.

In practical use, I find the SAL-2470F28Z to be an excellent lens at 70mm. I am more irritated by its propensity to show two well delimited unsharp triangles in the lowest corners when photographying people standing on grass at 24mm. Note that what irritates me is that these triangles are well delimited: the Nikon 24-70 has corners which are just as poor, but the change is more gradual. That is less noticeable.

[Ken R]

Two things regarding Dxo (and other reviewers/testers):

#1: Tests/Reviews should disclose at which focus distance the lenses were tested. Preferably it would be nice to have tests that cover at least two points, one at infinity and other at a medium distance typical or one akin to what that particular lens most common use is. (One reason why Lens Rentals have invested in several lens testing systems is to do just that)

#2: The DxO sensor testing data and overall rating number give a really good idea of how good a sensor is compared to others specially when the differences are considerable. A few points here and there are almost imperceptible in real world use and when comparing two sensors or cameras with close enough rating I would not use the rating as the determining factor in choosing between them. But having used a lot of cameras that are in the DxO database of tests and looking at how DxO has rated them and compared their real world performance by using them I somewhat agree with the DxO ratings. Not saying the ratings tell the whole story but they are valuable. Specially for comparing the dynamic range of cameras. Color is much more difficult since it is extremely dependent on processing and a bit more subjective. Again, IMHO.

[ErikKaffehr]

Hi,

Some good points from Ken R!

Here are two articles from Tim Ashly who used to have a very nice blog on the equipment he uses, and that was a wide selection of stuff he has owned.

The first one is on the DxO-mark in general:

http://tashley1.zenfolio.com/blog/2013/3/whats-your-doh-mark-score

The other one on the lens tests:

http://tashley1.zenfolio.com/blog/2013/3/dxo-mark-lenses-on-d800---my-real-world-response

With regard to the lens tests he makes several very good points. I would like to comment on two of them field curvature and focus shift.

Field curvature varies with focusing distance. A good practice is to test lenses at 50 times focal length. A large target is needed for that. Modern lenses often have floating elements (or groups) that compensate field curvature at different distances.

Field curvature would show up in the DxO test as loss of sharpness across the field, but it cannot be told apart from other aberration from the benchmark data.

Field curvature is interesting, because a lens with curved field can have very good sharpness at the point of focus but still perform bad on flat targets. The Zeiss macro Planar 120/4 (Hasselblad version) is a good example of that. See MTF data below, pretty ugly at infinity but really good at close up, which is it's intended range. But, would it be focused at a point of axis at infinity that point would still be in good sharpness, but centrum would be out focus. Stopping down to say f/11 DoF hides curvature and the lens looks pretty good.

Zeiss did produce a more complex design for the Contax, Apo Macro Planar 120/4, that lens had a floating group keeping field curvature at bay. For the Hasselblad V, Zeiss has three different Planar lenses. For long distance work Hasselblad recommends the 100/3.5 while for short distance they recommend the Macro Planar 120/4. They say that if the subject area is larger than 1 sqm (square meter) use the 100/3.5 else the 120/4, at least for reproduction type photography.

The other issue is focus shift, and generally a problem with large aperture lenses. When stopping down the point of focus moves. So if focusing is at full aperture slight misfocus will arise when stopped down. Photozone has a very good test of this effect.

Best regards
Erik

[voidshatter]

I've written many times about my technical issues with DXO. They very carefully (and I trust, honestly and accurately) measure a set of numbers which are of marginal use to most photographers.

I am the minority of photographers (not pro working in the studio) so I can see how the DXO scores matter in my real world usage for pixel peeping. For this I agree with Bill.

http://www.getdpi.com/forum/medium-format-systems-digital-backs/52844-backlight-landscape-photography-realized-say-no-silhouette.html

[Theodoros]

I am the minority of photographers (not pro working in the studio) so I can see how the DXO scores matter in my real world usage for pixel peeping. For this I agree with Bill.

http://www.getdpi.com/forum/medium-format-systems-digital-backs/52844-backlight-landscape-photography-realized-say-no-silhouette.html

Why would you disagree with this http://nikonrumors.com/2014/12/03/the-best-nikon-cameras-and-lenses-according-to-senscore-and-lenscore.aspx/#more-84608 (that disagrees with DXO) then?

[ErikKaffehr]

Hi,

Your posting illustrates the weak points of the MFDBs quite clearly. Doug himself posted a comparison of the IQ-250, IQ-260 and the IQ-280 which showed a great advantage of the IQ-250 over the IQ-260/IQ-280 in shadow detail.

Very clearly, DxO measures just some aspects of performance, but I would suggest that what they measure, they measure well.

Little doubt, when (or if) DxO tests the IQ-250 it will go to the top. The sensor size would give it an advantage of about 10 points on the DxO-mark scale.

Best regards
Erik

I am the minority of photographers (not pro working in the studio) so I can see how the DXO scores matter in my real world usage for pixel peeping. For this I agree with Bill.

http://www.getdpi.com/forum/medium-format-systems-digital-backs/52844-backlight-landscape-photography-realized-say-no-silhouette.html

[voidshatter]

Why would you disagree with this http://nikonrumors.com/2014/12/03/the-best-nikon-cameras-and-lenses-according-to-senscore-and-lenscore.aspx/#more-84608 (that disagrees with DXO) then?

I do not disagree with this. This does not contradict dxomark. They just have different weights from different aspects. If you shoot wedding where you are have limited light condition with a bound of shutter speed then of course senscore makes more sense than dxomark overall score.

[Theodoros]

Soooo, they're both right! ....no? ...and therefore so are other "weights", ...right?

[ErikKaffehr]

Hi,

According to the information at Senscore the major difference to DxO-mark that DxO-mark is to a large extent based on base ISO, with "low light" part taking high ISO noise into account, while the Senscore is covering a large ISO range.

On the other hand, DxO actually publishes data for all ISOs and even includes a few MFD systems.

Best regards
Erik

I do not disagree with this. This does not contradict dxomark. They just have different weights from different aspects. If you shoot wedding where you are have limited light condition with a bound of shutter speed then of course senscore makes more sense than dxomark overall score.