Comparing IQ of two identical lenses?

[PDeXplore]

So I have two copies of a Sigma 17-50 f/2.8 for my Nikon D3300. I wanted to compare the two so I printed out this chart (https://digital-photography-school.com/wp-content/uploads/2015/01/test-chart.pdf) and set my camera up on a tripod. I tested each lens at 17mm, going from f/2.8, 7.1, 11, 13, and 22. I set the camera close enough to fill the frame and used autofocus, then the lcd to double check focus. I repeated with the other lens, and then repeated the whole test at 50mm as well.

What I want to know is if this is a reasonable test to gauge which lens to keep. I also got some really interesting results that I don't quite understand. One of this biggest oddities was the light transmission. On one lens, the camera was consistently exposing for shorter exposures at the same aperture. The most extreme examples were at f/22, one lens was consistently taking exposures multiple seconds longer (as much as an extra 10 seconds at 50mm). Viewing the histograms in Lightroom, the exposures are fairly similar. To confirm the exposures weren't just 'different', the one lens had many exposures that were brighter, and shorter shutters. What could cause this? And yes, this was taken indoors under a light, so that shouldn't have changed from shot to shot.

I'm also verifying that nothing could've been wrong with my focus. In each test I only focused on the first test at f/2.8, and then left that alone as I changed my aperture. My first lens was very sharp through the image at f/2.8, but then got increasingly soft after f/7.1. My other lens was horribly soft in the lower right at f/2.8, but then impressively sharp throughout the whole image from f/7.1 onwards. Are these just fairly standard variances to expect from lens to lens, or should I consider repeating with different conditions or tests?

[smthopr]

It seems to me, that a practical test would investigate:

Which lens provides the best resolution, using auto focus. (as this may be how you usually use the lens)

The plane of focus, and whether one lens focuses more evenly side to side or up to down or corner vs corner. (I've got a 24mm lens that is sharper on one side than the other)

It's also possible that individual camera bodies could effect these tests.

Also note lens apertures are not perfectly accurate.  + or - 1/3 stop might well be within the range of the lens specification. I would check each lens wide open and compare exposures and you may well find that the two lenses match exposures when wide open.

Lastly, since you have zoom lenses, check each lens at different focal lengths.  One might be best at 20mm and the other at 35mm for example.

[FabienP]

Bruce is right, the decentering of zoom lenses will vary at different focal lengths (FL) and almost no zoom can be expected to be perfectly centered at all FLs. This is like winning the lottery.

The problem with using test charts at short distance is that any misalignement of the camera related to the chart will create the impression of decentering. Also, this could mask other problems that only occur with focus set at infinity. For those reasons, I prefer to test lenses at near infinity, with the added advantage that it matches my primary usage for landscape photography. Luckily, there are nearby landscape elements which can be used to assess sharpness in the corners at most FLs. Just turn the camera upside down to assess centering.

The behaviour of the second tested lens looks fishy and could prevent the use of the lens wide open. Could the softness of the first lens be explained by diffraction? This seems to kick in too early, even for an APS-C lens. Difficult to tell you more without seeing images.

Cheers,

Fabien