Some interesting findings on Sony FE-mount lenses by Camerarentals

[ErikKaffehr]

Hi,

CameraRentals have probably the most extensive lens testing equipment in the US, at least outside abbreviated organisations.

They started testing Sony FE mount lenses after some trepidations allowing them to mount them on the MTF test rig.

So, what are their findings? So far not so great: http://www.lensrentals.com/blog/2015/10/sony-e-mount-lens-sharpness-bench-tests

I don't have that much to add, except that LensRentals normally tests lenses at maximum aperture, AFAIK.

Does it contradict my experience? I don't know. It depends what to compare with, I don't have a ton of high performance lenses around. Roger Ciala halfway speculates that Sony may have some sharpening applied to raw data.

Initial test of the Sony 90/2.8G macro delivered very good data, but that was in camera testing. The MTF results are more like OK than excellent.

Best regards
Erik

[tnargs]

VERY interesting. So, if we want to use Sony lenses with non-Sony cameras.... (!)

[Paulo Bizarro]

I read the whole piece, thanks for the link. Honestly, what does it mean really? That the lenses we use are not perfect? What is the value of testing a lens without a camera? We all know that today technology has made it possible to correct for optical defects via in-camera software. Even the new Loxia 21 lens and the Batis 25 lens have their distortion corrected in camera. So what? So what if Sony cooks some sharpening in the raw files in camera?

At the end of the day, its the end result that matters; for example, the Sony G 90 macro reports and reviews all say how good a lens it is. If a test bench downgrades that some, I know who to trust.

[JaapD]

Indeed, Sony G 90 macro reports and reviews all say how good a lens it is. Now we can see what these ‘reviews’ are actually worth as we can clearly see that a Canon 100m f/2.8, a lens that’s already there for years, performs a lot better. Unfortunately there is no need for us to deny plain facts.

For me this is a real disappointment as with my next purchase I was already moving in the direction of a Sony a7rII with a few lenses. I was hoping for next-generation lenses to arrive, matching the step up to the 42 Mpix sensor. The CaNikon camp seems not so bad after all….

So what’s your opinion, were we expecting a bit too much from Sony/Zeiss?

[michael]

The DxO findings are flawed because they test with different cameras of differing resolutions over time.

The LensRental tests are less that informative because they are done only with the lens at maximum aperture.

My take on this type of testing is to take it all with a large grain of salt. I test my own lenses with series of consistent and repeatable real-world images. Close, medium and far. All major f stops.

This doesn't take long and tells me what I need to know.

The 35mm f/1.4 FE moderately sucks at f/1.4. By f/2 is very good and from f/4 onward it's superlative. I therefore know that my copy is a keeper, but that I should never shoot with it wide open.

But to read LensRentals report would have me believe that this is a moderately poor lens. Not so in my experience.

Michael

[Torbjörn Tapani]

It is wide open. Infinity. They have some additional glass in the optical path. While interesting I think you have to verify with on camera tests to draw any conclusions.

[Paulo Bizarro]

Indeed, Sony G 90 macro reports and reviews all say how good a lens it is. Now we can see what these ‘reviews’ are actually worth as we can clearly see that a Canon 100m f/2.8, a lens that’s already there for years, performs a lot better. Unfortunately there is no need for us to deny plain facts.

For me this is a real disappointment as with my next purchase I was already moving in the direction of a Sony a7rII with a few lenses. I was hoping for next-generation lenses to arrive, matching the step up to the 42 Mpix sensor. The CaNikon camp seems not so bad after all….

So what’s your opinion, were we expecting a bit too much from Sony/Zeiss?

I am not denying any facts. What I say is that I place much more trust in real world tests (see Michael's own comment above), than in tests that test the lenses wide open at infinity. Any photographer with a bit of experience knows that any lens will perform poorer wide open than stopped down a couple of stops.

IMO, we are expecting perfection, and that does not exist. I wonder how the USD 10k Leica M 50 would fare in such a test wide open... There is a limit to these sort of tests, really, after a while it borders on too much nerdiness. Really, if one can not take a good or very good photo with Sony and/or Zeiss glass in FE mount, or with Canon L glass, or Nikon quality glass, or whatever quality glass, than do not blame the glass.

I have used the Canon 100 IS macro in my DSLR days, a very good lens. From the reports I see, from real photos, the Sony 90 G is equally good.

[free1000]

The report tallies with my experience of the 55 1.8 which I feel is a very good lens, comparable to the Sigma Art 50 1.4 in resolution.  I also have the 58 1.4 Nikon and don't see it as a relevant point of comparison, its a specialist Bokeh lens.

I'm also very pleased with the FE 35 2.8 which they didn't test.

Real world tests are always more important, but I wouldn't rubbish these tests just because they are critical of some of the lenses.  Building those big heavy retrofocals often results in the kind of thing described here, its not too surprising.

From tests I've seen and comments I've heard from others, every one who has the 35 1.4 has been singing its praises, bench tests or not.   

 

[ErikKaffehr]

Hi,

Just to say, MTF testing is a the widely accepted method of lens testing. Zeiss uses it, Leitz uses it, Schneider uses it, Hasselblad uses and Sony uses it, too. Unfortunately it takes a instrumentation that costs a fortune to that.

For that reason, most MTF tests are done with lens mounted on camera. That has the advantage it can be done with very simple mean using a bit of tape (for instance) and some software like MTF mapper or Imatest.

Using MTF testing on the lens alone makes it possible to compare lens data between makes.

Now, all digital cameras have some optical assembly in front of the lens. This is often an IR-filter that used to be combined with two slices of birefringent crystals and a depolariser acting as an OLP filter. This assembly corresponds on Sony cameras to 2 mm of optical glass.

If the outlet pupil is near to the sensor, some of the beams passing trough the lens will pass that optical assembly at a large beam angle, shifting focus for those rays, a phenomena called astigmatism. With SLR lenses the outlet pupil is always distant from the sensor/film plane, so the sensor "glass" causes no problems. With mirrorless the outlet pupil can be close to the sensor plane, and this needs to be taken into account in the design of the lens. A good demo of this has been shown by 3D kraft, here.

This is Zeiss Biogon 35/2 at full aperture:


And this is the Loxia 35/2 redesigned for the Sony A7 with cover glass taken into account.


This issue is also described in a series of articles on LensRentals:
http://www.lensrentals.com/blog/2014/06/sensor-stack-thickness-when-does-it-matter
http://www.lensrentals.com/blog/2014/06/the-glass-in-the-path-sensor-stacks-and-adapted-lenses

Now, as you may see, LensRentals is very enthusiastic about MTF, but what is that famous number. A good explanation is given by Hubert Nasse of Zeiss
here
and here.

This nice video may give some more insights about lenses: https://www.youtube.com/watch?v=9cnEnRADDLo&ab_channel=MattGranger

What is my own take on the Lensrentals test? Well, it is difficult to MTF test FE mount lenses as they are electronically controlled. Lens rentals managed to build a rig allowing to control the camera in an MTF rig, they have tested a large number of lenses as that is a part of their quality assurance program, they don't want to ship bad lenses to their customers. So they have a lot of experience in this. On the other hand, this thing is new and they may have missed something.

I do own the Sony 90/2.8G and it is very sharp. I don't own any of the other lenses mentioned.

Two points to make. For some reason Lensrentals tests mostly at full aperture, this makes a lot of sense for quality assurance but many lenses are seldom used at full aperture. Personally I prefer to have MTF data for full aperture and for the optimal aperture. In Europe MTF charts are often given at 10/20/40 lp/mm for full aperture and stopped down. Hasselblad, Zeiss, Leica publish data that way. Schneider and Rodenstock sometimes present data at 15/30/60 lp/mm which may be more adequate for digital.

Personally, I feel it is good to have different inputs. Checking out all information can be helpful in making educated buying decisions. I would also say that I would like to see more moderate aperture high performance lenses.

Best regards
Erik

It is wide open. Infinity. They have some additional glass in the optical path. While interesting I think you have to verify with on camera tests to draw any conclusions.

[zlatko-b]

When the Sony FE 90/2.8 macro was introduced, there was much gloating by some people on Facebook and Youtube about how it "blows away" the Canon EF 100L IS Macro.  That was pretty hard to believe then.  Now we see that it wasn't true.

[ErikKaffehr]

Hi,

Let's say it puts in perspective?!

BR
Erik

When the Sony FE 90/2.8 macro was introduced, there was much gloating by some people on Facebook and Youtube about how it "blows away" the Canon EF 100L IS Macro.  That was pretty hard to believe then.  Now we see that it wasn't true.

[zlatko-b]

Hi,

Let's say it puts in perspective?!

BR
Erik

It does indeed.  In fact, they are both excellent lenses, and the Canon may be a tiny bit better.  I suspect the reason some people thought it "blows away" the Canon lens is because they were fooled by the DxO rankings, which depend on the sensor resolution.  Apparently DxO has yet to test any lens on a 50mp Canon, so every lens they tested on a 36mp Sony or 36mp Nikon gets a boost in their rankings.

[AlterEgo]

It does indeed.  In fact, they are both excellent lenses, and the Canon may be a tiny bit better.

and in some applications/situations FE90/2.8 mounted on a body without any mirror slap and shutter shock with combined IBIS and OIS and held shooting through viewfinder simply does better vs the test on stand... in some of course not.

[Hywel]

The thing which really interests me most about the LensRentals tests is that they test multiple copies and get some handle on sample to sample variation.

That one lens performs a bit better than another lens wide open at infinity isn't all that interesting- unless someone hits my exact shooting scenarios, the best it'll give me is an overall feeling for whether the lens is decent or not.

I won't really know whether or not I like it until I've shot with it a while- that's down to personal artistic choice. For example, I had a Canon 100 macro fifteen years ago. It was sharp as all heck, excellent for shooting closeups of bugs. But as a portrait lens I *hated* it, some combination of that sharpness and colour rendition and bokeh made it really, really ugly for people photography. Whereas the Panasonic 45 macro is really rather nice as a portrait lens.

Lens tests always score a lens for how well it resists flare- I'm much more interested in how gorgeous the flare looks when I provoke it, because I like flare and (especially) ghosting. I'm saving up for a nice anamorphic prime or two for stills as much as video, just to indulge my fondness :-)

But sample-to-sample variation tells you how well the lens is made. What are the manufacturing tolerances, how likely are you to get a good copy? If you get a dodgy one, what's the most likely to be wrong with it? That sort of information is very good to feed into buying decisions!

Cheers, Hywel

[Jack Hogan]

The LensRental tests are less that informative because they are done only with the lens at maximum aperture.

This has been repeated a few times in this thread but I don't believe it applies.

In typical photography and tests, scene-reflected light is assumed to arrive at the camera as a large wavefront perpendicular to the lens, which is then focused by the entire lens surface onto the sensing medium.  Larger apertures in this case mean less diffraction but more contributions from more glass, which may mean more aberrations - so at low f-numbers, as aperture is increased performance tends to decrease as a result of the typically larger aberrations.

I have never seen or used an optical bench.  However from the images of OLAF I have gleaned it looks to me that it shoots a laser-like beam of light to a single spot on the lens (as opposed to a wavefront) in order to determine something similar to the spot's related Point Spread Function, from which MTF and aberrations can be seen/derived.  In other words it measure the performance of the lens one spot at a time, only at that one spot.  If this is correct, aperture makes no difference whatsoever to aberrations (they are what they are at that one spot), although diffraction would still affect the results - so performing these tests at maximum aperture would provide the highest figures of merit for the lenses.

So I think Roger's optical bench tests are as objective and accurate as any especially because they are shot with the lens at maximum aperture.  On the other hand he himself cautions about the fact that the lens is tested at infinity focus and performance may vary if the subject is closer than that.

Jack

[Manoli]

However from the images of OLAF I have gleaned it looks to me that it shoots a laser-like beam of light to a single spot on the lens (as opposed to a wavefront) in order to determine something similar to the spot's related Point Spread Function, from which MTF and aberrations can be seen/derived.  In other words it measure the performance of the lens one spot at a time, only at that one spot.  If this is correct, aperture makes no difference whatsoever to aberrations (they are what they are at that one spot), although diffraction would still affect the results - so performing these tests at maximum aperture would provide the highest figures of merit for the lenses.

On the 35: 'The FE 35mm f/1.4 ZA lenses are all over the place. It actually is a bit worse than the graphs look because a lot of the variance is WITHIN a copy, not just copy-to-copy. None of the 10 copies we tested had even corners. And I'll editorialize and say that none of the dozens we've tested on Imatest had even corners either ..'

On the macro it's the opposite, a lot of center variation and not particularly worse in the corners. Again, significant copy-to-copy variation in overall sharpness, rather than any individual lenses having a bad corner.

Would you then agree, based on the results of this test, that his findings are both an accurate measure and representative of the QC issues he highlights ?

[Jack Hogan]

Would you then agree, based on the results of this test, that his findings are both an accurate measure and representative of the QC issues he highlights ?

Roger himself is not 100% sure about some of the results in this case because of the rigged setup, but I wholly agree that unless disproved it raises a lot of uncomfortable questions.  Doctor Roger knows his lenses and equipment well. 

As for me, being a purist, I would much prefer that Sony put the visual information into the raw file as captured - and perform pre-processing wizardry, if any, thereafter in the open.  The advantage I see in performing distortion correction before writing raw data to the file is simply making life easier for third party converters, still... On the other hand I can see few unselfserving reasons to do pre-sharpening because sophisticated capture sharpening is very memory, processor and power hungry.  Today it can always be done better in-computer than in-camera.

Jack

[Dr Tone]

Roger himself is not 100% sure about some of the results in this case because of the rigged setup, but I wholly agree that unless disproved it raises a lot of uncomfortable questions.  Doctor Roger knows his lenses and equipment well. 

As for me, being a purist, I would much prefer that Sony put the visual information into the raw file as captured - and perform pre-processing wizardry, if any, thereafter in the open.  The advantage I see in performing distortion correction before writing raw data to the file is simply making life easier for third party converters, still... On the other hand I can see few unselfserving reasons to do pre-sharpening because sophisticated capture sharpening is very memory, processor and power hungry.  Today it can always be done better in-computer than in-camera.

Jack

Is there still some wizardry being done if you turn all the lens correction options off in the body?  Distortion,CA & Vignetting corrections all have individual settings on my A7R2.

[BJL]

Indeed, Sony G 90 macro reports and reviews all say how good a lens it is. Now we can see what these ‘reviews’ are actually worth as we can clearly see that a Canon 100m f/2.8, a lens that’s already there for years, performs a lot better.

I might be missing something on how these MTF tests are done, as I am still trying to understand the comments by Jack Hogan, but for macro lenses, performance wide-open seems particularly irrelevant to typical real-world usage.  So I think it could be useful to see the more traditional pair of MTF measurements: wide open and at some common f-stop for all lenses, say f/5.6.  (An old standard was f/8, but with modern high resolution lenses and sensors, that could be too much affected by diffraction.)


P. S. I should have also said that infinity focus is a bit inappropriate for comparing macro lenses!  (Yes, I know that they are used for other things too.)

[Jack Hogan]

Is there still some wizardry being done if you turn all the lens correction options off in the body?  Distortion,CA & Vignetting corrections all have individual settings on my A7R2.

That's what Roger is implying explicitly.  He says he is pretty sure about distortion corrections.  He suspects sharpening.