Luminous Landscape Forum
Equipment & Techniques => Mirrorless Cameras => Topic started by: ErikKaffehr on October 22, 2015, 01:12:57 am
-
Hi,
CameraRentals have probably the most extensive lens testing equipment in the US, at least outside abbreviated organisations.
They started testing Sony FE mount lenses after some trepidations allowing them to mount them on the MTF test rig.
So, what are their findings? So far not so great: http://www.lensrentals.com/blog/2015/10/sony-e-mount-lens-sharpness-bench-tests
I don't have that much to add, except that LensRentals normally tests lenses at maximum aperture, AFAIK.
Does it contradict my experience? I don't know. It depends what to compare with, I don't have a ton of high performance lenses around. Roger Ciala halfway speculates that Sony may have some sharpening applied to raw data.
Initial test of the Sony 90/2.8G macro delivered very good data, but that was in camera testing. The MTF results are more like OK than excellent.
Best regards
Erik
-
VERY interesting. So, if we want to use Sony lenses with non-Sony cameras.... (!)
-
I read the whole piece, thanks for the link. Honestly, what does it mean really? That the lenses we use are not perfect? What is the value of testing a lens without a camera? We all know that today technology has made it possible to correct for optical defects via in-camera software. Even the new Loxia 21 lens and the Batis 25 lens have their distortion corrected in camera. So what? So what if Sony cooks some sharpening in the raw files in camera?
At the end of the day, its the end result that matters; for example, the Sony G 90 macro reports and reviews all say how good a lens it is. If a test bench downgrades that some, I know who to trust.
-
Indeed, Sony G 90 macro reports and reviews all say how good a lens it is. Now we can see what these ‘reviews’ are actually worth as we can clearly see that a Canon 100m f/2.8, a lens that’s already there for years, performs a lot better. Unfortunately there is no need for us to deny plain facts.
For me this is a real disappointment as with my next purchase I was already moving in the direction of a Sony a7rII with a few lenses. I was hoping for next-generation lenses to arrive, matching the step up to the 42 Mpix sensor. The CaNikon camp seems not so bad after all….
So what’s your opinion, were we expecting a bit too much from Sony/Zeiss?
-
The DxO findings are flawed because they test with different cameras of differing resolutions over time.
The LensRental tests are less that informative because they are done only with the lens at maximum aperture.
My take on this type of testing is to take it all with a large grain of salt. I test my own lenses with series of consistent and repeatable real-world images. Close, medium and far. All major f stops.
This doesn't take long and tells me what I need to know.
The 35mm f/1.4 FE moderately sucks at f/1.4. By f/2 is very good and from f/4 onward it's superlative. I therefore know that my copy is a keeper, but that I should never shoot with it wide open.
But to read LensRentals report would have me believe that this is a moderately poor lens. Not so in my experience.
Michael
-
It is wide open. Infinity. They have some additional glass in the optical path. While interesting I think you have to verify with on camera tests to draw any conclusions.
-
Indeed, Sony G 90 macro reports and reviews all say how good a lens it is. Now we can see what these ‘reviews’ are actually worth as we can clearly see that a Canon 100m f/2.8, a lens that’s already there for years, performs a lot better. Unfortunately there is no need for us to deny plain facts.
For me this is a real disappointment as with my next purchase I was already moving in the direction of a Sony a7rII with a few lenses. I was hoping for next-generation lenses to arrive, matching the step up to the 42 Mpix sensor. The CaNikon camp seems not so bad after all….
So what’s your opinion, were we expecting a bit too much from Sony/Zeiss?
I am not denying any facts. What I say is that I place much more trust in real world tests (see Michael's own comment above), than in tests that test the lenses wide open at infinity. Any photographer with a bit of experience knows that any lens will perform poorer wide open than stopped down a couple of stops.
IMO, we are expecting perfection, and that does not exist. I wonder how the USD 10k Leica M 50 would fare in such a test wide open... There is a limit to these sort of tests, really, after a while it borders on too much nerdiness. Really, if one can not take a good or very good photo with Sony and/or Zeiss glass in FE mount, or with Canon L glass, or Nikon quality glass, or whatever quality glass, than do not blame the glass.
I have used the Canon 100 IS macro in my DSLR days, a very good lens. From the reports I see, from real photos, the Sony 90 G is equally good.
-
The report tallies with my experience of the 55 1.8 which I feel is a very good lens, comparable to the Sigma Art 50 1.4 in resolution. I also have the 58 1.4 Nikon and don't see it as a relevant point of comparison, its a specialist Bokeh lens.
I'm also very pleased with the FE 35 2.8 which they didn't test.
Real world tests are always more important, but I wouldn't rubbish these tests just because they are critical of some of the lenses. Building those big heavy retrofocals often results in the kind of thing described here, its not too surprising.
From tests I've seen and comments I've heard from others, every one who has the 35 1.4 has been singing its praises, bench tests or not.
-
Hi,
Just to say, MTF testing is a the widely accepted method of lens testing. Zeiss uses it, Leitz uses it, Schneider uses it, Hasselblad uses and Sony uses it, too. Unfortunately it takes a instrumentation that costs a fortune to that.
For that reason, most MTF tests are done with lens mounted on camera. That has the advantage it can be done with very simple mean using a bit of tape (for instance) and some software like MTF mapper or Imatest.
Using MTF testing on the lens alone makes it possible to compare lens data between makes.
Now, all digital cameras have some optical assembly in front of the lens. This is often an IR-filter that used to be combined with two slices of birefringent crystals and a depolariser acting as an OLP filter. This assembly corresponds on Sony cameras to 2 mm of optical glass.
If the outlet pupil is near to the sensor, some of the beams passing trough the lens will pass that optical assembly at a large beam angle, shifting focus for those rays, a phenomena called astigmatism. With SLR lenses the outlet pupil is always distant from the sensor/film plane, so the sensor "glass" causes no problems. With mirrorless the outlet pupil can be close to the sensor plane, and this needs to be taken into account in the design of the lens. A good demo of this has been shown by 3D kraft, here (http://3dkraft.de/index.php?option=com_content&view=article&id=168:zeiss-loxia-2-35-short-comparison-review&catid=40:camerasandlenses&Itemid=2).
This is Zeiss Biogon 35/2 at full aperture:
(https://farm4.staticflickr.com/3893/15108807089_060a19069b_o.jpg)
And this is the Loxia 35/2 redesigned for the Sony A7 with cover glass taken into account.
(https://farm4.staticflickr.com/3898/15108950608_6e933ffa51_o.jpg)
This issue is also described in a series of articles on LensRentals:
http://www.lensrentals.com/blog/2014/06/sensor-stack-thickness-when-does-it-matter
http://www.lensrentals.com/blog/2014/06/the-glass-in-the-path-sensor-stacks-and-adapted-lenses
Now, as you may see, LensRentals is very enthusiastic about MTF, but what is that famous number. A good explanation is given by Hubert Nasse of Zeiss
here (http://lenspire.zeiss.com/sandbox/en/wp-content/uploads/sites/2/2015/09/cln30_en_web_special_mtf_011.pdf)
and here. (http://lenspire.zeiss.com/sandbox/en/wp-content/uploads/sites/2/2015/09/cln31_en_web_special_mtf_021.pdf)
This nice video may give some more insights about lenses: https://www.youtube.com/watch?v=9cnEnRADDLo&ab_channel=MattGranger
What is my own take on the Lensrentals test? Well, it is difficult to MTF test FE mount lenses as they are electronically controlled. Lens rentals managed to build a rig allowing to control the camera in an MTF rig, they have tested a large number of lenses as that is a part of their quality assurance program, they don't want to ship bad lenses to their customers. So they have a lot of experience in this. On the other hand, this thing is new and they may have missed something.
I do own the Sony 90/2.8G and it is very sharp. I don't own any of the other lenses mentioned.
Two points to make. For some reason Lensrentals tests mostly at full aperture, this makes a lot of sense for quality assurance but many lenses are seldom used at full aperture. Personally I prefer to have MTF data for full aperture and for the optimal aperture. In Europe MTF charts are often given at 10/20/40 lp/mm for full aperture and stopped down. Hasselblad, Zeiss, Leica publish data that way. Schneider and Rodenstock sometimes present data at 15/30/60 lp/mm which may be more adequate for digital.
Personally, I feel it is good to have different inputs. Checking out all information can be helpful in making educated buying decisions. I would also say that I would like to see more moderate aperture high performance lenses.
Best regards
Erik
It is wide open. Infinity. They have some additional glass in the optical path. While interesting I think you have to verify with on camera tests to draw any conclusions.
-
When the Sony FE 90/2.8 macro was introduced, there was much gloating by some people on Facebook and Youtube about how it "blows away" the Canon EF 100L IS Macro. That was pretty hard to believe then. Now we see that it wasn't true.
-
Hi,
Let's say it puts in perspective?!
BR
Erik
When the Sony FE 90/2.8 macro was introduced, there was much gloating by some people on Facebook and Youtube about how it "blows away" the Canon EF 100L IS Macro. That was pretty hard to believe then. Now we see that it wasn't true.
-
Hi,
Let's say it puts in perspective?!
BR
Erik
It does indeed. In fact, they are both excellent lenses, and the Canon may be a tiny bit better. I suspect the reason some people thought it "blows away" the Canon lens is because they were fooled by the DxO rankings, which depend on the sensor resolution. Apparently DxO has yet to test any lens on a 50mp Canon, so every lens they tested on a 36mp Sony or 36mp Nikon gets a boost in their rankings.
-
It does indeed. In fact, they are both excellent lenses, and the Canon may be a tiny bit better.
and in some applications/situations FE90/2.8 mounted on a body without any mirror slap and shutter shock with combined IBIS and OIS and held shooting through viewfinder simply does better vs the test on stand... in some of course not.
-
The thing which really interests me most about the LensRentals tests is that they test multiple copies and get some handle on sample to sample variation.
That one lens performs a bit better than another lens wide open at infinity isn't all that interesting- unless someone hits my exact shooting scenarios, the best it'll give me is an overall feeling for whether the lens is decent or not.
I won't really know whether or not I like it until I've shot with it a while- that's down to personal artistic choice. For example, I had a Canon 100 macro fifteen years ago. It was sharp as all heck, excellent for shooting closeups of bugs. But as a portrait lens I *hated* it, some combination of that sharpness and colour rendition and bokeh made it really, really ugly for people photography. Whereas the Panasonic 45 macro is really rather nice as a portrait lens.
Lens tests always score a lens for how well it resists flare- I'm much more interested in how gorgeous the flare looks when I provoke it, because I like flare and (especially) ghosting. I'm saving up for a nice anamorphic prime or two for stills as much as video, just to indulge my fondness :-)
But sample-to-sample variation tells you how well the lens is made. What are the manufacturing tolerances, how likely are you to get a good copy? If you get a dodgy one, what's the most likely to be wrong with it? That sort of information is very good to feed into buying decisions!
Cheers, Hywel
-
The LensRental tests are less that informative because they are done only with the lens at maximum aperture.
This has been repeated a few times in this thread but I don't believe it applies.
In typical photography and tests, scene-reflected light is assumed to arrive at the camera as a large wavefront perpendicular to the lens, which is then focused by the entire lens surface onto the sensing medium. Larger apertures in this case mean less diffraction but more contributions from more glass, which may mean more aberrations - so at low f-numbers, as aperture is increased performance tends to decrease as a result of the typically larger aberrations.
I have never seen or used an optical bench. However from the images of OLAF I have gleaned it looks to me that it shoots a laser-like beam of light to a single spot on the lens (as opposed to a wavefront) in order to determine something similar to the spot's related Point Spread Function, from which MTF and aberrations can be seen/derived. In other words it measure the performance of the lens one spot at a time, only at that one spot. If this is correct, aperture makes no difference whatsoever to aberrations (they are what they are at that one spot), although diffraction would still affect the results - so performing these tests at maximum aperture would provide the highest figures of merit for the lenses.
So I think Roger's optical bench tests are as objective and accurate as any especially because they are shot with the lens at maximum aperture. On the other hand he himself cautions about the fact that the lens is tested at infinity focus and performance may vary if the subject is closer than that.
Jack
-
However from the images of OLAF I have gleaned it looks to me that it shoots a laser-like beam of light to a single spot on the lens (as opposed to a wavefront) in order to determine something similar to the spot's related Point Spread Function, from which MTF and aberrations can be seen/derived. In other words it measure the performance of the lens one spot at a time, only at that one spot. If this is correct, aperture makes no difference whatsoever to aberrations (they are what they are at that one spot), although diffraction would still affect the results - so performing these tests at maximum aperture would provide the highest figures of merit for the lenses.
On the 35: 'The FE 35mm f/1.4 ZA lenses are all over the place. It actually is a bit worse than the graphs look because a lot of the variance is WITHIN a copy, not just copy-to-copy. None of the 10 copies we tested had even corners. And I'll editorialize and say that none of the dozens we've tested on Imatest had even corners either ..'
On the macro it's the opposite, a lot of center variation and not particularly worse in the corners. Again, significant copy-to-copy variation in overall sharpness, rather than any individual lenses having a bad corner.
Would you then agree, based on the results of this test, that his findings are both an accurate measure and representative of the QC issues he highlights ?
-
Would you then agree, based on the results of this test, that his findings are both an accurate measure and representative of the QC issues he highlights ?
Roger himself is not 100% sure about some of the results in this case because of the rigged setup, but I wholly agree that unless disproved it raises a lot of uncomfortable questions. Doctor Roger knows his lenses and equipment well.
As for me, being a purist, I would much prefer that Sony put the visual information into the raw file as captured - and perform pre-processing wizardry, if any, thereafter in the open. The advantage I see in performing distortion correction before writing raw data to the file is simply making life easier for third party converters, still... On the other hand I can see few unselfserving reasons to do pre-sharpening because sophisticated capture sharpening is very memory, processor and power hungry. Today it can always be done better in-computer than in-camera.
Jack
-
Roger himself is not 100% sure about some of the results in this case because of the rigged setup, but I wholly agree that unless disproved it raises a lot of uncomfortable questions. Doctor Roger knows his lenses and equipment well.
As for me, being a purist, I would much prefer that Sony put the visual information into the raw file as captured - and perform pre-processing wizardry, if any, thereafter in the open. The advantage I see in performing distortion correction before writing raw data to the file is simply making life easier for third party converters, still... On the other hand I can see few unselfserving reasons to do pre-sharpening because sophisticated capture sharpening is very memory, processor and power hungry. Today it can always be done better in-computer than in-camera.
Jack
Is there still some wizardry being done if you turn all the lens correction options off in the body? Distortion,CA & Vignetting corrections all have individual settings on my A7R2.
-
Indeed, Sony G 90 macro reports and reviews all say how good a lens it is. Now we can see what these ‘reviews’ are actually worth as we can clearly see that a Canon 100m f/2.8, a lens that’s already there for years, performs a lot better.
I might be missing something on how these MTF tests are done, as I am still trying to understand the comments by Jack Hogan, but for macro lenses, performance wide-open seems particularly irrelevant to typical real-world usage. So I think it could be useful to see the more traditional pair of MTF measurements: wide open and at some common f-stop for all lenses, say f/5.6. (An old standard was f/8, but with modern high resolution lenses and sensors, that could be too much affected by diffraction.)
P. S. I should have also said that infinity focus is a bit inappropriate for comparing macro lenses! (Yes, I know that they are used for other things too.)
-
Is there still some wizardry being done if you turn all the lens correction options off in the body? Distortion,CA & Vignetting corrections all have individual settings on my A7R2.
That's what Roger is implying explicitly. He says he is pretty sure about distortion corrections. He suspects sharpening.
-
IMO the value of Camerarentals' FE lens testing method lies in their detection of sample-to-sample variation free from whatever post-shot, pre-RAW-file-creation processing Sony may do. It's always good to know how consistent (or not) lens manufacturing is. But as to how any particular lens performs on their test rig…the rig isn't a Sony camera and what they're doing isn't photography. So ultimately you must take photos with a representative sample of a given lens to determine its worth.
-Dave-
-
Hi Jack,
I don't agree on this. An MTF measurement involves the whole surface of the lens and almost all lenses improve in MTF when stopped down. On a perfect system maximum MTF is gained at maximum aperture but most systems reach optimum MTF stopped down a bit.
A lens having high MTF will also have high MTF stopped down, well corrected is well corrected, after all. But a lens that is unusable at full aperture can be very good stopped down. Therefore, I would say the stopped down figure is the most important, unless shooting at full aperture.
It can be argued that if one buys an f/1.4 lens he/she pays for that lens to perform at f/1.4.
For me it is a bit interesting. The reason I bought the 90/2.8 G was that I wanted a lens around 85 mm that:
- That was very sharp at f/5.6
- That also was very sharp at maximum aperture
- Had very little axial chromatic aberration
Two lenses that obviously filled the bill were the Zeiss Otus 85/1.4 and the Zeiss 135/2 APO Sonnar, but the Otus was both to large and to expensive and the APO Sonnar is 135 mm, and I have found that 135 mm is something I seldom use. Other options were the Sony 90/2.8G and the Batis 85/1.8. The Sony 90/2.8G produced very good figures in Imatest based testing at Lensrentals and also very good results at DxO-mark. I am not a tester having access to both lenses, needed to buy one.
Now, I am quite happy with my 90/2.8G. It outresolves the A7rII sensor across the field at full aperture and it shows no or little green/magenta fringing on out of focus areas in the images I have shot this far. If this is due to software correction Sony writes very good image.
I include MTF plots for the Hasselblad Distagon 60/3.5 CF, the Hasselblad HC 50/3.5 II and the new Zeiss Loxia 21/2.8. All these are very fine lenses. Distagon 60/3.5 CF is an old design and improves significantly when stopped down. The HC 50/3.5 II is a very good performer even fully open while the new Loxia shows very little improvement when stopping down.
Best regards
Erik
This has been repeated a few times in this thread but I don't believe it applies.
In typical photography and tests, scene-reflected light is assumed to arrive at the camera as a large wavefront perpendicular to the lens, which is then focused by the entire lens surface onto the sensing medium. Larger apertures in this case mean less diffraction but more contributions from more glass, which may mean more aberrations - so at low f-numbers, as aperture is increased performance tends to decrease as a result of the typically larger aberrations.
I have never seen or used an optical bench. However from the images of OLAF I have gleaned it looks to me that it shoots a laser-like beam of light to a single spot on the lens (as opposed to a wavefront) in order to determine something similar to the spot's related Point Spread Function, from which MTF and aberrations can be seen/derived. In other words it measure the performance of the lens one spot at a time, only at that one spot. If this is correct, aperture makes no difference whatsoever to aberrations (they are what they are at that one spot), although diffraction would still affect the results - so performing these tests at maximum aperture would provide the highest figures of merit for the lenses.
So I think Roger's optical bench tests are as objective and accurate as any especially because they are shot with the lens at maximum aperture. On the other hand he himself cautions about the fact that the lens is tested at infinity focus and performance may vary if the subject is closer than that.
Jack
-
Hi Jack,
I don't agree on this. An MTF measurement involves the whole surface of the lens and almost all lenses improve in MTF when stopped down. On a perfect system maximum MTF is gained at maximum aperture but most systems reach optimum MTF stopped down a bit. A lens having high MTF will also have high MTF stopped down, well corrected is well corrected, after all. But a lens that is unusable at full aperture can be very good stopped down. Therefore, I would say the stopped down figure is the most important, unless shooting at full aperture.
Hi Erik,
I see your point and I agree that it would be useful to have values at other f-numbers for general photography purposes. Perhaps my previous post was a bit too quick off the post and I should have differentiated between checking manufacturing lens quality vs general photography usage. I can especially see how my statement that 'performing these tests at maximum aperture would provide the highest figures of merit for the lenses' could be misleading. For the former objective I still believe Roger's fully open approach gives the best results; but for the latter I agree that closing down the aperture may give more representative results for photographers at large.
My (limited) understanding of optical benches is that they do not measure the lens the way we use it (all at once). Instead, and simplifying for clarity, they map the 'smoothness/accuracy' of the surface of the effective lens tiny area by tiny area with a PSF 'microscope' and record its performance as they go along. The MTF curve that they publish is an average of several measurements at the same radial distance in the map (see Roger's quip about a lot of the variance being WITHIN a copy). Closing the aperture down will only make such measurements less accurate. This provides as good a lens-only manufacturing check as possible: they are way more accurate than what we can get by mounting the lens on a camera and trying to reverse out the effects of the sensor.
If an analogy helps, the difference between the two methods is similar to determining how well highly-polished satellite dishes are manufactured by measuring the signal at the output of the LNB - a fine method - versus shining a multitude of laser beams on each parabola and measuring reflected individual spot sizes and shapes near the focus point - a more laborious but more accurate overall method that also provides much more information about the quality of the build at various points on the parabola's surface. In the first case if the parabola's aperture is doubled the effective precision of the signal onto the focus point is invariably degraded because with fixed tolerances it is harder to hit a small target from further away; one could improve the effective precision by shielding the extra aperture and increasing exposure time. In the second case, however, if one wants to map the performance of the whole parabola, one might as well do it all the way to the edge of the aperture, no matter how large.
So with an optical bench I believe that one does want to measure MTF at the widest available aperture. Call it a lens manufacturing quality check. But photographers may be less interested in QC results than performance around their sweet-spot settings. Perhaps we could ask Roger to produce results at f/1.4, 2.8, 4, 5.6 and 8 for especially interesing lenses. I understand that it is quite a time consuming process, though, so this may not be a practical suggestion.
Jack