Pages: [1]   Go Down

Author Topic: Lens Testing  (Read 6785 times)

BruceHouston

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 308
Lens Testing
« on: April 25, 2008, 05:12:34 pm »

I am an advanced amateur wishing to test my own lenses (Canon EF-S 10-22mm; EF 50mm f/1.4; EF 24-105mm f/4L IS; and EF 70-200mm f/4L IS) quantitatively.

Is IMATEST the most cost-effective solution?  It seems a bit pricey at $200.00 just to run tests on four lenses.  And that is the "light" version that locks out certain types of testing and multiple simultaneous runs.  I suppose that I will spring for the license at this price if there is nothing else out there that will read an image file and provide quantitative MTF data.  (I will buy and grumble about having to spend so much to test my paltry four lenses.)

Any thoughts are very much appreciated.

Thanks,
Bruce
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Lens Testing
« Reply #1 on: April 25, 2008, 06:01:01 pm »

Hi,

Imatest is a fine tool. The light version is perfectly good for lens testing.

If Imatest is good for your needs depends, well, on your needs. If you want to compare with other tests it may not be easy, because you don't know the premises the other tests are using. If you want to understand the performance of your lenses Imatest is good.

Best regards
Erik

Quote
I am an advanced amateur wishing to test my own lenses (Canon EF-S 10-22mm; EF 50mm f/1.4; EF 24-105mm f/4L IS; and EF 70-200mm f/4L IS) quantitatively.

Is IMATEST the most cost-effective solution?  It seems a bit pricey at $200.00 just to run tests on four lenses.  And that is the "light" version that locks out certain types of testing and multiple simultaneous runs.  I suppose that I will spring for the license at this price if there is nothing else out there that will read an image file and provide quantitative MTF data.  (I will buy and grumble about having to spend so much to test my paltry four lenses.)

Any thoughts are very much appreciated.

Thanks,
Bruce
[a href=\"index.php?act=findpost&pid=191926\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

BruceHouston

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 308
Lens Testing
« Reply #2 on: April 25, 2008, 06:28:36 pm »

I am interested in IMATEST precisely because it provides a quantitative result that could be compared against other like test results for MTF.  Granted, the various test methodologies and test equipment (not to mention the variability between lens "copies" (read: serial numbers)) are likely to yield a range of results.  But at least there are numbers to compare, to determine whether one's lens is "way out there" as a good or bad anomaly.

The "newsprint on the wall" test, on the other hand, is completely subjective and yields no numbers to use in comparisons.

I just wondered if there is anything else out there competitive to IMATEST, or whether IMATEST is the "only game in town."  Possibly the latter, given the price.

Thanks,
Bruce
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Lens Testing
« Reply #3 on: April 25, 2008, 10:18:12 pm »

Quote
The "newsprint on the wall" test, on the other hand, is completely subjective and yields no numbers to use in comparisons.
[a href=\"index.php?act=findpost&pid=191940\"][{POST_SNAPBACK}][/a]


But you could probably 'objectify' the results sufficiently for your own purposes by designing your own newsprint test chart consisting of different size letters.

The legibility of text is a very good guide to lens sharpness. Even when you know what the word is, it's still easy to tell the difference between clearly legible, almost legible and completely illegible.

As far as I know, the Imatest results would be valid only for a specific camera/lens combination. Change the camera model and the results no longer apply. It's not really a lens test but a 'system' test, just as the newsprint test is.

With your own newsprint chart, all that's required is that you fill the frame with the chart, whatever the lens, the camera and the aperture, which means changing the distance to the chart according to the focal length of the lens..
Logged

BruceHouston

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 308
Lens Testing
« Reply #4 on: April 25, 2008, 11:48:47 pm »

Good points, Ray.  I agree that the method that you suggest would objectify the testing for purposes of comparing my lenses operating on my 40D to each other.  And, I agree that MTF values that I obtain using IMATEST on shots from my lens/40D combination will not necessarily correlate with published MTF values, given the "systemic" nature of the test.

Still, I am curious about how my MTF numbers compare to published values for a particular lens.  For example, if I see an MTF curve that, compared to published curves indicates that one of my lenses is substantially sub-par, I will know to send that lens to Canon service.

Best regards,
Bruce
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Lens Testing
« Reply #5 on: April 26, 2008, 12:13:55 am »

Quote
Still, I am curious about how my MTF numbers compare to published values for a particular lens.  For example, if I see an MTF curve that, compared to published curves indicates that one of my lenses is substantially sub-par, I will know to send that lens to Canon service.
[a href=\"index.php?act=findpost&pid=191975\"][{POST_SNAPBACK}][/a]

Bruce,
It's not clear to me if lens sharpness can be improved by Canon service. When I sent my 50/1.4 in for calibration on the grounds it wasn't as sharp as my el cheapo 50/1.8, I never mentioned it had a focussing problem. But they adjusted it for focussing nevertheless.

It's now really bad at autofocussing. But it's still just as sharp when manually focussed. There seems to be enormous confusion around this issue.
Logged

BruceHouston

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 308
Lens Testing
« Reply #6 on: April 26, 2008, 12:28:01 am »

Oh, boy; I am sorry to hear that.

I am re-entering photography as a hobby after having been absent for about 25 years.  I am simply trying to make sure that I have the best moderate-cost equipment (L-lenses) and that it is in good operating order at the outset.  Having spent several thousand dollars one would think that this should not be so difficult, but it is turning out to be more challenging than I had imagined.

On the bright side, perhaps the effort will make me more appreciative of those shart prints down the road...

Bruce
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Lens Testing
« Reply #7 on: April 26, 2008, 02:06:55 am »

Quote
Oh, boy; I am sorry to hear that.

I am re-entering photography as a hobby after having been absent for about 25 years.  I am simply trying to make sure that I have the best moderate-cost equipment (L-lenses) and that it is in good operating order at the outset.  Having spent several thousand dollars one would think that this should not be so difficult, but it is turning out to be more challenging than I had imagined.

On the bright side, perhaps the effort will make me more appreciative of those shart prints down the road...

Bruce
[a href=\"index.php?act=findpost&pid=191985\"][{POST_SNAPBACK}][/a]

Well, you probably know my views on this. I'm prepared to pay for someone to provide a full set of MTF charts for any lens I buy. I want to know what I'm paying for.

A few days ago, I visited some friends on the Gold Coast (Australia). They actually pay some one to groom, wash and shave their poodle every couple of weeks.

I'm a lens buyer and I can't pay someone to test and grade my lens before I buy it?

Ridiculous!
Logged

NashvilleMike

  • Guest
Lens Testing
« Reply #8 on: April 27, 2008, 07:01:06 pm »

Quote
I am interested in IMATEST precisely because it provides a quantitative result that could be compared against other like test results for MTF.  Granted, the various test methodologies and test equipment (not to mention the variability between lens "copies" (read: serial numbers)) are likely to yield a range of results.  But at least there are numbers to compare, to determine whether one's lens is "way out there" as a good or bad anomaly.

I'm afraid I'm going to bring some more bad news to your kitchen table!

Basically there is no even remotely feasible way to do MTF testing yourself in a manner that will let you compare against "other like test results" or manufacturers graphs, or even those on other sites. To do it, you'd need to invest in a test lab and use extremely precise controls and the exact same procedures and methods that the manufacturer/other site does.

- What makes this difficult is you have to realize that MTF testing isn't just a software package that you throw up and generate numbers to match what Nikon or Canon or Zeiss say they are - there are many different methods, different wavelengths, different test targets - essentially you really have to become a popular photography grade testing lab.

Then there's a whole laundry list of other issues that come to play, such as variability - for instance, if you buy Imatest, what are you going to do when you run it a couple of times, perhaps on different days, re-setting up your environment, and you get different results? I've actually tried Imatest - and got different results even on multiple passes of the same test target - showing me that a very scientific, careful, statistically controlled type of approach would be neccessary to do it, and I came up with the conclusion that it wasn't even remotely near worth the time to do it, and then again, the final numbers could only be useful in comparing in MY environment on THAT day - no way you could reconfigure your environment the next day and compare numbers from that, and no way you could compare numbers against anyone elses tests. Not worth it, and basically almost impossible for most people to do.

So: let's go back to your premise for WHY you want to do quantitative analysis on the lenses - I'm going to guess that you ultimately don't want to get stuck with one that, well, sucks. That's fair enough. It's the quantitative part that is near impossible. But that doesn't mean we can't do tests that tell us what we basically need to know. I certainly do lens testing - quite thorough tests actually, involving targets that accurately represent the two things I photograph the most a: studio work with strobe (90%+) and b: landscape at infinity in natural light, and I even run the tests 3 different times over 2 bodies to rule out my error and any variability I may have introduced.

My theory on the type of tests that WE can do, while not purely scientific in perhaps the manner you would like, certainly will let us know if the lens we purchased has problems, is based on this:

Problems with lenses typically manifest themselves in the following areas.

a ) lens element decentering - where the end result is the lens is sharper on one side more than the other

b ) lens autofocus calibration issues - the back or front focus that we've all heard about

c)  excessive dirt or junk in the lens (a tiny bit is normal, but at times you'll read of one lens that is just filthy)

Items 'a' and 'b' are by far the most common - I'd almost argue that item 'b' is the most common. What I've seen very, very, very little of is actual sharpness deviations in a lens that is properly functioning in terms of items 'a' and 'b' above being fine. I own some 'duplicate' copies of lenses (even though I'm a Nikon shooter) and while there have been calibration differences and at times slight decentering differences, there never has been a marked sharpness difference between the lenses. So thinking this through, I came to the conclusion that I needed to isolate out whether the lenses I've purchased exhibit either 'a' or 'b' above - and beyond that, I had to trust that if none of the typical items/defects were found that I was okay - since there is no way I can become a full fledged testing lab versus that of a professional full fledged testing lab.

And in practice, I've found out this works well. I run resolution target tests at studio distances with strobe (all tripod mounted, live-view carefully focused using sensor based contrast detection autofocus if the body has it, etc, etc) and then I repeat a series of tests outdoor with a subject that lets me evaluate side to side and corner to corner sharpness. In the end, I've identified some cases of 'b' in my own lenses, and for focus calibration issues I've found it easier to simply send them in for warranty service and they come back perfect.

Is this perfect? No - I'd love to be able to spend 30 minutes running a lens I buy and seeing if it matches up as close as possible with the manufacturers standard, but most assuredly there is simply no easy, moderate, or even moderately difficult way of doing this - in this fast food/ATM society we all want perfect answers now, instantly, but in this case it's not possible. So my testing approach is what I do, and has weeded out some issues, which were then dealt with. After that, the lenses go into "production" so to speak, where I evaluate them again - I should note it's been far more than once that I've found lens characteristics that are discovered in "real shooting" far more so than in lens tests, and then if I go out and take pictures and not worry about the pixel peeping any further.

So my wordy thesis here can be distilled down to: your original goal isn't realistically achievable, so review what you're trying to do (discover if a lens you aquired is a stinker) and rework your tests to cover the basics - get 80% of what you need tested and leave the other 20% on the table. Unless you're a millionaire and can afford your own full blown controlled testing environment for your four lenses of course.

-m
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Lens Testing
« Reply #9 on: April 28, 2008, 12:07:54 am »

Hi!

There are qute a few things Imatest can tell you:

1) It may be possible to compare your lenses, should I use the 16-24 or the 24-80 at 24mm?

2 ) You can find out the best aperture for the lens.

3) Yo can quantify decentering.

4) It calculates distorsion figures.

5) It measures chromatic aberration.

6) I can help a lot in understanding how your camera and lenses work together.

Lens testing is error prone and needs to be done with some care.

Best regards
Erik

Quote
I'm afraid I'm going to bring some more bad news to your kitchen table!

Basically there is no even remotely feasible way to do MTF testing yourself in a manner that will let you compare against "other like test results" or manufacturers graphs, or even those on other sites. To do it, you'd need to invest in a test lab and use extremely precise controls and the exact same procedures and methods that the manufacturer/other site does.

- What makes this difficult is you have to realize that MTF testing isn't just a software package that you throw up and generate numbers to match what Nikon or Canon or Zeiss say they are - there are many different methods, different wavelengths, different test targets - essentially you really have to become a popular photography grade testing lab.

Then there's a whole laundry list of other issues that come to play, such as variability - for instance, if you buy Imatest, what are you going to do when you run it a couple of times, perhaps on different days, re-setting up your environment, and you get different results? I've actually tried Imatest - and got different results even on multiple passes of the same test target - showing me that a very scientific, careful, statistically controlled type of approach would be neccessary to do it, and I came up with the conclusion that it wasn't even remotely near worth the time to do it, and then again, the final numbers could only be useful in comparing in MY environment on THAT day - no way you could reconfigure your environment the next day and compare numbers from that, and no way you could compare numbers against anyone elses tests. Not worth it, and basically almost impossible for most people to do.

So: let's go back to your premise for WHY you want to do quantitative analysis on the lenses - I'm going to guess that you ultimately don't want to get stuck with one that, well, sucks. That's fair enough. It's the quantitative part that is near impossible. But that doesn't mean we can't do tests that tell us what we basically need to know. I certainly do lens testing - quite thorough tests actually, involving targets that accurately represent the two things I photograph the most a: studio work with strobe (90%+) and b: landscape at infinity in natural light, and I even run the tests 3 different times over 2 bodies to rule out my error and any variability I may have introduced.

My theory on the type of tests that WE can do, while not purely scientific in perhaps the manner you would like, certainly will let us know if the lens we purchased has problems, is based on this:

Problems with lenses typically manifest themselves in the following areas.

a ) lens element decentering - where the end result is the lens is sharper on one side more than the other

b ) lens autofocus calibration issues - the back or front focus that we've all heard about

c)  excessive dirt or junk in the lens (a tiny bit is normal, but at times you'll read of one lens that is just filthy)

Items 'a' and 'b' are by far the most common - I'd almost argue that item 'b' is the most common. What I've seen very, very, very little of is actual sharpness deviations in a lens that is properly functioning in terms of items 'a' and 'b' above being fine. I own some 'duplicate' copies of lenses (even though I'm a Nikon shooter) and while there have been calibration differences and at times slight decentering differences, there never has been a marked sharpness difference between the lenses. So thinking this through, I came to the conclusion that I needed to isolate out whether the lenses I've purchased exhibit either 'a' or 'b' above - and beyond that, I had to trust that if none of the typical items/defects were found that I was okay - since there is no way I can become a full fledged testing lab versus that of a professional full fledged testing lab.

And in practice, I've found out this works well. I run resolution target tests at studio distances with strobe (all tripod mounted, live-view carefully focused using sensor based contrast detection autofocus if the body has it, etc, etc) and then I repeat a series of tests outdoor with a subject that lets me evaluate side to side and corner to corner sharpness. In the end, I've identified some cases of 'b' in my own lenses, and for focus calibration issues I've found it easier to simply send them in for warranty service and they come back perfect.

Is this perfect? No - I'd love to be able to spend 30 minutes running a lens I buy and seeing if it matches up as close as possible with the manufacturers standard, but most assuredly there is simply no easy, moderate, or even moderately difficult way of doing this - in this fast food/ATM society we all want perfect answers now, instantly, but in this case it's not possible. So my testing approach is what I do, and has weeded out some issues, which were then dealt with. After that, the lenses go into "production" so to speak, where I evaluate them again - I should note it's been far more than once that I've found lens characteristics that are discovered in "real shooting" far more so than in lens tests, and then if I go out and take pictures and not worry about the pixel peeping any further.

So my wordy thesis here can be distilled down to: your original goal isn't realistically achievable, so review what you're trying to do (discover if a lens you aquired is a stinker) and rework your tests to cover the basics - get 80% of what you need tested and leave the other 20% on the table. Unless you're a millionaire and can afford your own full blown controlled testing environment for your four lenses of course.

-m
[a href=\"index.php?act=findpost&pid=192190\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Lens Testing
« Reply #10 on: April 28, 2008, 12:18:13 am »

Hi!

If you buy a new lens and it is under-performing Canon (or whoever) should give you a new lens. Simple as that!

Regarding autofocus, it's a bit like this. Autofocus has limited precision. There are a few issues:

1) The resolution of the autofocus sensors is probably limited.
2) The autofocus sensor and the camera sensor must be exactly aligned, an impressive feat considering that light path to the sensor includes at least one probably two movable mirrors.
3) Autofocusing is essentially done in camera not in the lens. I don't really understand how you can calibrate a lens. In all servo systems there is a small "residual error", that error may be reduced with "calibration".

Best regards
Erik


Quote
Bruce,
It's not clear to me if lens sharpness can be improved by Canon service. When I sent my 50/1.4 in for calibration on the grounds it wasn't as sharp as my el cheapo 50/1.8, I never mentioned it had a focussing problem. But they adjusted it for focussing nevertheless.

It's now really bad at autofocussing. But it's still just as sharp when manually focussed. There seems to be enormous confusion around this issue.
[a href=\"index.php?act=findpost&pid=191983\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

NashvilleMike

  • Guest
Lens Testing
« Reply #11 on: April 28, 2008, 02:01:12 am »

Quote
Hi!

There are qute a few things Imatest can tell you:

1) It may be possible to compare your lenses, should I use the 16-24 or the 24-80 at 24mm?

2 ) You can find out the best aperture for the lens.

3) Yo can quantify decentering.

4) It calculates distorsion figures.

5) It measures chromatic aberration.

6) I can help a lot in understanding how your camera and lenses work together.

Lens testing is error prone and needs to be done with some care.

Best regards
Erik
[a href=\"index.php?act=findpost&pid=192228\"][{POST_SNAPBACK}][/a]

Oh, I totally agree it can be useful - but frankly, I can determine with my own tests if I've got decentering, CA, or which lens I should use at 24mm. Imatest might be fun for some - but in the end, I didn't see any reasonable need for it for my own evaluations. And given the OP was thinking he could use it to provide MTF graphs he could compare to the manufacturer, I had to mention that it would most certainly fail in this regard. But others might find Imatest useful even if I really don't.

I'm far more of a subjective lens evaluation type of guy anyway - experience has taught me that for sure. Others may have different opinions.

-m
Logged

BruceHouston

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 308
Lens Testing
« Reply #12 on: April 29, 2008, 03:07:21 am »

Quote
I'm afraid I'm going to bring some more bad news to your kitchen table!

Basically there is no even remotely feasible way to do MTF testing yourself in a manner that will let you compare against "other like test results" or manufacturers graphs, or even those on other sites. To do it, you'd need to invest in a test lab and use extremely precise controls and the exact same procedures and methods that the manufacturer/other site does.

- What makes this difficult is you have to realize that MTF testing isn't just a software package that you throw up and generate numbers to match what Nikon or Canon or Zeiss say they are - there are many different methods, different wavelengths, different test targets - essentially you really have to become a popular photography grade testing lab.

Then there's a whole laundry list of other issues that come to play, such as variability - for instance, if you buy Imatest, what are you going to do when you run it a couple of times, perhaps on different days, re-setting up your environment, and you get different results? I've actually tried Imatest - and got different results even on multiple passes of the same test target - showing me that a very scientific, careful, statistically controlled type of approach would be neccessary to do it, and I came up with the conclusion that it wasn't even remotely near worth the time to do it, and then again, the final numbers could only be useful in comparing in MY environment on THAT day - no way you could reconfigure your environment the next day and compare numbers from that, and no way you could compare numbers against anyone elses tests. Not worth it, and basically almost impossible for most people to do.

So: let's go back to your premise for WHY you want to do quantitative analysis on the lenses - I'm going to guess that you ultimately don't want to get stuck with one that, well, sucks. That's fair enough. It's the quantitative part that is near impossible. But that doesn't mean we can't do tests that tell us what we basically need to know. I certainly do lens testing - quite thorough tests actually, involving targets that accurately represent the two things I photograph the most a: studio work with strobe (90%+) and b: landscape at infinity in natural light, and I even run the tests 3 different times over 2 bodies to rule out my error and any variability I may have introduced.

My theory on the type of tests that WE can do, while not purely scientific in perhaps the manner you would like, certainly will let us know if the lens we purchased has problems, is based on this:

Problems with lenses typically manifest themselves in the following areas.

a ) lens element decentering - where the end result is the lens is sharper on one side more than the other

b ) lens autofocus calibration issues - the back or front focus that we've all heard about

c)  excessive dirt or junk in the lens (a tiny bit is normal, but at times you'll read of one lens that is just filthy)

Items 'a' and 'b' are by far the most common - I'd almost argue that item 'b' is the most common. What I've seen very, very, very little of is actual sharpness deviations in a lens that is properly functioning in terms of items 'a' and 'b' above being fine. I own some 'duplicate' copies of lenses (even though I'm a Nikon shooter) and while there have been calibration differences and at times slight decentering differences, there never has been a marked sharpness difference between the lenses. So thinking this through, I came to the conclusion that I needed to isolate out whether the lenses I've purchased exhibit either 'a' or 'b' above - and beyond that, I had to trust that if none of the typical items/defects were found that I was okay - since there is no way I can become a full fledged testing lab versus that of a professional full fledged testing lab.

And in practice, I've found out this works well. I run resolution target tests at studio distances with strobe (all tripod mounted, live-view carefully focused using sensor based contrast detection autofocus if the body has it, etc, etc) and then I repeat a series of tests outdoor with a subject that lets me evaluate side to side and corner to corner sharpness. In the end, I've identified some cases of 'b' in my own lenses, and for focus calibration issues I've found it easier to simply send them in for warranty service and they come back perfect.

Is this perfect? No - I'd love to be able to spend 30 minutes running a lens I buy and seeing if it matches up as close as possible with the manufacturers standard, but most assuredly there is simply no easy, moderate, or even moderately difficult way of doing this - in this fast food/ATM society we all want perfect answers now, instantly, but in this case it's not possible. So my testing approach is what I do, and has weeded out some issues, which were then dealt with. After that, the lenses go into "production" so to speak, where I evaluate them again - I should note it's been far more than once that I've found lens characteristics that are discovered in "real shooting" far more so than in lens tests, and then if I go out and take pictures and not worry about the pixel peeping any further.

So my wordy thesis here can be distilled down to: your original goal isn't realistically achievable, so review what you're trying to do (discover if a lens you aquired is a stinker) and rework your tests to cover the basics - get 80% of what you need tested and leave the other 20% on the table. Unless you're a millionaire and can afford your own full blown controlled testing environment for your four lenses of course.

-m
[a href=\"index.php?act=findpost&pid=192190\"][{POST_SNAPBACK}][/a]

Mike,

I emailed Klaus Schroiff, principal of the German testing website "PhotoZone," with test results that I obtained from my EF 24-105 f/4L with 40D body using Imatest.  It is uncanny how closely his kind response matched your comments, above.  This is disappointing, of course, but appears to represent reality, given the tight setup tolerances required to achieve repeatability of the MTF tests.

Thank you for taking the time to describe your experience and the practical methodology that you settled on.

Bruce
Logged
Pages: [1]   Go Up