MTF Charts vs. Photographs

[wolfnowl]

Mike Johnston at The Online Photographer has an interesting article on equipment (lens) testing, here:

http://theonlinephotographer.typepad.com/t...s-and-grap.html

[Panopeeper]

This is the typical rumbling of the digitally challenged, who is offended by something he does not understand. The proof is in the pudding; my cat looks much better than your dog, so my lens is better than yours.

Example: does anyone problem with adding vignetting to an image? I dont. However, I do have lots of problems with vignetting, which I don't want to have, but is there; thus the vignetting characteristics of a lens are very important for me - not based on the picture of a nice child. Btw, the correct term is optical vignetting, beside light fall-off.

[spidermike]

This is the typical rumbling of the digitally challenged, who is offended by something he does not understand. The proof is in the pudding; my cat looks much better than your dog, so my lens is better than yours.

Example: does anyone problem with adding vignetting to an image? I dont. However, I do have lots of problems with vignetting, which I don't want to have, but is there; thus the vignetting characteristics of a lens are very important for me - not based on the picture of a nice child. Btw, the correct term is optical vignetting, beside light fall-off.
[a href=\"index.php?act=findpost&pid=217142\"][{POST_SNAPBACK}][/a]

Maybe reading reviews is as subjective as looking at photographs    I took his column to mean that specifications is one thing but at the end of the day the art of photography is about the final result. Sure, if some highly technical photography is required then charts may be useful in making your final decision on which lens to choose; but to look at it from the other direction and say a photograph is poor because of the way the lens measures is crass.
 
I have a (dwindling) interest in hifi and the same arguments happen there (but much worse). Some show preference for gear based on on technical measurements and others purely on the sound. And at the end of the day if it sound right who cares how it measures?

[Ralph Wagner]

If I was hung up on MTF charts, edge sharpness, vignetting, soft wide open, noise at high ISO's, etc., etc. I would have given up on photography a long time ago. But all this stuff is for 'pixel peepers' and equipment perfectionists. Nothing wrong there. More power to them. I like my stuff aside from all the technical, microscopic examinations, and if you don't, that's okay too.

[CJL]

Example: does anyone problem with adding vignetting to an image? I dont. However, I do have lots of problems with vignetting, which I don't want to have, but is there; thus the vignetting characteristics of a lens are very important for me - not based on the picture of a nice child.

Vignetting (as used in the example photo of the child) seems to be most often used to help improve poor composition.  In most cases I don't mind it, but I do want the option of being able to capture an image without vignetting (or falloff) if I choose to do so.

It's a lot like sharpness... I realize it's just a crutch for the artistically challenged, but *I* want to be the one to decide when a photo is to be sharp or soft, not the equipment manufacturer.

[KevinA]

I couldn't give a dam about Lab tests, it works or it doesn't for me. No doubt the pixel peepers could argue I might as well shoot with a nokia as I am always shooting from aircraft and the vibration kills any resolution advantage of more pixels or better lens. Then again I know my smkIII gives me better files than my smkII or Kodak SLR/n and better also than when I subcontract work which comes back from various other Nikon or Canons. In over 30 years of pro shooting I've never felt the need to shoot brick walls or newspapers, it's the balance of day to day real shooting in different conditions that counts , not the hypothetical max you might get with everything nailed down in your favour.

Kevin.

[Pete Ferling]

I couldn't give a dam about Lab tests, it works or it doesn't for me. ...
Kevin.
[a href=\"index.php?act=findpost&pid=217534\"][{POST_SNAPBACK}][/a]

Same here.  Get the shot.  Not many have the choice of more expensive, or the right gear, etc.  If something interesting is happening now, do your best with what's at hand and get the shot.  I have many interesting shots that were not done with the best tool, ideal conditions, etc.  However the image itself is interesting and sells the message well.

Even from a technical standpoint, the world is not a pristine lab with balanced lights at 45 degree angles and objects facing at perfect parrallel planes.  Screw it.  Not every lens needs a red ring to be capable of making good shots.  Putting on my engineering hat, a resultant test of a chart only proves that a given lens was better or worse for a particular setup, at a given temperature, humidy, etc. etc. etc.  So, yes, the charts do prove some basis to considering purchasing.  But in real life, lens' can vary from one to another, and unless you have that very lens used in the test, and know it wasn't knocked around or dropped, you'll still have to do the only thing real to know for sure.  Buy it and use it.


Life rarely happens twice, just shoot.

[Ray]

I couldn't give a dam about Lab tests, it works or it doesn't for me.[a href=\"index.php?act=findpost&pid=217534\"][{POST_SNAPBACK}][/a]

Some of us might be interested in why it doesn't work when the lab tests indicate it should work.

I got the impression that Mike Johston was railing against ill-defined lab tests, subjective quality factors masquerading as lab tests, and MTF charts based on theoretical designs rather than real lenses.

Real and thoroughly executed lab tests and MTF charts, if one understands them, should help one to make informed decisions, not only when buying a lens but when selecting a particular aperture for a particular effect during shooting.

[Jeremy Roussak]

This is the typical rumbling of the digitally challenged, who is offended by something he does not understand. The proof is in the pudding; ...
[a href=\"index.php?act=findpost&pid=217142\"][{POST_SNAPBACK}][/a]

What on earth do you think the misquotation, "The proof is in the pudding" could mean?

Jeremy

[Jeremy Roussak]

Maybe reading reviews is as subjective as looking at photographs    I took his column to mean that specifications is one thing but at the end of the day the art of photography is about the final result. Sure, if some highly technical photography is required then charts may be useful in making your final decision on which lens to choose; but to look at it from the other direction and say a photograph is poor because of the way the lens measures is crass.
 
I have a (dwindling) interest in hifi and the same arguments happen there (but much worse). Some show preference for gear based on on technical measurements and others purely on the sound. And at the end of the day if it sound right who cares how it measures?
[a href=\"index.php?act=findpost&pid=217174\"][{POST_SNAPBACK}][/a]

I felt like that about hi-fi in the eighties, but you can only struggle against the compact disc for so long!

That's why Michael's reviews on this site are so useful and informative. They don't bother with laboratory testing: he just tells us how he found the piece of kit to handle and to behave in the field, and what his results look like. That's far more useful, if the author is someone you feel you can trust.

Jeremy

[bjanes]

Some of us might be interested in why it doesn't work when the lab tests indicate it should work.

I got the impression that Mike Johston was railing against ill-defined lab tests, subjective quality factors masquerading as lab tests, and MTF charts based on theoretical designs rather than real lenses.

Real and thoroughly executed lab tests and MTF charts, if one understands them, should help one to make informed decisions, not only when buying a lens but when selecting a particular aperture for a particular effect during shooting.
[a href=\"index.php?act=findpost&pid=217556\"][{POST_SNAPBACK}][/a]

I agree with Ray that well done lab tests can contribute significantly to our understanding of the performance of cameras and lenses. Evaluation of photographs is also important, but can be very subjective and depends to some extent on the scene--whether it requires MTF at high or low frequencies.

Unfortunately, I have seen neither photos or test data in this thread and the discussion thus far is not helpful.

Bill

[Pete Ferling]

I agree with Ray that well done lab tests can contribute significantly to our understanding of the performance of cameras and lenses. Evaluation of photographs is also important, but can be very subjective and depends to some extent on the scene--whether it requires MTF at high or low frequencies.

Unfortunately, I have seen neither photos or test data in this thread and the discussion thus far is not helpful.

Bill
[a href=\"index.php?act=findpost&pid=217619\"][{POST_SNAPBACK}][/a]

Truth be told,  testing more than one lens, say 100, and finding out that 90% of them all had a similar response curve would be more factual.  Granted, not everyone has time or money to test 100 lens, or even 10.  The manufaturer may run such numbers, and if so, then tests for your particular unit should be a close match to be considered accurate.  Again, it's a baseline on a set of standards.

Once you've deviated from that standard, and from the standpoint of scientific testing, it's pointless.

In argument for testing.  It's good to test your lens if you duplicate the setup and compare the results, but if a given lens has poor results compared to the baseline, then it can be considered a bad lens.  In a visual test, you would have to directly compare a known good lens to actually see the results.  One persons sharpness is anothers blur.

You have to have charts to know what your baseline is.  Use the charts but only for what they were intended.  Then use the lens to see if it's sharp enough for your needs.  Most folks find it easier to do the later.

[wolfnowl]

A follow up and a 'test it yourself' here, for those who are interested:

http://theonlinephotographer.typepad.com/t...r-from-rio.html

[Ray]

You have to have charts to know what your baseline is.  Use the charts but only for what they were intended.  Then use the lens to see if it's sharp enough for your needs.  Most folks find it easier to do the later.
[a href=\"index.php?act=findpost&pid=217692\"][{POST_SNAPBACK}][/a]

That's true. One needs some reference point. The charts provide a comparative reference, such as 'lens A' is as good as, or better than 'lens B'. If one already owns 'lens A' and one intends to buy 'lens B', one can carry out some testing on 'lens B' before buying, using 'lens A' as the standard.

However, there is a flaw in the process. How does one know that one's own copy of 'lens A' is typical? It might be an exceptionally fine copy, or it might be substandard.

If it's the former and the copy of 'lens B' that happens to be available for testing is typical, then one would probably reject it. If it's the latter and the copy of 'lens B' were equally substandard, one would probably accept it if the MTF charts had already created the expectation that both lenses are about equal in performance.

One would then be in the position of owning two substandard lenses.

[KevinA]

Some of us might be interested in why it doesn't work when the lab tests indicate it should work.

That's the point, lab test mean nothing in the field, you have to shoot over a period of time in the conditions to see if the equipment does what you need. You will soon see if a lens doesn't perform as you need or a camera handles well enough etc A lab test just shows it can take pictures of a chart bracket focused on a tripod etc no real indicator how it produces the goods at a sporting event, wedding, fashion shoot or the Zoo. The net has produced a plethora of mathematicians that decide wether a lens or camera works or not by producing charts and statistics and seldom pictures of anything of note. It's like deciding on a car by the 0-60 mph time. At best a chart will show that a lens camera combination isn't a complete dogs dinner.

Kevin.

[lovell]

I generally ignore all the MTF charts, as they're often too subjective, and often do not correlate to actual physical lens testing.  I often read reviews, then rent or borrow the model I'm interested in, and in this way I find out for myself.  As to many of the photo magazines, well, it seems they have never tested a lens they didn't think was "great"...I guess that policy is great for generating advertising dollars.

As to pixel peeping:

Think about this:  One can use a lens for years, producing thousands of "good" pictures with it, then come to find one day that pixel peeping will show the type of limitations that the glass has, that should've been obvious just looking at the pictures produced.

In other words, one should pixel peep as part of the acceptence testing of a new lens, because often "realworld" photographs will not shake out the optical issues.

Nothing wrong with pixel peeping and why there are those that poo-poo it is way, way beyond me.

I'm a pixel peeper and I'm glad to admit it!

But then, ALL my L glass copies are exceptional and had I not pixel peeped upon receiving them, I could not be able to write this.  

To spend $20,000 on L glass and not pixel peep would be the epitome of stupidity.  That warranty is just one year long for free calibrations and other required mitigations for faulty glass.

[Ray]

I generally ignore all the MTF charts, as they're often too subjective, and often do not correlate to actual physical lens testing.

I think you've got this the wrong way round, old chap. MTF charts are as objective as you can possibly get, provided they result from the testing of real lenses, like Photodo MTF charts do.

However, problems might arise if the tester has not been diligent and become aware from other sources that a particular lens is producing anomalous results and is not typical of the quality expected from that particular model. In such circumstances, the tester should obtain another copy of the lens from another manufacturing batch.

Think about this:  One can use a lens for years, producing thousands of "good" pictures with it, then come to find one day that pixel peeping will show the type of limitations that the glass has, that should've been obvious just looking at the pictures produced.

In other words, one should pixel peep as part of the acceptence testing of a new lens, because often "realworld" photographs will not shake out the optical issues.

In the absence of MTF charts, this is all one can do. Test the lens for oneself at various apertures. I'm a firm believer in the concept, "Know thy lenses". But it can be a tedious process. There are times when I think I'd like to buy a particular lens, then I think of all the testing that I'll feel obliged to do to ensure I have a good copy, and sometimes I decide it's not worth the trouble and I don't buy the lens.

I'm a modern person of the technological era. I would prefer any lens I buy to ship with a full set of MTF charts that describe its performance, that is, MTF charts that are specific to that individual copy of the lens.

I hope one day China will find an economical method of doing this, but I fear that the additional cost is not the issue. Manufacturers seem to rely upon a certain amount of B/S and hype to sell their products. Simultaneously providing an accurate performance description of their products could be seen as contradictory.

[Ray]

That's the point, lab test mean nothing in the field, you have to shoot over a period of time in the conditions to see if the equipment does what you need. You will soon see if a lens doesn't perform as you need or a camera handles well enough etc A lab test just shows it can take pictures of a chart bracket focused on a tripod etc no real indicator how it produces the goods at a sporting event, wedding, fashion shoot or the Zoo. The net has produced a plethora of mathematicians that decide wether a lens or camera works or not by producing charts and statistics and seldom pictures of anything of note. It's like deciding on a car by the 0-60 mph time. At best a chart will show that a lens camera combination isn't a complete dogs dinner.

Kevin.
[a href=\"index.php?act=findpost&pid=217876\"][{POST_SNAPBACK}][/a]

Kevin,
They might mean nothing to you, but they are meaningful to me. Below are a couple of charts from Photozone comparing the performance of the Canon EF-S 17-55/F2.8 zoom with the highly regarded Canon 50/F1.4 prime.

[attachment=8132:attachment]  [attachment=8133:attachment]

These charts are only an indirect indicator of lens performance. They are more correctly an indicator of 'system' resolution. The camera used was the 8mp Canon 350D, so the results should be relevant for owners of cameras with a similar pixel density, such as the 20D, 40D and 1Ds3 (in the case of the 50/1.4), although there's no information here about the edge performance of the 50/1.4 on the 1Ds3.

The above charts only describe the performance of the lenses at 50% MTF. The choice of the 50% figure appears to be based upon subjective factors in relation to real world images. Lenses which deliver a high resolution at 50% contrast tend to produce real world images that look sharp and detailed. Performance at 30% MTF (for example) would be less relevant to an appearance of sharpness because such low contrast signals tend to get buried in noise, unless one is shooting high contrast line charts, or unless one's real world images contain image components similar to lines charts, such as street signs, advertising billboards, tree branches against a light background, shop signs etc.  

Examining these bar charts, one should notice something quite remarkable. The EF-S zoom appears to be as sharp as the 50mm prime, across the apertures they both have in common, ie. from F2.8 to F8. My decision to buy this zoom was sparked by the sight of this chart comparison.

Unfortunately, the circumstances in the small store in a busy shopping centre in Bangkok (where I bought the lens) were not conducive to my testing the lens thoroughly before buying. But later testing has confirmed that this lens is indeed as sharp as my copy of the 50/1.4 prime, and just like the charts indicate, is not quite as sharp at the edges of the frame as the 50mm prime which is, of course, designed for a larger format.

However, one factor which the charts do not (and cannot) address is autofocussing accuracy. I'm disappointed with the autofoussing capability of this lens, even after calibration by Canon back home in Australia.

[KevinA]

However, one factor which the charts do not (and cannot) address is autofocussing accuracy. I'm disappointed with the autofoussing capability of this lens, even after calibration by Canon back home in Australia.
[a href=\"index.php?act=findpost&pid=217992\"][{POST_SNAPBACK}][/a]

There you are then, the MTF showed you what a lab test would do, in the real world it does not stack up because the AF is sub par, neither will it show the sloppy MF, or how the elements move when it gets hot or cold, all things that stop you matching the Lab test results. Also the MTF does not tell you of the weird things that happen with point source lights at night with the 50mm f1:4, its like shooting through wet glass. Neither do the MTF charts show the performance of the lens you bought, only the lens tested, if it was a Leica lens you could expect closer tolerances of the construction and materials, neither does a lab test show a lens performance on 3D objects or the bokeh.
 The 50 mm is sharp from f 3.5 onwards at f1:4 it's a special effect portrait lens, I don't think Canon planned it that way. I could right a book on the 17 - 40 mm lens, there has never been a chart made that could explain all its variable idiosyncrasies, you just learn them with use.

Cheers,

Kevin.

[Ray]

There you are then, the MTF showed you what a lab test would do, in the real world it does not stack up because the AF is sub par, neither will it show the sloppy MF, or how the elements move when it gets hot or cold, all things that stop you matching the Lab test results. Also the MTF does not tell you of the weird things that happen with point source lights at night with the 50mm f1:4, its like shooting through wet glass. Neither do the MTF charts show the performance of the lens you bought, only the lens tested, if it was a Leica lens you could expect closer tolerances of the construction and materials, neither does a lab test show a lens performance on 3D objects or the bokeh.
 The 50 mm is sharp from f 3.5 onwards at f1:4 it's a special effect portrait lens, I don't think Canon planned it that way. I could right a book on the 17 - 40 mm lens, there has never been a chart made that could explain all its variable idiosyncrasies, you just learn them with use.

Cheers,

Kevin.
[a href=\"index.php?act=findpost&pid=218043\"][{POST_SNAPBACK}][/a]

Well, you could continue to add to that list of things an MTF chart does not tell you, almost ad infinitum. It doesn't tell you what the weather will be like tomorrow, either. Obviously, the charts are only useful for what they do tell you, not for what they do not and cannot tell you.

If resolution behaviour from the centre to the corners of the image at various apertures is important to you and is therefore a major factor to consider when choosing a lens, then a collection of MTF charts might be a good starting point to narrow the selection.

Unfortunately, there are far too few MTF tests being carried out nowadays, and QC variation can be a problem, which is why I'd prefer to see a complete set of charts relevant to each lens before i buy a lens.