Pages: 1 [2] 3   Go Down

Author Topic: MTF Charts vs. Photographs  (Read 20417 times)

lovell

  • Full Member
  • ***
  • Offline Offline
  • Posts: 131
    • http://
MTF Charts vs. Photographs
« Reply #20 on: August 29, 2008, 01:39:40 pm »

Quote
I think you've got this the wrong way round, old chap. MTF charts are as objective as you can possibly get, provided they result from the testing of real lenses, like Photodo MTF charts do.

However, problems might arise if the tester has not been diligent and become aware from other sources that a particular lens is producing anomalous results and is not typical of the quality expected from that particular model. In such circumstances, the tester should obtain another copy of the lens from another manufacturing batch.
In the absence of MTF charts, this is all one can do. Test the lens for oneself at various apertures. I'm a firm believer in the concept, "Know thy lenses". But it can be a tedious process. There are times when I think I'd like to buy a particular lens, then I think of all the testing that I'll feel obliged to do to ensure I have a good copy, and sometimes I decide it's not worth the trouble and I don't buy the lens.

I'm a modern person of the technological era. I would prefer any lens I buy to ship with a full set of MTF charts that describe its performance, that is, MTF charts that are specific to that individual copy of the lens.

I hope one day China will find an economical method of doing this, but I fear that the additional cost is not the issue. Manufacturers seem to rely upon a certain amount of B/S and hype to sell their products. Simultaneously providing an accurate performance description of their products could be seen as contradictory.
[a href=\"index.php?act=findpost&pid=217985\"][{POST_SNAPBACK}][/a]

Well, even Canon admits that their MTF charts are NOT based on actual scientific testing.  Their charts are "guesses", "theoretical"....this is true!

So you really can't assume an MTF has much integrity.

In addition, the many "serious" online review sites OFTEN show results different then what I found with my lenses.

One more thing:  Even if an MTF is true, it is a a picture of the characteristics of ONE LENS, and may not necessarily be applicable to all copies of that lens.  I have found this to be true often.

At the end of the day, MTF charts are anecdotal at best.  Reviews are too, but more accurate, and allow one to "harmonize" results across dozens and sometimes hundreds of findings from actual owners.
« Last Edit: August 29, 2008, 01:42:32 pm by lovell »
Logged
After composition, everything else is secondary--Alfred Steiglitz, NYC, 1927.

I'm not afraid of death.  I just don't want to be there when it happens--Woody Allen, Annie Hall, '70s

John Camp

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2171
MTF Charts vs. Photographs
« Reply #21 on: August 29, 2008, 04:59:02 pm »

Quote
This is the typical rumbling of the digitally challenged, who is offended by something he does not understand. The proof is in the pudding; my cat looks much better than your dog, so my lens is better than yours.
[a href=\"index.php?act=findpost&pid=217142\"][{POST_SNAPBACK}][/a]

I think Mike probably knows as much about lens testing as anyone on the net. I believe he has chosen to do field testing because it works better. The problem with MTF charts is that a) they're either projections of lens performance in an ideal state (the theoretical performance), or B) a chart derived from a particular lens. The problem with the latter is seen in all the complaints from people who have sent lenses back to the manufacturer for sample faults -- so the MTF chart derived from one lens through actual measurement is not necessarily (or even usually) what you'll see from another sample of the same lens.

Mike Johnston's reviews tend to be something like Mike Reichmann's -- practical tests of what he believes to be a good sample of the lens, and then giving you the opinion of a longtime photography journalist about the quality of that lens in different practical situations. MTF charts don't show everything; I have a brilliant copy of a Nikon 14-24, absolutely tack sharp, and last week had several D3 pictures ruined by a veiling sun flare that I couldn't see on the LCD. So now I'll be a little more careful...and I would have already been a  little more careful if I'd seen a review from Mike Johnston or Mike Reichmann about this characteristic.

I really do think charts have their place -- and that place is a casual examination before purchase, just like you'd look at Consumer Report's numbers on a new car. But ultimately, you have to go with field performance, and I'll pay as careful attention to the road test opinions from Road & Track as I do the numbers from Consumer Reports. Same with cameras.  

JC
« Last Edit: August 29, 2008, 04:59:28 pm by John Camp »
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
MTF Charts vs. Photographs
« Reply #22 on: August 30, 2008, 01:04:18 am »

Quote
At the end of the day, MTF charts are anecdotal at best.  Reviews are too, but more accurate, and allow one to "harmonize" results across dozens and sometimes hundreds of findings from actual owners.
[a href=\"index.php?act=findpost&pid=218138\"][{POST_SNAPBACK}][/a]

At the end of the day, MTF charts are not anecdotal. That's why they are so useful. Unfortunately, we don't seem to have any. You are essentially criticising a non-existent phenomenon. The only collection of MTF charts of actual lenses that I am aware of, is the historical data that still exists on the Photodo site. Other charts, such as those in the Canon Lenswork books, are theoretical, as you've already pointed out.

The MTF bar charts at Photozone are at least current and are results of real tests, but are relevant only to the cropped format sensor.

All the lenses that I own that feature either on the Photodo site of old tests, or the newer Photozone site, have a resolution performance that seems to correspond fairly well with the real test reults on those sites.

The fact that lenses of the same model do vary from copy to copy or batch to batch is definitely a problem, which is why I'd like to see each lens ship with a full set of real MTF test results specific to that copy of that lens.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
MTF Charts vs. Photographs
« Reply #23 on: August 30, 2008, 09:26:38 am »

Quote
The MTF bar charts at Photozone are at least current and are results of real tests, but are relevant only to the cropped format sensor.

All the lenses that I own that feature either on the Photodo site of old tests, or the newer Photozone site, have a resolution performance that seems to correspond fairly well with the real test reults on those sites.
[{POST_SNAPBACK}][/a]

The Photozone tests are very well done, but they are for cropped sensors as you mention. Since the cropped sensors have a generally high pixel density they give a good figure for resolution in the central portion of the image but leave users of full frame sensors in the dark about edge sharpness.

The [a href=\"http://imaging.nikon.com/products/imaging/lineup/lens/af/zoom/af-s_vr_zoom70-200mmf_28g_if/index.htm]Nikkor 70-200 mm f/2.8 AFS[/url] has generally been regarded as a stellar lens, but D3 users were confronted with a nasty surprise when they noted poor sharpness at the periphery of the frame with this lens. However, that characteristic is fully documented in the MTF chart that Nikon publishes for this lens.

Quote
The fact that lenses of the same model do vary from copy to copy or batch to batch is definitely a problem, which is why I'd like to see each lens ship with a full set of real MTF test results specific to that copy of that lens.
[a href=\"index.php?act=findpost&pid=218219\"][{POST_SNAPBACK}][/a]

That would be nice, but is not likely to occur soon.  Bad copies of lenses occur even with expensive Zeiss and Leitz lenses and it is a good idea to test one's own lenses with Imatest and compare the results with those reported by Klauss on Photozone.de. In a couple of hours, one can get a good idea of the performance of the lens at various apertures and focal lengths in the case of zoom lenses. To accumulate such data from photographic tests would take weeks or months of testing.

Bill
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
MTF Charts vs. Photographs
« Reply #24 on: August 30, 2008, 10:21:27 pm »

Quote
That would be nice, but is not likely to occur soon.  Bad copies of lenses occur even with expensive Zeiss and Leitz lenses and it is a good idea to test one's own lenses with Imatest and compare the results with those reported by Klauss on Photozone.de. In a couple of hours, one can get a good idea of the performance of the lens at various apertures and focal lengths in the case of zoom lenses. To accumulate such data from photographic tests would take weeks or months of testing.

Bill
[a href=\"index.php?act=findpost&pid=218247\"][{POST_SNAPBACK}][/a]

Bill,
I agree that it's very unsatisfactory to be in a position where one gets an idea of the performance of a lens only after weeks or perhaps even months of use in the field. During such time, one might have taken a few lucky shots or a 'once in a lifetime' shot which is technically marred because one has used the lens at full aperture (for example), expecting it to be sharp because one had heard anecdotal reports on the internet that the lens is sharp wide open. The fact that your copy of the lens is not sharp at full aperture is a useful piece of information you've discovered in the field, and you have this once-in-a-lifetime, lucky shot to prove it.

If one is concerned about technical image quality, it seems to be necessary to always test a new lens one has bought before the expiry of the return period.

Now this testing process might be fun the first time and perhaps even the second and third time and, if one is particularly technically minded, it might be fun for the rest of one's life.

However, speaking for myself, I would rather pay an extra $100 or so on top of the normal price of a lens in order to know what I'm buying and in order to avoid spending probably considerably more than $100 worth of my own time testing a lens and possibly returning the lens, then testing the second copy and perhaps even returning that second copy before going through the whole process again.

In the recent past, I tested 3 copies of the Canon 10-22 zoom before I found one that autofocussed accurately and which was almost as sharp as my Sigma 15-30. (Finding one that was equally sharp might have been a never ending process).

I once spent the equivalent of more than a whole day testing the Canon 400/F5.6 prime. The time I spent included initial comparisons with my 100-400/5.6 zoom at 400mm; repeated tests because at first I couldn't believe that my zoom was sharper at full aperture and thought that maybe there was an autofocussing issue; processing such test images and burning to CD as evidence; driving back to the store to return the lens for another copy (a 2 hour drive since I live outside the city); returning without a second copy because the store didn't have one; (the salesman suggested he could send the lens in for calibration and if it wasn't improved then he'd give me a refund); repeating the testing procedure with the same lens after calibration, only to discover that it was still not as sharp as my zoom at F5.6, although slightly improved as a result of the calibration; another 2 hour drive back to the store to return the lens and collect my refund.

Geez! What a waste of my time! Now, to avoid impressions of exaggeration and hyperbole, I'll admit that the 12 hours of driving involved in this exercise cannot wholly be allocated to the decision to purchase this lens. When I spend two hours driving into town, it's for other purposes as well. I could have done all transactions regarding the purchase of this lens through Australia Post. Instead of a 2 hour drive on 6 occasions, it could have been a 20 minute drive to the nearest Post office on 6 occasions. Add the time spent unpacking and repacking the lens and the time spent chatting with the postmaster, that's probably a total of only 4 hours, not to mention the expense of fuel and wear & tear of the car.

The bottom line is, I spent more than a day of my life attempting to buy a lens that proved to be inadequate for my purposes, and I still don't have a copy of that lens because I'm reluctant to repeat this time-consuming and expensive process of testing.

Consider what could have happened in this case if all lenses were to ship with a comprehensive set of MTF charts. I'd walk into the store with a copy of the MTF charts relating to my 100-400 zoom. I'd tell  the salesman that I wasn't particularly happy with the sharpness of my 100-400 IS zoom at F5.6 and that I'd like to check out the 400/5.6 prime which I'd heard is sharper and which I'd expect to be sharper because it's a prime lens.

The salesman removes the set of charts in an envelope attached to the sealed box containing the 400/5.6 prime. I compare the charts of both lenses at F5.6. Both of us can see that my 100-400 zoom is sharper, so I reject the lens on the spot. No need for 12 hours, or even four hours of driving, and a day of my life has been saved for more useful activities, such as expressing my views on LL   .
Logged

spidermike

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 535
MTF Charts vs. Photographs
« Reply #25 on: September 02, 2008, 10:31:47 am »

Quote
Consider what could have happened in this case if all lenses were to ship with a comprehensive set of MTF charts. I'd walk into the store with a copy of the MTF charts relating to my 100-400 zoom. I'd tell  the salesman that I wasn't particularly happy with the sharpness of my 100-400 IS zoom at F5.6 and that I'd like to check out the 400/5.6 prime which I'd heard is sharper and which I'd expect to be sharper because it's a prime lens.

The salesman removes the set of charts in an envelope attached to the sealed box containing the 400/5.6 prime. I compare the charts of both lenses at F5.6. Both of us can see that my 100-400 zoom is sharper, so I reject the lens on the spot. No need for 12 hours, or even four hours of driving, and a day of my life has been saved for more useful activities, such as expressing my views on LL   .
[a href=\"index.php?act=findpost&pid=218388\"][{POST_SNAPBACK}][/a]

I can think of no other product where this is expected. Cars? Hifi? Fishing gear?

The effects on marketing could be interesting... You rejected three copies of the 10-22 zoom because they were not up to your standard so how should those now be sold? It is now known they are not 'perfect' in performance, so should they be graded as B+ because they are not up to scratch? And price-reduced accordingly? It could follow on that the copy you bought be classed as A++ and have a premium price.
Should a super-quality 10-22 be classed as 'L' quality?  
Or a poor-compliance 'L' lens have this classification removed?  
Logged

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
MTF Charts vs. Photographs
« Reply #26 on: September 02, 2008, 12:35:40 pm »

Quote
I can think of no other product where this is expected. Cars? Hifi? Fishing gear?

The effects on marketing could be interesting... You rejected three copies of the 10-22 zoom because they were not up to your standard so how should those now be sold? It is now known they are not 'perfect' in performance, so should they be graded as B+ because they are not up to scratch? And price-reduced accordingly?


Spidermike

I can´t say that your analogy with cars, hi-fi and fishing holds.

The car´s performace isn´t measured to the very fine tolerances that lenses have to be: 0 to 60mph in how many secs and millisecs? and music quality is as much a product of the listener´s ear as the machines whilst fishing, in the end, depends on how hungry/dumb the poor old fish.

A lens does nothing except pass light. If it can´t do that to spec then it has failed. It should not, then, be re-sold at all - it should be scrapped.

Of course this would cost somebody (everybody?) money, but then so be it. Like Ray, there are better things to do in life than buy second-best thinking one has bought best. If you want to make comparisons, then would you willingly fly if you imagined the aircraft was possibly flawed? Of course not; and your ticket costs accordingly, with massive fines if negligence is proven.

I can´t ever remember this happening with either my Haselblad C lenses nor with the Nikkors that I also bought during the period up to the mid-eighties. However, after that, and with later purchases, I have had moments of doubt with Nikkor optics too. I currently own a 2.8/135 which, certainly at 2.8 and possibly throughout the range of stops, fails to equal what I used to take for granted with my earlier 3.5/135. What is the point of 2.8 if it is nothing but a focussing aid? And the rest of the aperture range is inferior too?

In my opinion, it is a wholesale dumbing down of life and expectations. Those with early experience are powerless to do anything about it and those newer to the game take it as the norm. Never in my life had I heard of a return period in case of the lens not being up to snuff; I wonder just how well advertised such a concept really is. But I can see why it now has to exist!

Rob C
« Last Edit: September 02, 2008, 12:37:09 pm by Rob C »
Logged

spidermike

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 535
MTF Charts vs. Photographs
« Reply #27 on: September 03, 2008, 08:12:03 am »

Quote
The car´s performace isn´t measured to the very fine tolerances that lenses have to be: 0 to 60mph in how many secs and millisecs? and music quality is as much a product of the listener´s ear as the machines whilst fishing, in the end, depends on how hungry/dumb the poor old fish.
True. But sound engineering is crucial to all those products - and they do have QC whether or not is as high as for lenses is a moot point (think piston-bore tolerances on a Maserati engine block). And the very fact that Canon is still in business suggests most users find the variation in quality to be acceptable - so it is in fact just as subjective as cars and hifi.

Quote
A lens does nothing except pass light. If it can´t do that to spec then it has failed. It should not, then, be re-sold at all - it should be scrapped.
But it does do that and within Canon's QC limits. If someone doesn't like their QC limits then the choice is simple.

Quote
Of course this would cost somebody (everybody?) money, but then so be it.
Think of the costs of individual MTFs and the consequently higher rejection rate - these would increase manufacturing costs and would give disproportionate increases in sales cost. You may be happy to pay up, but think of the millions who would be excluded. You talk about dumbing down (not a comment I agree with), but if it wasn't for the mass market it creates, would we really have the money to pay for the research to develop affordable 12, 15, or 22 Mega-pixel cameras?



The more I think about this, the more mind-boggling it becomes: to maintain income you would need to sell all lenses that fall within the current QC limits: a high-performing 'non-L' lens would impinge on the 'L' market; and a high-performing 'L' lens would have to be sold be a 'Super 'L' lens.
You would need to have differential costings to differentiate the model ranges: the lower-QC lenses selling at the current market price, the higher-QC lenses having a price premium to pay for the MTFs on all lenses (plus, no doubt, an added mark-up!)

So would you rather pay a (probably significant) premium to make your shopping easier, or would you prefer to stay with the current QC limits and try out 3 or 4 copies of the lens to get what you would call 'a good copy'? My guess is that the acceptable price differential is not as much as you like to think.

I didn't mean this to be a treatise on marketing but I got carried away.  
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
MTF Charts vs. Photographs
« Reply #28 on: September 03, 2008, 10:09:54 am »

Quote
The more I think about this, the more mind-boggling it becomes: to maintain income you would need to sell all lenses that fall within the current QC limits: a high-performing 'non-L' lens would impinge on the 'L' market; and a high-performing 'L' lens would have to be sold be a 'Super 'L' lens.
You would need to have differential costings to differentiate the model ranges: the lower-QC lenses selling at the current market price, the higher-QC lenses having a price premium to pay for the MTFs on all lenses (plus, no doubt, an added mark-up!)

So would you rather pay a (probably significant) premium to make your shopping easier, or would you prefer to stay with the current QC limits and try out 3 or 4 copies of the lens to get what you would call 'a good copy'? My guess is that the acceptable price differential is not as much as you like to think.

I didn't mean this to be a treatise on marketing but I got carried away. 
[a href=\"index.php?act=findpost&pid=219138\"][{POST_SNAPBACK}][/a]

Yes. You got it. Of course you would want to sell all lenses and grade them accordingly. The principle is, the customer should know what he's buying and not have to spend hours checking to see if he's actually received the quality of product he thought he was buying.

The Canon 400/5.6 prime that I rejected after spending hours of my time testing and comparing with another lens, was not necessarily faulty. I'm sure it would have been sold on to another customer. It had been sent away for calibration and had been returned as being in good order. Another customer might be quite happy with its general sharpness, not realising (and perhaps not even caring) that it was close to the bottom of the QC range of acceptable sharpness for that model.

Don't we all expect to pay more for a quality product? There's only one ultra-wide-angle EF-S zoom, the 10-22mm. Would you not prefer to have a choice of say, 3 different grades at 3 different prices? We could then have endless discussions on internet forums about the differences in resolution amongst the different grades and whether or not Grade A (EF-S 10-22) at double the price of Grade C was worth the extra money.  

Incidentally, my hi fi loudspeakers, bought over 20 years ago and still going strong (Celestion SL600), each came with their own frequency response chart. It's supposed to be a real test of the decibel sound level in relation to all frequencies from 20Hz to 20,000Hz, and the charts are slightly different for each speaker in the pair as one would expect.
Logged

lovell

  • Full Member
  • ***
  • Offline Offline
  • Posts: 131
    • http://
MTF Charts vs. Photographs
« Reply #29 on: September 03, 2008, 02:05:44 pm »

Although lense are mass produced, and on an assembly line, they are unique creatures.  Even the hand built L lenses of Canon fame are built on an assembly line.  

So purchasing a lens is about cherry picking, as if one is buying peaches.  Lots of okay peaches, a few really sweet really good peaches, and lots of mushy bruised ones too.  Lenses are not commodities like cars, trucks, watches coming off the assembly line.  This is because there are so many variations in the manufacturing processes that are required to make a lens.  And more then that, the margin for acceptability is so narrow.

To me, a lens kit or "stable" is about building it over years, and slowly.  This can mean replaceing that 50mm or 24-70 zoom a few times, or getting it calibrated, and this can take a long time.  It took me 8 years to finalize my own stable of glass, and so until the makers provide individual MTF charts in each box, I know of no other way to insure good glass.

Now fast forward 5 or 10 years. Will our stable of lenses resolve to the 50mp or 150mp sensors?  They won't!  Now we'll have to start the process all over again with new glass....
Logged
After composition, everything else is secondary--Alfred Steiglitz, NYC, 1927.

I'm not afraid of death.  I just don't want to be there when it happens--Woody Allen, Annie Hall, '70s

Slough

  • Guest
MTF Charts vs. Photographs
« Reply #30 on: September 03, 2008, 02:44:19 pm »

Quote
I think you've got this the wrong way round, old chap. MTF charts are as objective as you can possibly get, provided they result from the testing of real lenses, like Photodo MTF charts do.

My own experience is that MTF plots are not particularly useful, although they can suggest how resolution varies across the frame at a given aperture.

They are objective, but then again so is the measurement of the diameter of the front element. The question really is whether or not they contain useful information. Unfortunately they tell you nothing about flare, ghosting, CA, bokeh, colour cast, etc. And of course the MTF is measured at only one subject distance and it is known that in many cases a lens can perform quite differently at infinity compared to the mid range, and close range.

Using an MTF to characterise a lens is a bit like judging the appearance of a car by looking at the front alone.

In fact I think it is worse, as some lenses that appear to have good MTF plots are dogs in practice. You could argue that since my experience is based on my lenses, that I had bad samples. But if that is the case - and I do not believe that it is - then MTF plots are pointless because sample variation is too significant. In fact I simply do not trust them for anything other than eliminating obviously bad lenses.

Oh yes, and these days you really need to know how a lens performs on a particular camera, as the lens camera interaction is significant.

I prefer to use reviews from people who are known to be careful workers as chances are they have performed brick wall tests, and used the lens in the field, in various situations. After all, there is no point in using a lens which performs well only in a very artificial situation.
Logged

Slough

  • Guest
MTF Charts vs. Photographs
« Reply #31 on: September 03, 2008, 02:57:31 pm »

Quote
To me, a lens kit or "stable" is about building it over years, and slowly.  This can mean replaceing that 50mm or 24-70 zoom a few times, or getting it calibrated, and this can take a long time.  It took me 8 years to finalize my own stable of glass, and so until the makers provide individual MTF charts in each box, I know of no other way to insure good glass.
[a href=\"index.php?act=findpost&pid=219224\"][{POST_SNAPBACK}][/a]


There does seem to be an awful lot of Canon users saying they have had to return a lens to be 'calibrated', and very few Nikon users complaining. I wonder if this is the case, or if there are factors at work here which mask a similar issue with Nikon lenses? I know that Rorslett mentioned about sample variation in 17-35mm zooms in the early days of production. There seem to be no cases of 14-24 lenses being squiffy, at least not online. I wonder if the higher prices of Nikon lenses is due to better QC?

(I use Nikon just in case anyone wonders. )
Logged

lovell

  • Full Member
  • ***
  • Offline Offline
  • Posts: 131
    • http://
MTF Charts vs. Photographs
« Reply #32 on: September 03, 2008, 07:59:23 pm »

Quote
There does seem to be an awful lot of Canon users saying they have had to return a lens to be 'calibrated', and very few Nikon users complaining. I wonder if this is the case, or if there are factors at work here which mask a similar issue with Nikon lenses? I know that Rorslett mentioned about sample variation in 17-35mm zooms in the early days of production. There seem to be no cases of 14-24 lenses being squiffy, at least not online. I wonder if the higher prices of Nikon lenses is due to better QC?

(I use Nikon just in case anyone wonders. )
[a href=\"index.php?act=findpost&pid=219238\"][{POST_SNAPBACK}][/a]

Rest assurd...Nikon has the same rate of lens issues as Canon...all my Nikon shooting buddies attest to this, and if you look on Nikon sites, you too will see this as true.  But to be sure, I do believe that Nikon and Canon make great lenses, and often better then the best Germen models....ever try the Canon 35L?  Just one example of near perfection, and a lens that the Germen makes have little or nothing over.
Logged
After composition, everything else is secondary--Alfred Steiglitz, NYC, 1927.

I'm not afraid of death.  I just don't want to be there when it happens--Woody Allen, Annie Hall, '70s

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
MTF Charts vs. Photographs
« Reply #33 on: September 03, 2008, 09:43:28 pm »

Quote
My own experience is that MTF plots are not particularly useful, although they can suggest how resolution varies across the frame at a given aperture.

Yes. I understand that, which is why I'm advocating real MTF tests of the lens you buy. Such tests would not be merely suggestive of resolution but would be indicative of actual resolution.

Quote
Using an MTF to characterise a lens is a bit like judging the appearance of a car by looking at the front alone.

That's exactly what it is not. Judging the appearance of a car by looking at the front would be like judging the quality of a lens only from the existing specifications which the manufacturer offers.

Quote
The question really is whether or not they contain useful information. Unfortunately they tell you nothing about flare, ghosting, CA, bokeh, colour cast, etc. And of course the MTF is measured at only one subject distance and it is known that in many cases a lens can perform quite differently at infinity compared to the mid range, and close range.

If resolution peformance at various apertures is a useful thing to know, then of course MTF charts contain useful information. If you are not concerned about or interested in resolution and are only concerned about matters such as flare, waterproofing etc., then of course the MTF chart is not providing you with useful information.

MTF charts can provide an indication of the quality of bokeh (where there's little divergence between the sagittal and meridional lines, bokeh is said to be good).  MTF tests can also be carried out at different focussing distances. I believe some macro lenses can be sharper at close distances than at infinity. The fact that MTF charts do not tell you everything about a lens is no reason for not having such tests. I'm not advocating that other sources of information about a lens should be discouraged. You seem to be arguing that there's not much point in having an eye test because such a test does not provided useful information on the state of your bowels.

Quote
I prefer to use reviews from people who are known to be careful workers as chances are they have performed brick wall tests, and used the lens in the field, in various situations. After all, there is no point in using a lens which performs well only in a very artificial situation.

You mean you prefer apples to oranges although you've never tasted an orange? Unless you have acquired one of the lenses which Photodo tested, you probably don't have any MTF charts specific to an individual copy of any lens you own. And even if you did, it wouldn't be much use to you in isolation. However, a system in which all lenses ship with MTF charts would be very useful.

As for reviews from people who are known to be careful workers, how does that help you in a situation where QC variation is common?

As a matter of interest, I bought the Canon 400/5.6 prime after reading Michael's review of it in which he compared it with his copy of the 100-400 IS zoom and demonstrated with shots of brick walls that it was clearly sharper at F5.6.

My copy of that lens clearly wasn't sharper but it's not clear to me to what extent I might simply have an above average copy of the 100-400 IS zoom. The purpose of real MTF tests specific to each lens is to provide a universal reference point to make resolution comparisons accurate and easy. A metre is a metre wherever you are.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
MTF Charts vs. Photographs
« Reply #34 on: September 03, 2008, 11:45:44 pm »

Well, perhaps the vendors would improve their tolerances so that we would not have out of spec lenses leaving the factory.

Erik

Quote
I can think of no other product where this is expected. Cars? Hifi? Fishing gear?

The effects on marketing could be interesting... You rejected three copies of the 10-22 zoom because they were not up to your standard so how should those now be sold? It is now known they are not 'perfect' in performance, so should they be graded as B+ because they are not up to scratch? And price-reduced accordingly? It could follow on that the copy you bought be classed as A++ and have a premium price.
Should a super-quality 10-22 be classed as 'L' quality?   
Or a poor-compliance 'L' lens have this classification removed? 
[a href=\"index.php?act=findpost&pid=218921\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
MTF Charts vs. Photographs
« Reply #35 on: September 04, 2008, 12:00:34 am »

Just a question, anyone having an idea what Canon means by calibrating a lens? The only thing I can see calibrated in a lens is the aperture control and autofocus mechanisms (although autofocus is mainly done by the camera). The bayonet can probably adjusted to. But what do they do about decentered lenses? Do they disassemble/reassemble the lenses?

Best regards
Erik


Quote
Although lense are mass produced, and on an assembly line, they are unique creatures.  Even the hand built L lenses of Canon fame are built on an assembly line. 

So purchasing a lens is about cherry picking, as if one is buying peaches.  Lots of okay peaches, a few really sweet really good peaches, and lots of mushy bruised ones too.  Lenses are not commodities like cars, trucks, watches coming off the assembly line.  This is because there are so many variations in the manufacturing processes that are required to make a lens.  And more then that, the margin for acceptability is so narrow.

To me, a lens kit or "stable" is about building it over years, and slowly.  This can mean replaceing that 50mm or 24-70 zoom a few times, or getting it calibrated, and this can take a long time.  It took me 8 years to finalize my own stable of glass, and so until the makers provide individual MTF charts in each box, I know of no other way to insure good glass.

Now fast forward 5 or 10 years. Will our stable of lenses resolve to the 50mp or 150mp sensors?  They won't!  Now we'll have to start the process all over again with new glass....
[a href=\"index.php?act=findpost&pid=219224\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
MTF Charts vs. Photographs
« Reply #36 on: September 04, 2008, 12:12:47 am »

Hi,

Anyone knows what "lens calibration" means?

Erik

Quote
There does seem to be an awful lot of Canon users saying they have had to return a lens to be 'calibrated', and very few Nikon users complaining. I wonder if this is the case, or if there are factors at work here which mask a similar issue with Nikon lenses? I know that Rorslett mentioned about sample variation in 17-35mm zooms in the early days of production. There seem to be no cases of 14-24 lenses being squiffy, at least not online. I wonder if the higher prices of Nikon lenses is due to better QC?

(I use Nikon just in case anyone wonders. )
[a href=\"index.php?act=findpost&pid=219238\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
MTF Charts vs. Photographs
« Reply #37 on: September 04, 2008, 01:23:06 am »

Hi,

The German monthly "Color Foto" has added focus accuracy to their lens tests, my reflection is:

a) That's a good thing
 There are significant variation, which is a bad thing.

I think that MTF tests are a very good indicator of the sharpness of the lens. There are other parameters which at least in some situations are more important than sharpness, especially flare and ghosting.

Some optical problems can be corrected easily in postprocessing, so I may not care that much about those, examples are:

Vignetting
Distorsion
Lateral chromatic aberration

Finally, there is a difference on how a lens works in the "lab" and in the field. A lens that has issues in the "lab" will also have issues in the field. Knowing the strengths and weaknesses of your lens is always helpful.

I think that testing your own lenses is a very good idea. I had experience with two lemon lenses, so I feel the issue is real. To put things in perspective I owned something like 25 lenses. Now days designs are complex and there is pressure on costs which leads to perhaps more sample variations than we had before.

Fixed focals do have some advantages, but my experience is that in many cases my vantage point is fixed. Having zooms allows med to crop in the camera so I don't need to crop in the image.

Erik
 

Quote
Kevin,
They might mean nothing to you, but they are meaningful to me. Below are a couple of charts from Photozone comparing the performance of the Canon EF-S 17-55/F2.8 zoom with the highly regarded Canon 50/F1.4 prime.

[attachment=8132:attachment]  [attachment=8133:attachment]

These charts are only an indirect indicator of lens performance. They are more correctly an indicator of 'system' resolution. The camera used was the 8mp Canon 350D, so the results should be relevant for owners of cameras with a similar pixel density, such as the 20D, 40D and 1Ds3 (in the case of the 50/1.4), although there's no information here about the edge performance of the 50/1.4 on the 1Ds3.

The above charts only describe the performance of the lenses at 50% MTF. The choice of the 50% figure appears to be based upon subjective factors in relation to real world images. Lenses which deliver a high resolution at 50% contrast tend to produce real world images that look sharp and detailed. Performance at 30% MTF (for example) would be less relevant to an appearance of sharpness because such low contrast signals tend to get buried in noise, unless one is shooting high contrast line charts, or unless one's real world images contain image components similar to lines charts, such as street signs, advertising billboards, tree branches against a light background, shop signs etc. 

Examining these bar charts, one should notice something quite remarkable. The EF-S zoom appears to be as sharp as the 50mm prime, across the apertures they both have in common, ie. from F2.8 to F8. My decision to buy this zoom was sparked by the sight of this chart comparison.

Unfortunately, the circumstances in the small store in a busy shopping centre in Bangkok (where I bought the lens) were not conducive to my testing the lens thoroughly before buying. But later testing has confirmed that this lens is indeed as sharp as my copy of the 50/1.4 prime, and just like the charts indicate, is not quite as sharp at the edges of the frame as the 50mm prime which is, of course, designed for a larger format.

However, one factor which the charts do not (and cannot) address is autofocussing accuracy. I'm disappointed with the autofoussing capability of this lens, even after calibration by Canon back home in Australia.
[a href=\"index.php?act=findpost&pid=217992\"][{POST_SNAPBACK}][/a]
Logged
Erik Kaffehr
 

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
MTF Charts vs. Photographs
« Reply #38 on: September 04, 2008, 03:14:46 am »

Quote
Well, perhaps the vendors would improve their tolerances so that we would not have out of spec lenses leaving the factory.

Erik
[a href=\"index.php?act=findpost&pid=219341\"][{POST_SNAPBACK}][/a]


In order to improve the tolerance, they might have to do something like an individual MTF test on each lens. If they do this, I'd like to have the result for reference purposes and comparison with other purchases.

If they find a way of improving tolerances through another process, then that's fine with me also because Photodo type MTF tests will then be more relevant to all copies of a particular model of lens.
Logged

Slough

  • Guest
MTF Charts vs. Photographs
« Reply #39 on: September 04, 2008, 02:16:19 pm »

Ray: You seem to have missed the points I made.

Quote
Yes. I understand that, which is why I'm advocating real MTF tests of the lens you buy. Such tests would not be merely suggestive of resolution but would be indicative of actual resolution.

I am confused. My comments apply to any MTF chart.  

Quote
That's exactly what it is not. Judging the appearance of a car by looking at the front would be like judging the quality of a lens only from the existing specifications which the manufacturer offers.


That is exactly what it is. As I explained quite clearly, an MTF only tells you about part of the optical performance of a lens. My reasons were given in the earlier post to which I refer you.

Quote
If resolution peformance at various apertures is a useful thing to know, then of course MTF charts contain useful information. If you are not concerned about or interested in resolution and are only concerned about matters such as flare, waterproofing etc., then of course the MTF chart is not providing you with useful information.


I am interested in optical performance in the sense of how well it creates images.


Quote
MTF charts can provide an indication of the quality of bokeh (where there's little divergence between the sagittal and meridional lines, bokeh is said to be good). 

That I doubt. Bokeh is all about how the lens images light in regions outside the plane of focus.

Quote
MTF tests can also be carried out at different focussing distances.

But they aren't. You referred to PhotoDo. They are done at one distance.

Quote
I believe some macro lenses can be sharper at close distances than at infinity. The fact that MTF charts do not tell you everything about a lens is no reason for not having such tests.

Yes, but if the tests tell you so little, they become misleading.

Quote
I'm not advocating that other sources of information about a lens should be discouraged. You seem to be arguing that there's not much point in having an eye test because such a test does not provided useful information on the state of your bowels.
You mean you prefer apples to oranges although you've never tasted an orange?


You have completely lost me. I am saying that an MTF tells you so little about the optical performance of a lens as to be misleading.

Quote
Unless you have acquired one of the lenses which Photodo tested, you probably don't have any MTF charts specific to an individual copy of any lens you own. And even if you did, it wouldn't be much use to you in isolation. However, a system in which all lenses ship with MTF charts would be very useful.

But you referred to PhotoDo as being of use! Are you changing your mind?

Quote
As for reviews from people who are known to be careful workers, how does that help you in a situation where QC variation is common?

I would not want to buy a lens from such a manufacturer.

Quote
As a matter of interest, I bought the Canon 400/5.6 prime after reading Michael's review of it in which he compared it with his copy of the 100-400 IS zoom and demonstrated with shots of brick walls that it was clearly sharper at F5.6.

My copy of that lens clearly wasn't sharper but it's not clear to me to what extent I might simply have an above average copy of the 100-400 IS zoom. The purpose of real MTF tests specific to each lens is to provide a universal reference point to make resolution comparisons accurate and easy. A metre is a metre wherever you are.
[a href=\"index.php?act=findpost&pid=219327\"][{POST_SNAPBACK}][/a]


Ray: You confuse me as you have changed your argument. The point I was making is that MTF charts are not a good basis for deciding between two different lenses, maybe with similar specs but from different manufacturers

To remind you, here is your post that I was referring to:

"I think you've got this the wrong way round, old chap. MTF charts are as objective as you can possibly get, provided they result from the testing of real lenses, like Photodo MTF charts do."

In the context of the above, my response makes sense.

The point you made in your reply - which is quite different - is that an MTF can provide a way to check that a particular sample of a lens is within the manufacturers specs. I do not argue with that as quantities such as colour cast, flare resistance, bokeh, contrast etc would tend to be consistent between samples of the same lens, and all you want is to check for gross deviations in resolution, thus indicating something very wrong.

It is a little hard to respond if you make one argument, and then respond with a totally different one.
Logged
Pages: 1 [2] 3   Go Up