Sharpening ... Not the Generally Accepted Way!

[Jack Hogan]

I doubt that I would get as good as this in the field, so I wonder, Jack, if you could explain why getting an optimally focused image is useful for your modelling ... because it's pretty tricky to achieve!

You are right, Robert, Robert Cicala of lensrentals.com says that a difference of about 10% in MTF50 is barely noticeable.  I tend to agree.  The reason for going the extra distance when attempting to determine the parameters for Capture Sharpening (recall: capture sharpening = restore sharpness lost during capture process = camera/lens hardware dependent) is that otherwise we cannot 'see' them and unless someone at Canon obliges with the figures we have to guesstimate.

For instance a key one is the strength of the AA filter. I assume that the 1DsIII has an AA filter in a classic 4-dot beam splitting configuration like the Exmor sensored cameras I am more familiar with.  Since most such AAs cause a shift of about +/- 0.35 pixels, we should be able to see a zero around there in the relative MTF curve (in cycle/pixels it is 0.25/offset):



So we know that the A7s AA appears to be about +/- 0.363 pixels in strength, and if we wanted to attempt to remove its effects through deconvolution we would have a good estimate as far as what the shape and size of its PSF are concerned.  However if the spatial resolution information is buried in a morass of lens induced blur we are not going to be able to find what we seek.  Heck it might be that the Canons do not have a 4-dot beam splitter, or that it is a lot less strong, in which case all bets are off (the slanted edge method becomes exponentially unreliable past Nyquist)

Jack

PS Since I suspected that my D610 (like the A7 and other Exmors of its generation) has AA action in one direction only, I figured that if I divided the MTF obtained from a vertical edge by the one obtained from a horizontal edge in the same capture the result should be the missing element = the MTF of the AA filter. And low and behold, right as theory had predicted (ignore stuff after the zero, there is too much noise and too little energy there for the division of two small numbers to be meaningful - it was quite a noisy image to start with):

[Robert Ardill]

You are right, Robert, Robert Cicala of lensrentals.com says that a difference of about 10% in MTF50 is barely noticeable.  I tend to agree.  The reason for going the extra distance when attempting to determine the parameters for Capture Sharpening (recall: capture sharpening = restore sharpness lost during capture process = camera/lens hardware dependent) is that otherwise we cannot 'see' them and unless someone at Canon obliges with the figures we have to guesstimate.

For instance a key one is the strength of the AA filter. I assume that the 1DsIII has an AA filter in a classic 4-dot beam splitting configuration like the Exmor sensored cameras I am more familiar with.  Since most such AAs cause a shift of about +/- 0.35 pixels, we should be able to see a zero around there in the relative MTF curve (in cycle/pixels it is 0.25/offset):



So we know that the A7s AA appears to be about +/- 0.363 pixels in strength, and if we wanted to attempt to remove its effects through deconvolution we would have a good estimate as far as what the shape and size of its PSF are concerned.  However if the spatial resolution information is buried in a morass of lens induced blur we are not going to be able to find what we seek.  Heck it might be that the Canons do not have a 4-dot beam splitter, or that it is a lot less strong, in which case all bets are off (the slanted edge method becomes exponentially unreliable past Nyquist)

Jack

PS Since I suspected that my D610 (like the A7 and other Exmors of its generation) has AA action in one direction only, I figured that if I divided the MTF obtained from a vertical edge by the one obtained from a horizontal edge in the same capture the result should be the missing element = the MTF of the AA filter. And low and behold, right as theory had predicted (ignore stuff after the zero, there is too much noise and too little energy there for the division of two small numbers to be meaningful - it was quite a noisy image to start with):

Hmmm ... very interesting, although I need to get my thinking cap on to make sense of it!  It really is a shame the camera manufacturers are so unforthcoming with information.

I take it that the MTFs for diffraction (at a fixed aperture), pixel aperture and the AA filter are all constant? Also that diffraction and pixel aperture MTFs can be quite accurately estimated?  That leaves the unknowns, which are the lens blur and AA filter.  So, if you take two shots, the only difference being a slight change in the lens blur ... could you not then work out the AA from that?  Notice that I say you, because I certainly could not!  And no doubt it's not possible to do or you would be doing it already.

I had a quick look at MTF Mapper and it seems very good.  If you could give me your command arguments I could use it to check my image before sending it to you.

Robert

[Jack Hogan]

I take it that the MTFs for diffraction (at a fixed aperture), pixel aperture and the AA filter are all constant? Also that diffraction and pixel aperture MTFs can be quite accurately estimated?

Yes, and the more you narrow the wavelength of the light the better, that's why I like to work with the green CFA raw channel only, which for some Nikon cameras has 1/2 power bandwidth of around 540nm +/-50ish.

That leaves the unknowns, which are the lens blur and AA filter.  So, if you take two shots, the only difference being a slight change in the lens blur ... could you not then work out the AA from that?  Notice that I say you, because I certainly could not!  And no doubt it's not possible to do or you would be doing it already.

Lens blur is the hardest of the simple components to model because it depends on so many variables (if we concentrate on the center only of well corrected lenses at least SA, CAs and defocus): it changes model significantly and non-linearly with even small incremental variations.  So far I have concentrated on modeling well corrected prime lenses with small amounts of defocus in the center of the FOV.  By small I mean less than half a wavelength of optical path difference (Lord Rayghley's criterion for in-focus was 1/4 lambda OPD).  It has the finickiest theory and it is the plug in my overall model: diffraction, pixel aperture and AA are set according to their physical properties and camera settings.  Solver than varies OPD to get the best fit to measured data.  There is always a residual value because no lens is ever perfect.  I  have never seen it at less than 0.215 lambda, which corresponds to a lens blur diameter of about 5.3um (on a 2.4um pitched RX100vIII).

I had a quick look at MTF Mapper and it seems very good.  If you could give me your command arguments I could use it to check my image before sending it to you.

I don't have Imatest so perhaps you can set it up to do the same thing, and trust me it would be much easier.  MTF Mapper is excellent because it allows one to work directly on the green channel raw data, without introducing demosaicing blur into the mix.  The author, Frans van den Bergh is a very smart and helpful guy whose blog got me going on this frequency domain trip. On the other hand it is an open source command line program which is not as user friendly as commercial products.  This is the way I use it, you may not want to once you realize what's involved

1) First create a TIFF of the raw data with dcraw -D -4 -T filename.cr2;
2) Open filename.tiff in a good editor and save a 400x200 pixel crop (horizontal edge, 200x400 vertical) of the central edge you'd like to analyze in a file called, say, h.tif making sure the top left most pixel of h.tif corresponds to a Red pixel in the original raw data (use RawDigger for that)
3) run the command line "mtf_mapper h.tif g:\ -arbef --bayer green -t x", assuming that you are working in directory g:\ and x is the threshold (your last two images worked with x=0.5)
4) MTF Mapper produces a number of text files and Annotate.png: open mtf_sfr.txt in Excel using the data import function.  There should be four lines with 65 values each.  The first value of each line is the angle of the edge (ideally it should be somewhere between 5-10 degrees).  The remaining 64 values are the MTF curve in 1/64th cycles/pixel increments, starting with 0 cy/px which clearly has an MTF value of 1.  Choose the line that corresponds to the edge (see the Annotate.PNG file) and plot it.

Voila', that's the MTF curve of just the two green raw channels.  Alternatively send me the file (one at a time please) and I'll do it for you - I've got batch files for most of this but they reflect how I work, call other programs and they are not easy to explain or set up if starting from scratch.

Jack

[Robert Ardill]

Yes, and the more you narrow the wavelength of the light the better, that's why I like to work with the green CFA raw channel only, which for some Nikon cameras has 1/2 power bandwidth of around 540nm +/-50ish.

Lens blur is the hardest of the simple components to model because it depends on so many variables (if we concentrate on the center only of well corrected lenses at least SA, CAs and defocus): it changes model significantly and non-linearly with even small incremental variations.  So far I have concentrated on modeling well corrected prime lenses with small amounts of defocus in the center of the FOV.  By small I mean less than half a wavelength of optical path difference (Lord Rayghley's criterion for in-focus was 1/4 lambda OPD).  It has the finickiest theory and it is the plug in my overall model: diffraction, pixel aperture and AA are set according to their physical properties and camera settings.  Solver than varies OPD to get the best fit to measured data.  There is always a residual value because no lens is ever perfect.  I  have never seen it at less than 0.215 lambda, which corresponds to a lens blur diameter of about 5.3um (on a 2.4um pitched RX100vIII).

Yikes .  I don't know how one would go about focusing a lens to that accuracy. The Canon EOS utility does give remote control over the focusing, but not with mirror lock-up, which is a shame.  Also, the focusing step-size doesn't seem to be all that fine.  So not only is lens blur the hardest component to model ... but it's also very hard to minimize it in the capture!

I don't have Imatest so perhaps you can set it up to do the same thing, and trust me it would be much easier.  MTF Mapper is excellent because it allows one to work directly on the green channel raw data, without introducing demosaicing blur into the mix.  The author, Frans van den Bergh is a very smart and helpful guy whose blog got me going on this frequency domain trip. On the other hand it is an open source command line program which is not as user friendly as commercial products.  This is the way I use it, you may not want to once you realize what's involved

1) First create a TIFF of the raw data with dcraw -D -4 -T filename.cr2;
2) Open filename.tiff in a good editor and save a 400x200 pixel crop (horizontal edge, 200x400 vertical) of the central edge you'd like to analyze in a file called, say, h.tif making sure the top left most pixel of h.tif corresponds to a Red pixel in the original raw data (use RawDigger for that)
3) run the command line "mtf_mapper h.tif g:\ -arbef --bayer green -t x", assuming that you are working in directory g:\ and x is the threshold (your last two images worked with x=0.5)
4) MTF Mapper produces a number of text files and Annotate.png: open mtf_sfr.txt in Excel using the data import function.  There should be four lines with 65 values each.  The first value of each line is the angle of the edge (ideally it should be somewhere between 5-10 degrees).  The remaining 64 values are the MTF curve in 1/64th cycles/pixel increments, starting with 0 cy/px which clearly has an MTF value of 1.  Choose the line that corresponds to the edge (see the Annotate.PNG file) and plot it.

Voila', that's the MTF curve of just the two green raw channels.  Alternatively send me the file (one at a time please) and I'll do it for you - I've got batch files for most of this but they reflect how I work, call other programs and they are not easy to explain or set up if starting from scratch.

Voila indeed .  Or 'Just-like-that'  as Tommy Cooper would have said.

I can do all of that with Photoshop and Imatest fairly easily (split into R,G,B images, select the 200x400 edge, do the MTF) ... but what I can't do is to find out if the top leftmost pixel was a red pixel ... without RawDigger, that is, which I don't have.  Is that very necessary? (because it adds quite a bit of complication).  As the image has been demosaiced, I don't see what difference it would make what color the pixel was (using Imatest, that is), but perhaps it does?

Certainly doing the MTF on the green channel rather than the RGB image did improve the reading.

At this level, surely the light source would be pretty important?  Higher frequency better??

Cheers,

Robert

[Jack Hogan]

As the image has been demosaiced, I don't see what difference it would make what color the pixel was (using Imatest, that is), but perhaps it does?

Ah, but that's the point.  It isn't demosaiced.  The way I showed you it is just the two green raw channels straight off the sensor.

Jack

[Jim Kasson]

At this level, surely the light source would be pretty important?  Higher frequency better??

Let me jump in here to give you some of the things I've found out in the last six or 8 months of doing slanted edge target shooting.

Specular highlights are your enemy. One way to get rid of them is to use matte paper, but that makes it more difficult to achieve high spatial frequencies on the target itself.

Another way to reduce them is to have a diffuse light source. Soft boxes are good. Bouncing the light off a matte reflector is good.

Camera motion is your enemy. Fast shutter speeds help (sometimes -- faster isn't always better). Stiff tripods help. But the thing that helps the most is short duration electronic flash in a dark room. I use the Paul Buff Einsteins, which can produce a t.1 below 100 usec when set up right.

Along those lines, use trailing curtain synch to reduce shutter shock effect.

Mirror locked up? Absolutely.

Cable/electronic release or self-timer or shutter delay. For sure.

EFCS. If you got one, use it.

Cutting thin black plastic with a paper cutter can sometime produce a clean edge. If you can find die-cut plastic, even better.

Good luck,

Jim

[Robert Ardill]

Ah, but that's the point.  It isn't demosaiced.  The way I showed you it is just the two green raw channels straight off the sensor.

Jack

I get it

Robert

[Robert Ardill]

Specular highlights are your enemy. One way to get rid of them is to use matte paper, but that makes it more difficult to achieve high spatial frequencies on the target itself.

I can second that!

Camera motion is your enemy. Fast shutter speeds help (sometimes -- faster isn't always better). Stiff tripods help. But the thing that helps the most is short duration electronic flash in a dark room. I use the Paul Buff Einsteins, which can produce a t.1 below 100 usec when set up right.

Do you use a long exposure on the camera and use the flash only (so no shutter movement at all?).  Sounds like a good trick!

Cutting thin black plastic with a paper cutter can sometime produce a clean edge. If you can find die-cut plastic, even better.

That sounds like a good idea too!

Thanks!

Robert

[Jim Kasson]

Do you use a long exposure on the camera and use the flash only (so no shutter movement at all?).  Sounds like a good trick!

On most cameras, with most lenses, it doesn't take that long of an exposure to let the first curtain vibrations die down. 1/25 with trailing curtain synch will usually do it. 1/8 would be even safer, if you don't want to run tests. The faster the shutter speed the more residual room light is allowable.

Jim

[Robert Ardill]

On most cameras, with most lenses, it doesn't take that long of an exposure to let the first curtain vibrations die down. 1/25 with trailing curtain synch will usually do it. 1/8 would be even safer, if you don't want to run tests. The faster the shutter speed the more residual room light is allowable.

Jim

Hi Jim,

I don't see any way of doing this on the 1Ds3.  I can change from 1st to 2nd curtain sync for flash of course, but there's no delay that I can change.  Would either 1st or 2nd make any difference to camera shake?  I wouldn't have thought so.

I thought that what you suggested is to photograph in a very dark room with a long exposure (say 3 seconds) and manually trigger the flash after a second or so ... in which case there would be no mirror lock-up needed and no issue with shutter vibration.  Did I misunderstand you?

Cheers

Robert

[Jim Kasson]

I don't see any way of doing this on the 1Ds3.  I can change from 1st to 2nd curtain sync for flash of course, but there's no delay that I can change.  Would either 1st or 2nd make any difference to camera shake?  I wouldn't have thought so.

You can lock the mirror up on the Canon, right? You can use an electronic release or the self-timer to trip the shutter, right? If you can do that, the only vibration you have to worry about is the first shutter curtain. By using a longish exposure and trailing curtain synch, you can let the vibrations from the opening of the first curtain die down before the flash goes off.

Does it make a difference? It did for me with the Sony a7R, but it's got a particularly problematical shutter. If you can get your flash duration well under a millisecond, it's probably a "can't hurt, might help" thing.

Here's another thought. Does your camera have EFCS? Use that.

I thought that what you suggested is to photograph in a very dark room with a long exposure (say 3 seconds) and manually trigger the flash after a second or so ... in which case there would be no mirror lock-up needed and no issue with shutter vibration.  Did I misunderstand you?

That works, too, but it's easier to let the camera trigger the flash at the end of the exposure with trailing curtain synch.

Jim

[Robert Ardill]

You can lock the mirror up on the Canon, right? You can use an electronic release or the self-timer to trip the shutter, right? If you can do that, the only vibration you have to worry about is the first shutter curtain. By using a longish exposure and trailing curtain synch, you can let the vibrations from the opening of the first curtain die down before the flash goes off.

Does it make a difference? It did for me with the Sony a7R, but it's got a particularly problematical shutter. If you can get your flash duration well under a millisecond, it's probably a "can't hurt, might help" thing.

Here's another thought. Does your camera have EFCS? Use that.


That works, too, but it's easier to let the camera trigger the flash at the end of the exposure with trailing curtain synch.

Jim

Hi Jim,

As far as I know the 1Ds3 doesn't have EFCS.  Of course it has mirror lock-up etc, and I can trigger the camera remotely.  But in the test shots I've done these seem to make little difference, so I wonder if shutter shake would be significant.  It is a very heavy camera and I have a good tripod, so I suspect that the image softness I'm seeing has more to do with my lenses not being as good as they should be, and possibly even more to my test conditions (like the target print and lighting) not being too good.

Robert

[Jim Kasson]

As far as I know the 1Ds3 doesn't have EFCS.  Of course it has mirror lock-up etc, and I can trigger the camera remotely.  But in the test shots I've done these seem to make little difference, so I wonder if shutter shake would be significant.  It is a very heavy camera and I have a good tripod, so I suspect that the image softness I'm seeing has more to do with my lenses not being as good as they should be, and possibly even more to my test conditions (like the target print and lighting) not being too good.

You're probably right. As your lenses get better, you may want to revisit this issue.

http://blog.kasson.com/?p=4359

Jim

[Robert Ardill]

You're probably right. As your lenses get better, you may want to revisit this issue.

http://blog.kasson.com/?p=4359

Jim

Yes, well it's hard to know without doing a whole lot more testing.  The lenses I have are very good - admittedly not L-series primes, but the 100mm macro f2.8 and 50mm macro 2.5 are both excellent lenses and my zoom lenses (24-105 F4L and 70-200 F4L IS) are also very good. I should be getting well over 3000 lw/ph from all of these lenses, so I think my testing technique is more to blame than the lenses.  At any rate the results I get in the field are very acceptable to me, so I don't plan to change the lenses.

Out of interest, I did a test with a Samyang 14mm lens and I got a 10-90% edge rise of 1.41 pixels ... which is better than the results I got from my Canon lenses!  The only real difference is that the focal length to lens/print distance ratio is over 100:1 with the Samyang, whereas with the other lenses the ratio is more in the region of 15-30:1, so most likely my test print is to blame.

It's certainly been very interesting to see how effective the deconvolution is, using FocusMagic, for example. It's one thing to look at an image and another to see the MTF and edge rise on a test chart - but when both are telling you that you are getting as good resolution as is possible, well then it's not hard to be convinced that this is the way to go.

Robert

[Jack Hogan]

It's one thing to look at an image and another to see the MTF and edge rise on a test chart - but when both are telling you that you are getting as good resolution as is possible, well then it's not hard to be convinced that this is the way to go.

That's what this is all about, figuring out how to make the equipment deliver what it is supposed to - and no less.

[Robert Ardill]

That's what this is all about, figuring out how to make the equipment deliver what it is supposed to - and no less.

Yes, absolutely.  Actually, the reason I bought Imatest Studio in the first place is that I had bought a 17-40mm F4L lens that I wasn't very happy with and the testing showed the weakness in the lens - so I got rid of it. Then I tested the 24-70F2.8L lens I had and that came up short too. So I got rid of that one. I then bought a 24-105 F4L lens and returned two of the lenses before I got one I was reasonably happy with.  Anyway, the point I'm making is that if one is prepared to print large charts and go to the trouble of setting up the tests properly that there is a lot to be learnt from these tests.  Center focus is just one thing ... and not necessarily the most important if the point of interest of your image is off-center.  So there could be a trade off between accepting a softness subject focus ... or taking a wider-angle shot and cropping it, for example.  Other things like mirror lock-up, a good tripod etc., also make a big difference, of course.

Anyway, getting back to the subject of modelling the camera system - is it still worth doing do you feel?  If so I can print a large chart and set up the equipment to get the best sharpness possible.  But if you think that there is no real practical benefit possible because the system is too complex and there are too many unknowns, well then I'll save myself the trouble (and you too .

I guess my question would be: if you can pretty accurately get the PSF just for the sensor and demosaicing, do you think there is a benefit from deconvolving the image for this first (before doing a guesstimate lens deblur).  I think you have already said that you do think that this would be good (but the problem is getting the pretty accurate PSF!).

If you do think it's worth carrying on, would it help to take test shots with a very simple lens like the 50mm Macro F2.5? (to eliminate the lens complexity as much as possible).

Also, since with MTF Mapper you can work directly on the raw channels, could you not then model the blurring just due to the demosaicing, and so deconvolve that part of the blurring separately?

Robert

[Jack Hogan]

I guess my question would be: if you can pretty accurately get the PSF just for the sensor and demosaicing, do you think there is a benefit from deconvolving the image for this first (before doing a guesstimate lens deblur).

Hi Robert,

To me this frequency domain trip is mainly a learning exercise in how lenses and cameras interact with detail in the scene.  I am focusing in on the hardware to limit the number of variables involved and because it may come in handy when evaluating what equipment to purchase. The way I am using the slanted edge method does not deal with demosaicing at all so I am mainly dealing with the lens (diffraction and blur), the AA and pixel pitch as you have probably seen in graphs produced with the model.  For modeling simplicity I concentrate on the center of the image which is not necessarily an indication of lens performance throughout the FOV.  As you see there are many variables unaccounted for that can contribute to the formation of a Gaussian MTF.  I think the average bloke does just fine by simply playing with the sliders in existing tools and reading reviews on decent sites like DxOmark.com and lenstip.com.

On the other hand now that I understand things a little better I think deconvolution plug-in designers could work a little harder at producing more flexible and controllable products.  For instance, are we sure that the deconvolution PSF used in a DSLR with an old-style AA would be suitable as-is with just a different radius/strength on a brand spanking new one sans AA?  I personally think not.

Jack

[Bart_van_der_Wolf]

On the other hand now that I understand things a little better I think deconvolution plug-in designers could work a little harder at producing more flexible and controllable products.  For instance, are we sure that the deconvolution PSF used in a DSLR with an old-style AA would be suitable as-is with just a different radius/strength on a brand spanking new one sans AA?  I personally think not.

Hi Jack,

I agree. And, as you have experienced yourself, the PSF is not always circularly symmetric (isotropic). There are several things that can  be updated for a more modern sharpening tool. To avoid complexity for inexperienced users, there are lots of things that can be done in the Human Interface design to help with finding the optimal settings. Starting with more sensible (based on aperture used) defaults is but one of them.

Cheers,
Bart

[Robert Ardill]

Hi Jack and Bart,

First of all, many thanks for all of your help ... I for one have learnt a whole lot from this discussion and I am quite sure that my 'sharpening' will be a whole lot better than it was in the past as a result.

I can't help feeling that there is still quite a lot that could be done to selectively remove blurring due to, for example, the AA filter ... in isolation from the other causes such as the lens.  Perhaps this is too complex for us to do (well certainly I would need to do a whole lot of relearning of my maths before I could even begin to attempt it!), but it surely should be possible for the camera manufacturers, say.  After all, it isn't rocket science at this stage to extract the relevant frequencies in the frequency domain by comparing with and without the AA filter.  Once that is known then it should be relatively straightforward to remove the unwanted signals (says I glibly . Whether it would be worth doing or not I don't know ... but it might be an alternative to buying two cameras, one with an AA filter and one without.

At any rate this is as far as it's practical to go at this point, but it is great that there are now tools out there that go quite a long way to help us working photographers .

Cheers,

Robert