Detail preserved after downsampling?

[ErikKaffehr]

Hi,

Imatest, the program I used uses a concept of standardized sharpening to take into account the different amount of sharpening in the processing pipeline. Black lines are measured MTF data and the red ones are corrected for "standard" sharpening. As both data sets were converted using same settings in Lightroom I'd suggest that the black curves would be comparable.

Best regards
Erik

Erik,

    Thanks for all those graphs.

What do the red lines marked with suffix  "(corr)" signify?

Do the results showing sharpening with bicubic down sampling go with the comment in the thread on "N7 downsampled vs M9" http://www.luminous-landscape.com/forum/index.php?topic=60303.msg486061#msg486061 about the hazards of comparing downsampled to unsharpened images?

Or is it instead that by passing through JPEG conversion first, as in your first set of graphs, some sharpness advantage of the D3X files is lost?

Either way, it makes me realize how difficult resolution and sharpness comparisons are due to the intervention of different demosaicing algorithms and such.


P.S. I found the following reference explaining "(corr)". Those curves are with some standardized sharpening, which tries to compensate for the different sharpening in images from different sources. So this does seem the best comparison to use, and it also indicates the sharpening in bi-cubic down sampling from the D3X, since this sharpened red curve is no higher than the unsharpened curve.

http://www.imatest.com/docs/sharpness_comparisons/

[ErikKaffehr]

Luke,

Regrading comparing the TIFF and JPEG cases I would say that JPEG artifacts is a red herring. It is a very high quality JPEG conversion and I don't think possible JPEG artifacts upset Imatest.

I hope that other posters, more knowledgeable than me will chime in. Here is what I saw:

1) The JPEG images:



Here I can see that the two conversions are very similar. In this case I wouldn't expect to see benefits from the higher resolution of the D3X nor significant artifacts.

2) Bicubic sharper



In this case I see two key differences, one is the higher MTF at medium frequencies in the scaled down image. The other one is that MTF around Nyquist is rather high in the downscaled image. According to most sources MTF exceeding 20-30% at Nyquist frequency would cause problems with aliasing.

3) Aliasing



Both images show contrast inversion on the resolution target. The D3X (on top) shows it around 13  while D3S (bottom) shows it around 25. The D3X image is using an old version of the target photographed at double distance.

Both images are here at same size, the D3X image downsampled using bicubic sharper. Note that "Bicubic sharper" added some visible halos around the numbers.

I would recommend these two articles by Bart van der Wolf:

http://bvdwolf.home.xs4all.nl/main/foto/down_sample/down_sample.htm

http://bvdwolf.home.xs4all.nl/main/foto/down_sample/example1.htm


Best regards
Erik

This really helps to address some key questions about the benefits or otherwise of high resolution sensors.  

I too was curious about MTF50(CORR).  I see Imatest has a reference page on measuring sharpness, but I couldn't find reference to it there.  Perhaps I needed to drill down further.
http://www.imatest.com/docs/sharpness/

I'm focusing on the TIF test, which doesn't involve questions about JPG artifacts and conversion.

It seems that the downsampled D3x image has significantly higher MTF than the native D3s image.  If I'm taking MTF50(CORR) to be the relevant comparison, there seems to be some advantage to the D3x image just beginning around 1000 LP/PH, and becoming somewhat significant by 1500 LP/PH.  I had not expected to see such a broad difference, but more of a difference up around the Nyquist frequency of the D3s sensor.  Am I reading it correctly?  It makes me wonder how other factors of conversion, downsampling method, sharpening, etc, figure into the results.

Do I also read correctly that the edge profile is sharper on the D3x, but also has a "ring," possibly an artifact of bicubic sharper?  I wonder how that would fare using Lanczos windowing?

This is really useful data, Erik, and I'm looking forward to seeing your interpretation.

[LKaven]

Hmm.  Erik, is it your feeling that

(1) all of the increase in MTF in the bicubic sharpening case is explained wholly by the sharpening?

(2) the downsampled JPG conversion accurately represents the RAW image such that Imatest results are not being skewed by compression artifacts? 

I'm always wary of bicubic sharper down downsampling.  It always seems to produce a ring and I wouldn't use it as standard.  But this makes me wonder what is the optimal way to downsample an image in a case like this.

For example, among things various people have mentioned as a part of their regimen: would one deconvolve (to solve for the OLPF) as a standard sharpening before downsampling, would one blur slightly to reduce antialiasing before downsampling, would one use say a Lanczos windowed SINc to downsample?  What would be optimal?

Finally, there's one thing that an Imatest target doesn't capture, and that is color gradation and fidelity, the enhancement of which adds to the perception of detail. 

[Bart_van_der_Wolf]

Luke,

Regrading comparing the TIFF and JPEG cases I would say that JPEG artifacts is a red herring. It is a very high quality JPEG conversion and I don't think possible JPEG artifacts upset Imatest.

I hope that other posters, more knowledgeable than me will chime in. Here is what I saw:

1) The JPEG images:
[...]
Here I can see that the two conversions are very similar. In this case I wouldn't expect to see benefits from the higher resolution of the D3X nor significant artifacts.

Indeed, quite similar.

2) Bicubic sharper
[...]
In this case I see two key differences, one is the higher MTF at medium frequencies in the scaled down image. The other one is that MTF around Nyquist is rather high in the downscaled image. According to most sources MTF exceeding 20-30% at Nyquist frequency would cause problems with aliasing.

The downsampling filter did boost the medium frequencies but also introduced halos and aliasing artifacts. I wouldn't use Photoshop's "Bicubic sharper" because of that behavior. Regular bicubic is much better behaved, but by no means ideal. A Lanczos filter does much better in keeping sharpness, and avoiding aliasing artifacts as much as possible. 

3) Aliasing
[...]
Both images are here at same size, the D3X image downsampled using bicubic sharper. Note that "Bicubic sharper" added some visible halos around the numbers.

The "Bicubic sharper" halos are as predicted in my articles that you linked to. It beats me why Photoshop still doesn't offer a choice for Lanczos when downsampling, where Lightroom has a better (almost as good as Lanzos) downsampling filter.

Cheers,
Bart

[Bart_van_der_Wolf]

Hmm.  Erik, is it your feeling that

(1) all of the increase in MTF in the bicubic sharpening case is explained wholly by the sharpening?

One could repeat the downsampling with Bicubic (regular) and see what difference the 'sharper' version introduces, or one could compare with a Lanczos filtered downsample, which is considerd close to ideal.

(2) the downsampled JPG conversion accurately represents the RAW image such that Imatest results are not being skewed by compression artifacts?

There are differences, e.g. in actual gamma and tonemapping and depending on the degree of compression, but Imatest attempts to reduce the influence of those by using oversampling, thus benefitting from statistical averaging of errors. The best input for Imatest are Raw conversions with linear gamma.  

For example, among things various people have mentioned as a part of their regimen: would one deconvolve (to solve for the OLPF) as a standard sharpening before downsampling, would one blur slightly to reduce antialiasing before downsampling, would one use say a Lanczos windowed SINc to downsample?  What would be optimal?

Deconvolution sharpening before downsampling doesn't help, and could possibly hurt the result. Preblurring is mandatory to reduce aliasing artifacts, and Lanczos is considered to be close to optimal.

Finally, there's one thing that an Imatest target doesn't capture, and that is color gradation and fidelity, the enhancement of which adds to the perception of detail. 

Imatest performs all sorts of tests, including color accuracy.

Cheers,
Bart

[LKaven]

Bart, what would you consider to be the optimal pre-blur radius for downsampling?

[Bart_van_der_Wolf]

Bart, what would you consider to be the optimal pre-blur radius for downsampling?

Luke,

Anything that's going to be smaller than 1 pixel in the downsampled image will add to the aliasing risk. So with, say 7:1 downsampling, a radius of 3.5 should theoretically cover it, but what weight to give to those pixels to preserve some of the information / resolution / average color ..., enters Lanczos.

A Gaussian blur as implemented in Photoshop would need to have a much smaller radius, because it will also include many more pixels outside of the required radius, thus causing a very soft result. As I described in the articles that Erik linked to, for a downsampling factor of 7, I would use a radius of about 7 x 0.25 = 1.8 for a quick fix before downsampling with PS Bicubic, and apply some sharpening to the result. But it'll be inferior to a straight Lanczos downsample in ImageMagick.

Cheers,
Bart

[madmanchan]

There is a tradeoff in the resample kernel.  You can preserve more details with a so-called "sharpening" kernel like the one used in Bicubic Sharper but you may see edge-halo artifacts in some cases, depending on the magnitude of the negative lobe in the kernel.  If you use a kernel without any negative lobes (like a Gaussian) you can reduce aliasing, but your results will also be softer.  You can try to do the optimal thing for aliasing using a (windowed) sinc filter like Lanczos, but then you can (often will) get other problems -- e.g., ringing (*). 

There are also limits of detail preservation, regardless of what resample method you choose.  As a simple example, if you choose your output image to be 1 pixel in size (1 pixel high, 1 pixel wide), I think it should be clear you're not going to see much detail regardless of what resolution you started from, or what resample kernel you used.   

Eric


(*) In other words, just because an kernel is optimal or nearly optimal for one property -- e.g., aliasing -- does not mean it is optimal for other aspects of image quality.  See Don Mitchell's SIGGRAPH paper on reconstruction filters, for example.

[hjulenissen]

Deconvolution sharpening before downsampling doesn't help, and could possibly hurt the result. Preblurring is mandatory to reduce aliasing artifacts, and Lanczos is considered to be close to optimal.

This is interesting. I would have thought that the following steps would be closer to optimal:
1. Use deconvolution to obtain an array properly lowpass-filtered according to Nyquist (i.e. maximize flatness in the passband below fs/2, maximize attenuation above fs/2).
2. Use a proper resampling filter (I consider the analog prototype filter shape to be integral to the resampling process). Lanczos order2/3 or some other similar passband/stopband/spatial-ringing trade-off.
3. Sharpen to taste (and according to deficiencies in output medium).

-h

[LKaven]

This is interesting. I would have thought that the following steps would be closer to optimal:
1. Use deconvolution to obtain an array properly lowpass-filtered according to Nyquist (i.e. maximize flatness in the passband below fs/2, maximize attenuation above fs/2).
...

That's what I thought; but I admit, I don't have a clear reason for it.  I wonder why Bart doesn't recommend that. 

[BJL]

Erik,

    About your comment that

...  MTF around Nyquist is rather high in the downscaled image. According to most sources MTF exceeding 20-30% at Nyquist frequency would cause problems with aliasing.

I would think that this aliasing worry only applies to the Nyquist frequency of the signal before down sampling. But I appeal to the local signaling processing authorities for advice on this!

[hjulenissen]

Erik,

    About your comment thatI would think that this aliasing worry only applies to the Nyquist frequency of the signal before down sampling. But I appeal to the local signaling processing authorities for advice on this!

Not sure if this is what you are asking, but generally when you are doing resampling from rate r1 to rate r2, the lowpass filter needs to have sufficient attenuation at and above min(r1/2,r2/2).

If a 1000x1000 pixel image, when downscaled to 100x100 pixels shows a lot of energy at 50 lp/pw (or ph) for wide-band input, I believe there is reason to be concerned about aliasing.

-h

[BJL]

Not sure if this is what you are asking, but generally when you are doing resampling from rate r1 to rate r2, the lowpass filter needs to have sufficient attenuation at and above min(r1/2,r2/2).

OK, I see that ... but is it not possible, starting with a signal at a higher sampling rate r1, to filter with a near brick-wall filter in the digital domain, cutting off at r2/2, so that the guideline of MTF < 20-30% does not apply? I am guessing that this 20-30 guideline relates to the limitations of analogue low-pass filtering (as with OLPF's), not to the greater flexibility possible with DSP. Though it probably does not matter for the case at hand, where no MTF curve looks vaguely like a brick-wall, and all of them have significant signal above f(Nyquist).

Another point: from a 24MP Bayer CFA sensor, I think that the pixel spacing relevant to the Nyquist frequency for green is the spacing of the green pixels, which is sqrt(2) times pixels width (and for R and B the Nyquist frequency is lower) while for 12MP down-sampled output with full RGB data at each point, the relevant spacing is the width of the new, bigger pixels ... which is sqrt(2) times the original pixels, and so equals the spacing of green pixels. So it seems that the Nyquist frequency is no lower for the downsampling 12MP full RGB output than for the 24MP Bayer CFA input, and is if anything higher for the R/B "colour" information, where aliasing is most noticeable.

[hjulenissen]

OK, I see that ... but is it not possible, starting with a signal at a higher sampling rate r1, to filter with a near brick-wall filter in the digital domain, cutting off at r2/2, so that the guideline of MTF < 20-30% does not apply? I am guessing that this 20-30 guideline relates to the limitations of analogue low-pass filtering (as with OLPF's), not to the greater flexibility possible with DSP. Though it probably does not matter for the case at hand, where no MTF curve looks vaguely like a brick-wall, and all of them have significant signal above f(Nyquist).

Talking only about linear, space-invariant filtering here.

Brick-wall filters have a tendency to have large kernels, leading to issues with spatial ringing. The "ideal" filter from a passband/stopband optimization view would be an infinite sinc(x) function. I believe that the same can be achieved in practice for a finite image by processing in the 2-d DFT domain.

"Lanczos" filters are a trade-off where the infinite sinc(x) function is windowed so as to narrow down the kernel, leading to less brick-wall-like frequency domain behaviour. This is an example of "design by windowing" as opposed to directly calculating the kernel from some set of constraints. Usually, direct synthesis has more degrees of freedom to satisfy the constraints, but it seems that lanczos are pretty close to what most people prefer.

This whitepaper from 21 years ago sums it up pretty well, I think:
http://www.realitypixels.com/turk/computergraphics/ResamplingFilters.pdf

See also this:
http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP092.pdf

In these tests, two types of filter frequency characteristic were used. One was a practical implementation of a well-known sharp-cut filter (the ITU Rec. 601[1] channel-filter), and the other a softer filter that avoids ringing or overshoots (the Modified Raised Cosine filter).
...
Each of these two filters consists of 43 taps, and may be called a prototype. The Rec. 601 prototype filter has a rise-time (the 10 – 90 % edge rise-time) of 2 pixels and the Modified Raised Cosine prototype filter has a rise-time of 4 pixels.
...
the filtering process was modified to include a conversion to light-proportional signals by raising the unfiltered signals to a power, filtering and then raising the result to the reciprocal power. The power chosen was 2.35
...
the ‘ringing’ on edges caused by the Rec. 601 filtering serves to give a subjective impression of increased sharpness which allows the rise-time to be increased to give the same degree of apparent sharpness,
...
The most important result is that, when using the sharp-cut “Rec. 601” filter, the average visible limit of edge rise-time lies in the range 0.997 to 1.111 minutes of arc, with a confidence of 95%. Interestingly, the generally accepted figure for visual acuity is approximately 1 minute of arc.

Another point: from a 24MP Bayer CFA sensor, I think that the pixel spacing relevant to the Nyquist frequency for green is the spacing of the green pixels, which is sqrt(2) times pixels width (and for R and B the Nyquist frequency is lower) while for 12MP down-sampled output with full RGB data at each point, the relevant spacing is the width of the new, bigger pixels ... which is sqrt(2) times the original pixels, and so equals the spacing of green pixels. So it seems that the Nyquist frequency is no lower for the downsampling 12MP full RGB output than for the 24MP Bayer CFA input, and is if anything higher for the R/B "colour" information, where aliasing is most noticeable.

If you have aliasing at the capture stage, you may struggle with it later on no matter how much you downsample. Imagine a classical western-movie with wagon-wheels that seem to roll backwards or stand still due to the 24fps capture rate. If the movie is "resampled" to 12fps or 60fps, those problems will still be a problem, since the violation of Nyquists sampling theoreme means that the information is ambiguous.

I do believe that a "perfect" OLPF would have a higher spatial cutoff for the green channel than red/blue, but I imagine that it would be hard to make. As each sensel is not a point-sampler, but have an extent of its own, the optimal PSF for the color-channel-dependent OLPF may be non-trivial.

-h

[joofa]

Those of you who are photographers and want image detail (with or without downsampling) on real images, which you capture with your camera (and not some junky nerd lab rat who can't think beyond decades old and outdated MTF charts  ), can use my freely available detail measure plugin for Photoshop CS3 or Photoshop Elements 8, from the website link in my signature. This tool will let you run detail measure on the whole image or an area selection using the PS marquee tool, as shown below, which is useful for finding detail measures in different parts of an image:

Note that detail measure is reported in [0-1] range with higher numbers indicative of more detail. I ran the JDM on a whole image shown above with various downsmapling methods in PS and got the following:

Origianl image: 0.0835
Downsampling using:
- Nearest Neighbor: 0.1156
- Bilinear: 0.0879
- BiCubic: 0.0750
- BiCubic Smoother: 0.0617
- BiCubis Sharper: 0.0853

You can also download the free Raw Import plugin from the same download page and play around with detail measure (and/or other analyses) on separate R,G1,G2,B channels using the Raw Import plugin. And, free yourself of the issues such as which MTF chart to use, distance to an MTF chart, slanted this or that, sensor size, image size, size normalized or not, hey wait I can't see the lines, oops the light is not uniform, blah, blah, blah ...

Hope you find it useful.

Sincerely,

Joofa

[ErikKaffehr]

Hi,

So your tool finds that "bicubic sharper" yields more detail. But it is real detail or fake detail?! In the case I tested it clearly produced visible haloes.

Best regards
Erik

Those of you who are photographers and want image detail (with or without downsampling) on real images, which you capture with your camera (and not some junky nerd lab rat who can't think beyond decades old and outdated MTF charts  ), can use my freely available detail measure plugin for Photoshop CS3 or Photoshop Elements 8, from the website link in my signature. This tool will let you run detail measure on the whole image or an area selection using the PS marquee tool, as shown below, which is useful for finding detail measures in different parts of an image:

Note that detail measure is reported in [0-1] range with higher numbers indicative of more detail. I ran the JDM on a whole image shown above with various downsmapling methods in PS and got the following:

Origianl image: 0.0835
Downsampling using:
- Nearest Neighbor: 0.1156
- Bilinear: 0.0879
- BiCubic: 0.0750
- BiCubic Smoother: 0.0617
- BiCubis Sharper: 0.0853

You can also download the free Raw Import plugin from the same download page and play around with detail measure (and/or other analyses) on separate R,G1,G2,B channels using the Raw Import plugin. And, free yourself of the issues such as which MTF chart to use, distance to an MTF chart, slanted this or that, sensor size, image size, size normalized or not, hey wait I can't see the lines, oops the light is not uniform, blah, blah, blah ...

Hope you find it useful.

Sincerely,

Joofa

[joofa]

Hi,

So your tool finds that "bicubic sharper" yields more detail. But it is real detail or fake detail?! In the case I tested it clearly produced visible haloes.

Best regards
Erik

It measures the "perceived" sharpness in an image, even if introduced by image filtering. So actually it does agree with PS BiCubic Sharper being "sharper" than BiCubic and BiCubic smoother. So you can use it to find a measure of "sharpening ratio" after applying a sharpening step on an image:

Joofa

[joofa]

I ran the JIDM on a whole image shown above with various downsmapling methods in PS and got the following:

Origianl image: 0.0835
Downsampling using:
- Nearest Neighbor: 0.1156
- Bilinear: 0.0879
- BiCubic: 0.0750
- BiCubic Smoother: 0.0617
- BiCubic Sharper: 0.0853

Joofa

According to JIDM, apparently downsampling (50%) by BiCubic and BiCubic Smoother in Photoshop actually reduces image detail (sharpness) compared to the original image, where as BiCubic sharper results in a little sharper than the original. What this reveals is that PS implementation of BiCubic and BiCubic Smoother is attenuating more signal in the passband than perhaps what is desired, for the asked downsampling ratio (50%). Apparently, as a conjecture, PS implementation of these algorithms might shown quite different response for a different downsampling ratio (guess based upon BiCubic theory) - the bottom line being PS downsampling is not good enough.

Joofa

[Bart_van_der_Wolf]

This is interesting. I would have thought that the following steps would be closer to optimal:
1. Use deconvolution to obtain an array properly lowpass-filtered according to Nyquist (i.e. maximize flatness in the passband below fs/2, maximize attenuation above fs/2).

In actual practice there isn't something like a brick-wall filter, it would also introduce ringing. We're forced to seek a compromise.

2. Use a proper resampling filter (I consider the analog prototype filter shape to be integral to the resampling process). Lanczos order2/3 or some other similar passband/stopband/spatial-ringing trade-off.

As I have been saying, a Lanczos windowed Sinc filter is close to optimal.

3. Sharpen to taste (and according to deficiencies in output medium).

Yes, but be careful because you'll (re)introduce stairstepping (=aliasing artifacts).

Cheers,
Bart

[Bart_van_der_Wolf]

Erik,

    About your comment that I would think that this aliasing worry only applies to the Nyquist frequency of the signal before down sampling. But I appeal to the local signaling processing authorities for advice on this!

That's correct, it only tells that there is some signal (e.g. stair-stepping, or accidental signal alignment with the sensel grid) which is a potential cause for aliasing upon further postprocessing.

Cheers,
Bart