The Future of CCD Sensors

[eronald]

I dare you.  

What, to demonstrate even more convincingly that I'm an idiot and a show off ?  I thought I'd done that quite nicely by myself.

Or shall we actually talk about the way what we measure is/should be related to what we perceive as good quality imaging? D'you have a topic? Thing is, if we manage to get the smarter people here to chime in a bit, about what should be measured, we might gain something. One of my professors, a physicist by training, once invited me to dinner with his family and pointedly told me that "Les mathématiques sont là pour enfoncer le portes ouvertes" ie. mathematics serves to fully force open some doors which intuition or experimental work has already unlocked.

In this sense, people talking about eg. the "imaginary" perceived difference between CCD and CMOS is useful, because at some point one of the engineers from a camera corp lurking on this forum may think "Hey, I know what they're talking about ..." and then ask to have this aspect added to a standard measurement suite.

As an example, looking at imagery, I believe that CCD cameras have been historically better on texture like skin tone; however to improve skin tone one would need a good characterisation of how we see texture. My belief is we need to move from Fourier to wavelets. Here are some references, which show what can be captured by the current state of the art applied maths point of view.

http://www.image-engineering.de/index.php?option=com_content&view=article&id=570&Itemid=210
https://www.ceremade.dauphine.fr/~peyre/numerical-tour/tours/ (scroll to image processing)

One problem is that the applied maths crowd are often behind the technology; in fact these days one could argue that many of the research academics are situated on a different continent from the commercial technology and don't really have direct access to most of its characteristics.

Edmund

[jerome_m]

My "dare" was not really serious, but I like the concepts in the first document you cited. Seems to be an approach about measuring texture loss which could give good results.

[ErikKaffehr]

Hi,

The methodology described in the first article is used by the German monthly "Color Photo", but I think they only use it on JPEGs.

I don't think we see similar noise reduction in raw files, although some noise reduction in raw have been noted by DxO on some cameras. They check for this routinely.

Best regards
Erik

[eronald]

My "dare" was not really serious, but I like the concepts in the first document you cited. Seems to be an approach about measuring texture loss which could give good results.

 I dunno. I'm very interested in some of the wavelet transform ideas - not the details, of course I don't understand those but I'm a bit put off by the fact that discrete wavelet transforms seem not to be translation invariant, so I think I'm going to have to go back to earlier work. I just read through the first half of the Wong book on discrete Fourier analysis which shows nicely how the frequency and time-frequency models interact.  The internet is incredible as a library.

Edmund

[eronald]

Hi,

The methodology described in the first article is used by the German monthly "Color Photo", but I think they only use it on JPEGs.

I don't think we see similar noise reduction in raw files, although some noise reduction in raw have been noted by DxO on some cameras. They check for this routinely.

Best regards
Erik

We may not be seeing noise reduction on raw, but we sure are seeing a considerable amount of subjective texture loss.
It would be nice to be able to translate this subjective perception into a measurable quality.

Edmund

[hjulenissen]

I dunno. I'm very interested in some of the wavelet transform ideas - not the details, of course I don't understand those but I'm a bit put off by the fact that discrete wavelet transforms seem not to be translation invariant, so I think I'm going to have to go back to earlier work. I just read through the first half of the Wong book on discrete Fourier analysis which shows nicely how the frequency and time-frequency models interact.  The internet is incredible as a library.

Edmund

I think that wavelets were hyped in the 90s. Although the rationale behind and continous behaviour of wavelets may excite physicists and mathematicians, I think that the practical possibilities have (up until now) been quite similar to e.g. the time/frequency analysis filterbanks that have been used in e.g. electronic engineering for half a century or more.

-h

[hjulenissen]

We may not be seeing noise reduction on raw, but we sure are seeing a considerable amount of subjective texture loss.
It would be nice to be able to translate this subjective perception into a measurable quality.

Edmund

Is the cutoff for this texture loss sensel-pitch-dependant? If so, one would assume that something like the D800 can be compared to a D700 under identical conditions to figure out what kind of flaws the D700 has? It might also be interesting to compare the Leica M9 with the Leica M monochrome to see if color capability has anything to do with it.

I think there is considerable uncertainty in the Bayer model for the last spatial octave. OLPF, Color filter, demosaic all relate in complex ways, and proprietary raw developers might do all kinds of optimizations in order to satisfy their definition of good images.

-h

[eronald]

I think that wavelets were hyped in the 90s. Although the rationale behind and continous behaviour of wavelets may excite physicists and mathematicians, I think that the practical possibilities have (up until now) been quite similar to e.g. the time/frequency analysis filterbanks that have been used in e.g. electronic engineering for half a century or more.

-h

I don't know. I would agree there has been a lot of wavelet hype.  However, when we image a subject, in first analysis we image 3D texture interacting with lighting: Moss, brick, clouds, water, snow, skin are not shaded solids.

Texture is not something well described by conventional geometry. An analysis of texture implies the understanding of the object under various transforms of which scale transforms seem to be part, and whether one likes it or not this is not something which Fourier does naturally. I mean, one can do it with Fourier constructions, but Fourier does not -to me at least-  seem to be something that generates or analyses textures naturally. Speaking as a layman of course. I don't know whether wavelets really do it or have to be coerced - fractal methods seem to go some way there naturally, a lot of the CGI techniques are fractal-based- so it's interesting to look, to see what might be "catching the eye" when we perceive texture, and what the maths could be, and what one can then set as criteria for good texture reproduction.

One  phenomenon I've noticed is that we seem to like images where the reproduction noise merges in some way with the imaged texture. This is the sort of thing which is an interaction of perception, capture technology and object geometry, and is an application that cannot be invented or predicted by mathematical theory alone.

Look, I'm not claiming I understand any of this stuff, I'm just saying that it needs to be looked at, and the methods we have been using were probably not the right ones, which is why people have been pretending that any object in a scene is an assemblage of smoothly shaded solids, and as a result many cameras yield images of people and landscapes that look as if they were ... dead assemblages of smoothly shaded solids.

I think it's time I went back and read some more of those references
Pseudo-scientific discussions are the written equivalent, I guess, of cat photographs.

Edmund

[hjulenissen]

I don't know. I would agree there has been a lot of wavelet hype.  However, when we image a subject, in first analysis we image 3D texture interacting with lighting: Moss, brick, clouds, water, snow, skin are not shaded solids.

Texture is not something well described by conventional geometry. An analysis of texture implies the understanding of the object under various transforms of which scale transforms seem to be part, and whether one likes it or not this is not something which Fourier does naturally. I mean, one can do it with Fourier constructions, but Fourier does not -to me at least-  seem to be something that generates or analyses textures naturally.

If you are talking about closely emulating the HVS, then I guess that one would need to discuss with the people who do just that. I don't know those things.

If you are talking about analysing texture, I would claim that even the classic JPEG encoder does 2-d (separable) localized transforms of luma that could be linked to local texture? The thing about these transforms is that they tend to be invertible (or approximately so), meaning that they keep all information. The only thing that separate them is how they _present_ the information. A full frame FFT maintains all information, but good luck analyzing it visually to make statements about the texture of some object in the upper right corner.

Wavelets seems to be just a class of digital convolution filters with some interesting time/frequency properties. Other linear filters may have other properties that may or may not map better to a given scene, how the HVS works, and how we want to present this information for manual/automated analysis.

-h

[eronald]

Wavelets seems to be just a class of digital convolution filters with some interesting time/frequency properties. Other linear filters may have other properties that may or may not map better to a given scene, how the HVS works, and how we want to present this information for manual/automated analysis.

-h

I'd guess any linear transform in line or grid sample space can be seen as a set of convolution filters; this is a basic tenet of linear systems theory. However various representations allow you to recognize features better. Look at Fourier - the transform can be represented as a set of convolutions but perspective has changed hugely when you move from the temporal to the frequential, and some computations can be speeded up beautifully.

For texture you'd want the class to work nicely with affine transforms eg. moving an object around in space. Rotate a granite cube or a block of marble away from you and it is still seen as a granite or marble cube. I don't think Fourier is that happy about all of those affine mappings. Wavelets of course may have that difficulty as well - whatever formulation one uses  to describe textures would seem to benefit from a fractal approach in the sense that it describes power inter-relationships *between* frequencies. The obvious problem I see of course is that power spectra are not themselves linear (?), nor is the surface diffusion when an object is rotated in fixed lighting, so we now have one foot inside and one outside the linear systems theory pool.

I'm sorry if I sound a bit vague - first of all I'm geriatric, and second there seem to be "precise" mathematicians and "vague" mathematcians, and I certainly have always been at the extreme of the "vague" category, having to relearn the basics of something everytime I want to use it. Luckily these days there is the Internet and Google to teach one everything one needs.

Edmund

[bjanes]

I measured MTF for the lenses, and the Sonnar is a tiny bit better than the 70-400/4-5.6 I used at 150 mm (f/8). See enclosed figures, Sonnar on top and the 70-400/4-5.6 at bottom. The MTF measurement is a bit off-axis but still in the central area.

Erik,

A bit more fun comparing apples to oranges for pixel peepers. I photographed a slanted edge with the Nikon D800e and the Zeiss 135 mm f/2 Apo Sonnar lens and analyzed the images with Imatest using the same settings that you used, rendering with ACR 8.4.1 and PV2012, using the highest quality JPEG (since it preserves the EXIF data). No sharpening was applied. The Nikon lacks a low pass filter and has a pixel size of 4.87 μm. The Nyquist frequency is 102 lp/mm. This compares to your Sony A77 which has a pixel size of 3.9 μm and a Nyquist of 128 lp/mm. Since your Imatest results show minimal response at Nyquist, I presume a low pass filter is present.

The Sony should out-resolve the Nikon since it has a finer pixel pitch, but may need deconvolution sharpening to offset the effect of the low pass filter.

The Nikon with the Zeiss lens outresolves the Sony in these tests, illustrating that the lens is usually more important than the sensor. However the higher response at Nyquist indicates alaising. However, I rarely observe alaising in normal images, but it does show up in spades with Bart's sinusoidal star chart.

Your insights are welcome.

Regards,

Bill

[Fine_Art]

Wavelets are used extensively in the processing of astro-photos, in fact a lot of the software for that niche has wavelets built in. You might want to discuss the pros/cons with an astronomer. Mike Unsold (a mathematician) who wrote Images Plus uses wavelets in his sharpening demos that come with the program. He is helpful if you contact him.

[Jim Kasson]

Wavelets are used extensively in the processing of astro-photos...

And they form the basis of JPEG 2000.

Definitely not a fringe technology.

Jim

[ErikKaffehr]

Bill,


Thanks for responding. Just a small point, the only reason I posted the MTF curves was to demonstrate that the lenses used in the test were comparable.

What I have seen in my test is that small pixels reproduce fine natural structures more truly than larger pixels. The 6.8 micron pixel image had more fake detail than real detail, while the 3.8 micron pixel image had natural strains and very little aliasing.

In my original test I also had a 6 micron pixel camera with OLP filter. What I saw was that the OLP filter reduced colour moiré, but there was still some moiré and very much fake detail. So the conclusion I arrived at is that small pixels are more important to keep aliasing down than the OLP filter. It seems that the OLP filter is dimensioned to reduce colour aliasing, but doesn't affect grayscale moiré.

With the P45+ (6.8 my, no OLP) there is a tendency to "color aliasing" on many images. Capture One is handling that much better than LR5, but doesn't eliminate it fully. It seems "under lab conditions" that it is fully present at f/11 and needs f/16 to fully going away.

Now, the question is what we can see on screen and print. JPEG compression reduces colour information and some of the fake colour goes with it. Anyway it would not visible on web size prints. Larger prints, I don't know.

A final thought may be that resolution may not be so important, as low frequency details seem to be far more important for human vision.

As a side note, thanks for posting the MTF data. It may give me some ideas on the possible gain by a high quality lens. Could you post the raw, I would be most thankful.

Best regards
Erik

Erik,

A bit more fun comparing apples to oranges for pixel peepers. I photographed a slanted edge with the Nikon D800e and the Zeiss 135 mm f/2 Apo Sonnar lens and analyzed the images with Imatest using the same settings that you used, rendering with ACR 8.4.1 and PV2012, using the highest quality JPEG (since it preserves the EXIF data). No sharpening was applied. The Nikon lacks a low pass filter and has a pixel size of 4.87 μm. The Nyquist frequency is 102 lp/mm. This compares to your Sony A77 which has a pixel size of 3.9 μm and a Nyquist of 128 lp/mm. Since your Imatest results show minimal response at Nyquist, I presume a low pass filter is present.

The Sony should out-resolve the Nikon since it has a finer pixel pitch, but may need deconvolution sharpening to offset the effect of the low pass filter.

The Nikon with the Zeiss lens outresolves the Sony in these tests, illustrating that the lens is usually more important than the sensor. However the higher response at Nyquist indicates alaising. However, I rarely observe alaising in normal images, but it does show up in spades with Bart's sinusoidal star chart.

Your insights are welcome.

Regards,

Bill

[hjulenissen]

And they form the basis of JPEG 2000.

Definitely not a fringe technology.

Jim

JPEG 2000 is hardly a successful standard. Does that make it "fringe"? (not sure that the lacking success of Jpeg2k is dues to wavelets, though).

-h

[hjulenissen]

Wavelets are used extensively in the processing of astro-photos, in fact a lot of the software for that niche has wavelets built in. You might want to discuss the pros/cons with an astronomer. Mike Unsold (a mathematician) who wrote Images Plus uses wavelets in his sharpening demos that come with the program. He is helpful if you contact him.

I am not saying that wavelets are without use. I am saying that wavelets seems to be a re-invention of filterbanks. New terminology, fascinating mathematical derivations, but as far as I can see, few new results.

It is like the physicists decided that they needed signal processing, and instead of purchasing a book on signal processing, they used their (no doubt) brilliant minds to reinvent signal processing on their own. No doubt they did _something_ novel along the way, but the energy could perhaps have been put to better use by improving the existing stuff.

Yes, my professor was a wavelet sceptic. And perhaps if wavelets was presented as "a subgroup of filterbanks with some interesting properties" instead of the cure for all things bad and the Gabor limit , I would be more positive. Ah well:
"When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong."
-Arthur C. Clarke

-h

[Bart_van_der_Wolf]

In my original test I also had a 6 micron pixel camera with OLP filter. What I saw was that the OLP filter reduced colour moiré, but there was still some moiré and very much fake detail. So the conclusion I arrived at is that small pixels are more important to keep aliasing down than the OLP filter. It seems that the OLP filter is dimensioned to reduce colour aliasing, but doesn't affect grayscale moiré.

Hi Erik,

That's correct. If the goal were to eliminate aliasing on the undersampled R/B filtered channels the OLPF would throw away a lot of useful (less undersampled) G filtered data. So it aims at reducing the False Color artifacting, and some Luminance aliasing that's inherent to discrete sampling.

Cheers,
Bart

[bjanes]

As a side note, thanks for posting the MTF data. It may give me some ideas on the possible gain by a high quality lens. Could you post the raw, I would be most thankful.

Erik,

Here is a link to the raw file which I uploaded to Adobe Creative Cloud. If you have not used this facility, simply load the link into your browser and click on the download button and then on the dropdown menu with the name of the file. It downloaded on my machine as _DSC3084.X-NIKON-NEF and had to be renamed to _DSC3084.NEF in order to be opened in ACR.

Regards,

Bill

http://adobe.ly/1rCmPEO

[ErikKaffehr]

Bill,

I have downloaded the file. Thanks a lot!

Best regards
Erik

Erik,

Here is a link to the raw file which I uploaded to Adobe Creative Cloud. If you have not used this facility, simply load the link into your browser and click on the download button and then on the dropdown menu with the name of the file. It downloaded on my machine as _DSC3084.X-NIKON-NEF and had to be renamed to _DSC3084.NEF in order to be opened in ACR.

Regards,

Bill

http://adobe.ly/1rCmPEO