Demosaicing IR Images w/o Interpolation

[Jim Kasson]

For the last week or so, I have been spending way too much time trying to develop a way to demosaic infrared raw files without interpolation. My reasoning is that, with a deep IR filter in front of the sensor, that the spectral response of each of the CFA planes is pretty close to that of the others, so if I can equalize the response of the planes over some largeish area, I won't need to interpolate. The result should be a sharper image.

The first area I chose for equalization was the whole image. I calculated the average of each color plane, and then I equalized each pixel in each plane so that all four planes had the same average value. In another, similar, experiment, I left the two green planes alone and just equalized the red and the blue planes so that they had the same average value as one of the green planes. Both techniques produced substantially the same results.

I compared the results to ACR demosaicing with default settings and the Matlab Image Pack implementation of bi-linear demosaicing. My first conclusion is that there is a lot of secret sauce in ACR, which makes for quite a high bar in comparison tests. I had to add some home-brew deconvolution to my images to make for a fair comparison. My second conclusion is that my algorithm is only partially successful; although the unsharpened images look pretty good, deconvolution causes some checkerboard artifacts in areas of low spatial frequency. I have some ideas about fixing this, but I'm unwilling to invest much more time until I get some different test files -- see below.

My third conclusion is that the anti-aliasing filter in my test camera, an IR-converted D3x, blurs the image hitting the sensor enough that there's not much sharpness to be gained by interpolation-free demosaicing.

I would like to do some testing with a raw file from a deep-IR camera that doesn't have an anti-aliasing filter. Can anyone here provide me with such a file?

Thanks,

Jim

[Jim Kasson]

No much activity on my request, but I did get some D800E deep-IR images from another source, and I'd like to report on what I've found.

But first, there is a new IR-processing capability in version 2.1 of Iridient Developer (ID, not to be confused with InDesign, which is Id) that you ought to know about if you make B&W images from raw files produced by IR-modified cameras. I saw Brian Griffith at the April 17th CPA Raw Processing Workshop in Menlo Park, and he and I talked about the special problems of making B&W images from IR files. I proposed the no-interpolation algorithm, but Brian had another idea. In the B&W conversion tab (I'll call it a tab, but it doesn't really look like a tab) of ID 2.1, there's a new option: the raw channel mixer. This option bypasses the transformation of the camera primaries to a standard color space, and does the mixing of the camera primary values directly. The upshot is, if you choose green-only, that your converted image has only half the pixels interpolated, and therefore should be sharper. It's noisier, too, by a factor of 1.414, but that shouldn't be a problem in the mid to high values, and you could do a conversion weighting all three channels more-or-less-equally, and use that with a luminance mask to get better sign'-to-noise ratio in the low values. Perhaps Brian will automate this in the future.

I'd determined before that my technique offered no advantages on images made with cameras that employed anti-aliasing (AA) filters. I figured that, if that were true, Brian's new method wouldn't either. I'm since done limited testing that pretty much verifies that.

Lloyd Chambers graciously sent me some raw files made with an IR-modified D800E. The images were exposed with lens that is very sharp in IR, the Coastal Optics 60mm f/4 Macro. Lloyd used a B+W 093 filter on the lens to make sure that near-IR wavelengths were excluded. By the way, I recommend Lloyd's website to anyone intereted in IR, or in photography for that matter. It's mostly a pay site, but I think it's well worth it.

I subjected Lloyd's raw IR images to processing in ID 2.1 using the raw channel mixer with all channels represented in 40/30/30 ratio, and using the green-only option. I also processed them using my no-interpolation algorithm, which balances the red and green channels so that their average values across the entire image is the same as the upper right green channel in the mosaic. (The D800 has an RGGB pattern). The green channels were left alone, even though there were slight differences in the average values.

Comparison of the three sets of images has led me to the following conclusions:

1)      The interpolationless demosaicing works fine on the deep-IR images Lloyd sent me. The variation in spectral response of the CFA channels is low enough in the deep IR that there are no visible (to my eyes) artifacts even with considerable sharpening.

2)      There is some minor improvement in detail over using the raw green channels only in Iridient Developer 2.1. The improvement is not visible until both images are sharpened.

3)      The green-channel only image is sharper than one produced in Iridient Developer with the raw channel mixer set to the “neutral” preset, which mixes in the red and blue channels, which are presumably less sharp because more interpolation is necessary to produce them because of the relative population of the red-sensing pixels and blue-sensing pixels to the green-sensing ones. As with the previous comparison, it is necessary to sharpen the images to see much in the way of difference.

4)      All these differences are minor, and are not visible at all on cameras with a medium or strong AA filter.

If you are doing deep-IR work with the D800E or any other camera with no AA filter, I recommend that you take a good look at Iridient Developer. Heck, take a good look at it anyway; it has much to recommend it, including its (to me) delightful transparency.

I expected the differences to be greater. Lloyd suspects that having half the AA filter in place in the D800E may be having some effect.

I would show you some images, but they are not mine, they're Lloyd's.

Thanks to both Brian and Lloyd.

Jim

[t6b9p]

Hi Jim

this is something I have been interested in for some time.
http://www.luminous-landscape.com/forum/index.php?topic=29119.0;topicseen

Inspired by my own experiments and that of iRAW Mono by khromagery.com and Guillermo Luijk (link above).

I am travelling for the next 5 weeks but would like to discuss this further when I get back. I can provide RAW 830nmIR files from a D200 (soon to be replaced) if you require them.

[Jim Kasson]

this is something I have been interested in for some time.
http://www.luminous-landscape.com/forum/index.php?topic=29119.0;topicseen

Inspired by my own experiments and that of iRAW Mono by khromagery.com and Guillermo Luijk (link above).

That's a great link. I had no idea. Guillermo Luijk's thinking mirrors Brian's, and anybody with 75 bucks and a Mac can try it out.  Here's a line to Brian's web site.

My approach is more extreme, and can run into problems if there's significant spectral response variation among the three (or four, depending on how you count) channels.  That's why I think my technique is only good for deep-IR images.

Brian's green-only method ought to work just fine for near-IR images as well, with the understanding that in presence of channel-to-channel variation, picking one channel will affect the tone curve in much the same way as putting a filter over a lens does. Brian's raw channel mixer used with some mixing of the red and/or blue channels, can yield pretty much all the benefits of traditional IR to B&W conversion without the problems that Guillermo Luijk identified in the thread you posted.

Jim

[Jim Kasson]

I can provide RAW 830nmIR files from a D200 (soon to be replaced) if you require them.

I'd like that, although I believe the D200 has an AA filter and therefore I wouldn't expect to see much sharpness improvement. I could be wrong, and I'd be interested in broadening my sample space. Shoot me a message when you return from your trip.

If you want to play around yourself, Iridient Developer is a good place to start.  If you have the inclination and access to a computer running Matlab (including the Image Pack), I can send you the source code.

Have a good trip!

Jim

[t6b9p]

My D200 is internally modified by removing the ICF/AA and replacing it with an 830IR filter - so no AA filter anymore.

I did notice "maze" (checkerboard) problems during "heavy" processing with the D200IR830 and found it was primarily due to a G channel imbalance see my posting http://nikongear.com/live/index.php?/topic/39891-maze-artifacts-does-the-d200-have-a-subtle-g-channel-imbalance/.

I have seen similar issues reported for visible images as well from some other Nikon models. I suspect it appears more of a problem in the 830nm IR due to the need to stretch the contrast. Not sure if this is an issue with the newer CMOS Nikons vs the earlier CCD Nikons.

Unfortunately I haven't had access to Matlab since 1992 but my son-in-law may have it , I will check with him and will contact you when I return.

Shane

[Jim Kasson]

My D200 is internally modified by removing the ICF/AA and replacing it with an 830IR filter - so no AA filter anymore.

I did notice "maze" (checkerboard) problems during "heavy" processing with the D200IR830 and found it was primarily due to a G channel imbalance...

Shane,

It's easy for me to modify my program to compensate for a certain kind of G channel imbalance: one in which the average values across the entire image of the G and G1 channels are different. I did notice some of this with the D800E and my D3x images, but it was very slight. I am trying without success to imagine a systematic mechanism for having it happen, unless it's that the dyes for the G and G1 filters come through different "pipes" from different sources.

Jim

[bwana]

After reading this thread (and the links as well) I come away with the feeling that Bayer interpolation is not affecting the image sharpness all that much. I too (like Jim and others here) thought that the Bayer process effectively reduces your pixel count by using information of neighboring pixels to contaminate each other - all in the name of reconstructing color information. And isnt this the whole point of the leica monochrome? Get rid of the bayer filter and only measure pure luminance information-to get sharper pix?

I know I should do my own work, but do not have an IR modified camera or even the necessary software ( I dont have a mac so i cant run ID, I cannot find IR MONO, anyone know where it lives - it's not 0bvious at irphoto.com or khromagery.com). I have discovered tho that the Fuji X100s makes gorgeous pics using an R72 filter. I can hand hold this setup and get pretty good pix (after conversion to b&w). http://goo.gl/B4ZCR

I would expect removing the bayer process from raw processing would at least double resolution, but my impression is that only a subtle improvement can be discerned. I imagine I could answer my own question if I  had the necessary math skills but am wondering if Jim or anyone else can say why in words. If someone can explain special relativity in four letter words or less (http://www.muppetlabs.com/~breadbox/txt/al.html) surely this conundrum can be unwound.

[Iluvmycam]

Do you have any before and after photos of your work?

[bwana]

All i did was switch to B&W in lightroom and start with the false color jpg out of the camera.

[Jim Kasson]

After reading this thread (and the links as well) I come away with the feeling that Bayer interpolation is not affecting the image sharpness all that much. I too (like Jim and others here) thought that the Bayer process effectively reduces your pixel count by using information of neighboring pixels to contaminate each other...
I would expect removing the Bayer process from raw processing would at least double resolution, but my impression is that only a subtle improvement can be discerned. I imagine I could answer my own question if I  had the necessary math skills but am wondering if Jim or anyone else can say why in words.

The short answer is in two parts: 1) the antialiasing filter that is probably, but not certainly, in front of the sensor in your camera blurs things to keep fine detail from being sharply imaged, and 2) demosaicing software has become very sophisticated and effective in the last ten years or so.

Jim

[Guillermo Luijk]

ADAPTIVE LOW NOISE B&W CONVERSION

Curiously I was recently thinking of how a B&W conversion algorithm could work focusing on improving output SNR. It is well known that:
1. SNR gets higher the higher RAW values are encoded, no matter the RAW channel.
2. Linear combination of 2 or more (in this case 3) signals can be optimally weighted: Y=k1*R+k2*G+k3*B (k1+k2+k3=1) in order to maximize SNR on Y. I once did the calculations once for the bi-dimensional case assuming noise adding in quadrature: N^2 = P1^2 + P2^2

each colour plot represents the extra exposure of the higher RAW value
p is the mixing factor for the higher RAW value, 1-p for the lower RAW value
The red line represents p values for optimal mix (max. SNR) of the two differently exposed RAW values
Blue is the simple average of the two RAW values
Green is taking the highest RAW value
Y-axis represents the % of noise transferred to the final pixel with respect to the noise in the noisiest input pixel

It is easy to see that for an extra exposure above +1 +1/3, the contribution of the least exposed shot becomes negligible and we're better served just by picking the pixels of the higher exposure RAW file. The consequence for this in HDR imaging, is that if we shoot 2EV apart the fusion software should not even consider any progressive blending; otherwise it would lose sharpness and gain nothing.

A simple algorithm could:
1. Create first 3 intensity maps by deblurring the R, G and B individual channels
2. On each pixel, use the 3 intensities calculated in the previous step to calculate the optimal k1, k2, k3 set to optimise SNR on Y for that pixel

Since RAW data gets usually higher values in the G channel, its contribution to the output Y would generally prevail to maximize final SNR. It would be interesting to compare B&W valid textures obtained in this way vs a conventional B&W conversion method, specially in the deep shadows (noisiest areas that could benefit more from the SNR optimisation).

Another story is how the final image would look like in terms of colour to greyscale translation, since the contribution of all 3 input channels would softly but surely vary along all the image.


ADAPTIVE HIGH LOCAL CONTRAST B&W CONVERSION

The same scheme could be used in this case to maximize local contrast (i.e. texture detail). Specially if noise is not an issue, the Y channel mix could weight more the channel with the higher local contrast for every area in the image. So if some spectral band gets more variations in some part of the scene, the corresponding channel would prevail over flatter channels.


A mixed low noise/high local contrast scheme could be implemented as well. The first approach could help to differentiate noise from texture.

Jim I miss some sample images from your tests to make this thread more visual and funny

Regards

[Jim Kasson]

Jim I miss some sample images from your tests to make this thread more visual and funny

Good post. I'll have to think about the implications of what you're saying before I can really come to terms with it.

Unfortunately, Guillermo, the IR-converted D800E images that I was using belong to Lloyd Chambers, and he doesn't want them posted.

Funny?

Jim

[hjulenissen]

I was not aware that (deep) IR spectral response tended to be the same for r/g/b. Do you have any sources on this?

Demosaicing when the scene can be assumed to be achromatic should indeed be simpler, and allow for somewhat more aggressive detail recovery. But how large is the potential gain?

-h

[Jim Kasson]

I was not aware that (deep) IR spectral response tended to be the same for r/g/b. Do you have any sources on this?

I started down this road mostly through my own experiences looking at the files I got out of my IR-modified cameras. The appeared monochromatic, albeit with a reddish cast. My testing seems to indicate that that's mostly true with the D3x and D800E at least. There is an offhand reference to that situation here.

Demosaicing when the scene can be assumed to be achromatic should indeed be simpler, and allow for somewhat more aggressive detail recovery. But how large is the potential gain?

As far as I've been able to tell, the gains are small.

Jim

[Bart_van_der_Wolf]

I was not aware that (deep) IR spectral response tended to be the same for r/g/b. Do you have any sources on this?

Hi,

The Bayer CFA dyes are transparent to IR, as are most dyes.

Cheers,
Bart

[hjulenissen]

As far as I've been able to tell, the gains are small.

It would be interesting to see how this affects the conditions for deconvolution.

AFAIK, the CFA is considered an obstacle for doing deconvolution. If deep-ir passes through all CFA channels equally (or at least can be made equal by simple global or regional "white-balancing"), then deconvolution algorithms should see no CFA either, and more aggressive deconvolution might be possible?

I guess that the difference between regular CFA cameras and achromatic cameras (the Leica models springs to mind) can be used as indicators of the limit to how much can be gained from your idea?

-h

[Jack Hogan]

The Bayer CFA dyes are transparent to IR, as are most dyes.

Interesting, I never realized that:



Removing the IR filter from such a sensor in the presence of uniform light from 400-1000nm would 'artificially' boost the counts of each of the three channels.  Eyeballing it, if we take 800nm as the old cutoff, it looks like it would add about 33% to the red channel count,  and 50% to the green and blue.

Jack

[32BT]

My reasoning is that, with a deep IR filter in front of the sensor, that the spectral response of each of the CFA planes is pretty close to that of the others, so if I can equalize the response of the planes over some largeish area, I won't need to interpolate. The result should be a sharper image.

Unfortunately, this doesn't work in but the most ideal case. The reason it doesn't work is that even the slightest imbalance between channels will show up as a checkerboard pattern which will be significantly more obtrusive than any interpolation "errors" of modern RAW converters. The imbalance between channels will virtually always happen, because the sensitivity of the individual channel filters is not equal, so any "color" changes in the desired spectral range will result in differences, even if the response per channel is equalized. The sensor will still detect hue-differences, and the data will still depict those hue-differences, which shows up as checkerboard patterns.

So, this will work only if the spectral response of the sensels is equal within the desired range. (Not if the tristimulus response is equal)

[Jim Kasson]

Unfortunately, this doesn't work in but the most ideal case. The reason it doesn't work is that even the slightest imbalance between channels will show up as a checkerboard pattern which will be significantly more obtrusive than any interpolation "errors" of modern RAW converters. The imbalance between channels will virtually always happen, because the sensitivity of the individual channel filters is not equal, so any "color" changes in the desired spectral range will result in differences, even if the response per channel is equalized.

That is not my experience. Once the channels are equalized, there is no checkerboard that my eyes can see even after the image is sharpened with both the D3x and D800e, in the deep IR.  

Jim