The problem with demosaicing

[Samotano]

The more I think about the demosaicing process,  the more I convince myself that it is a good process addressing a weak technology (I am by no means and expert).  Estimating pixel values based on neighbouring pixels brings on the issue of errors.  Some models are surely better then other, but they are still estimates.  There is no substitute for observed data points.
What I don't understand is why sensor or camera maker don't invest more into alternative technologies that would eliminate the need of estimating pixels.
Foveon's X3 (as well as scanning backs like the BetterLight) is an example of this. There is a natural elegance in its theory: no need to estimate pixels.  Granted there seems to be other challenges, but estimating what hasn't been observed is not one of them.
I do not have a Sigma SD, and only read a couple of reviews on this camera, therefore I am not aware of their shortcomings.  

What is stopping camera makers into concentrating their efforts on alternatives to the Bayer sensor? Why aren't more camera makers adopting Foveon's X3? Why don't camera makers create a consortium to research methods in designing new sensors?
I would be interested to know what some users of this forum think about this topic...

[Graeme Nattress]

It's a very political question.

Although the "best" way to reconstruct a Bayer pattern to RGB is an unsolved problem, it is, for the most part fairly transparent and not as detrimental to the image as is often thought.

And the other problem is the answer to this question also brings in another political topic, that of optical low pass filters.

The shortcoming of the Foveon approach is that it  uses silicon depth as a colour filter, and silicon is a poor colour filter. That means it's more tricky to extract good colour from a Foveon sensor, and the colour tends to be noisier as the image data appears less saturated and therefore needs amplifying more to get reasonable colour.

Scanning backs avoid the Foveon issues above, but cannot cope with any significant shot  movement without artifacting (strange colour fringes).

And in the video world, we have a 3 chip + prism approach, which gives the co-sited RGB colours that people like in the Foveon, but in many cases  the chips are actually offset and if  the RGB is decoded (many video cameras record sub-sampled chroma and with this you don't notice the offset) you can see this offset, which somewhat defeats  the point of the approach. The prism introduces optical artifacts, and limits lens choice.

Basically, there's no perfect approach to sensors, and Bayer issues are reasonably easy to  understand, but in practice are no worse (if handled correctly) than the issues with other approaches.

My bias is obvious with my work on the RED Digital Cinema camera, which uses a 12mp custom design Bayer pattern sensor, and I have written the demosaicing and raw image processing software for the applications that go with the camera.

What it comes down to is that there is not an optimum solution, but there are optimum solutions for specific shooting requirements, and it's only through knowledge of the pros and cons of each approach can you choose the correct camera for your requirement.

Graeme

[marcmccalmont]

It is interesting to note that your eye has discrete R,G,B color receptors (cones) and a lot more B&W receptors (rods) . So, good enough for my eye, good enough for my camera. I’d like to see a R,G,B,L (luminance) sensor in a pseudo random pattern (Latin squares vs. Bayer) w/o an AA filter.
Marc

[Graeme Nattress]

Rods aren't really used for normal daylight vision though, and the cones are not really RGB either.

There are lots of ideas and mosaic patterns you can use instead of the Bayer one, although Bayer is popular because it works, produces  good results and is easy to decode and a lot of the decode research is done with that pattern in mind.

Graeme

[marcmccalmont]

Rods aren't really used for normal daylight vision though, and the cones are not really RGB either.

There are lots of ideas and mosaic patterns you can use instead of the Bayer one, although Bayer is popular because it works, produces  good results and is easy to decode and a lot of the decode research is done with that pattern in mind.

Graeme
[a href=\"index.php?act=findpost&pid=170041\"][{POST_SNAPBACK}][/a]

Are there any books/white papers/references that you would recommend too build ones fundamental knowledge of digital capture?
Marc

[Graeme Nattress]

The problem with the colour science books I've read, is they're dry and full of formula, and don't address the nuances of how sensors work in practice, and the "art" of making a usable image. I don't think there's a book specifically geared towards digital capture.

I did find Billmeyer and Saltzman's "Principles of Color Technology" quite readable and informative though. It has a section on image capture. If you want the formula and equations though, Wyszecki and Stiles "Color Science: Concepts and Methods, Quantitative Data and Formulae" is my desk reference.

Graeme

[Samotano]

It's a very political question.

Although the "best" way to reconstruct a Bayer pattern to RGB is an unsolved problem, it is, for the most part fairly transparent and not as detrimental to the image as is often thought.

And the other problem is the answer to this question also brings in another political topic, that of optical low pass filters.

The shortcoming of the Foveon approach is that it  uses silicon depth as a colour filter, and silicon is a poor colour filter. That means it's more tricky to extract good colour from a Foveon sensor, and the colour tends to be noisier as the image data appears less saturated and therefore needs amplifying more to get reasonable colour.

Scanning backs avoid the Foveon issues above, but cannot cope with any significant shot  movement without artifacting (strange colour fringes).

And in the video world, we have a 3 chip + prism approach, which gives the co-sited RGB colours that people like in the Foveon, but in many cases  the chips are actually offset and if  the RGB is decoded (many video cameras record sub-sampled chroma and with this you don't notice the offset) you can see this offset, which somewhat defeats  the point of the approach. The prism introduces optical artifacts, and limits lens choice.

Basically, there's no perfect approach to sensors, and Bayer issues are reasonably easy to  understand, but in practice are no worse (if handled correctly) than the issues with other approaches.
...
Graeme
[a href=\"index.php?act=findpost&pid=169998\"][{POST_SNAPBACK}][/a]

Interesting.  I still wonder though if Foveon's technological advances are dwarfed by the Bayer pattern adopters (i.e. 95% of camera makers), thus the issues you mention may just be the results os a disparity in advancement between the two.  I am curious as to your initial statement "... best way to reconstruct a bayer pattern to rgb is an unsolved problem".  How much better can we get at reconstructing missing data?

[Graeme Nattress]

The issue of silicon being a poor colour filter is inherent in the Foveon design as the mosaic is to the Bayer pattern.

As for how much better bayer pattern decodes can get? It's the sort of problem that can be helped by throwing lots of computing power at it, as the more things you can think of to do to help the decode, the longer they generally take and people don't usually want to wait a long time to get an image out. Often with digital photograph, people want resolution above all else, and to a point that's reasonable that you want good resolution, but in the end, other factors are more important. I'd say that ensuring lack of decode artifacts is more important than resolution per-se, and that's how I code my demosaic, to ensure minimum artifacting and good resolution, rather than trying to push for higher resolution and consequently more artifacts.

There's a big reason why many use the Bayer pattern - it works, it's economic, and you get a 2/3 saving in uncompressed file sizes. That's not to be sniffed at. On the other side, there's complex processing to be done to make it back to a viewable image.

Graeme

[Samotano]

The issue of silicon being a poor colour filter is inherent in the Foveon design as the mosaic is to the Bayer pattern.

As for how much better bayer pattern decodes can get? It's the sort of problem that can be helped by throwing lots of computing power at it, as the more things you can think of to do to help the decode, the longer they generally take and people don't usually want to wait a long time to get an image out. Often with digital photograph, people want resolution above all else, and to a point that's reasonable that you want good resolution, but in the end, other factors are more important. I'd say that ensuring lack of decode artifacts is more important than resolution per-se, and that's how I code my demosaic, to ensure minimum artifacting and good resolution, rather than trying to push for higher resolution and consequently more artifacts.

There's a big reason why many use the Bayer pattern - it works, it's economic, and you get a 2/3 saving in uncompressed file sizes. That's not to be sniffed at. On the other side, there's complex processing to be done to make it back to a viewable image.

Graeme
[a href=\"index.php?act=findpost&pid=170109\"][{POST_SNAPBACK}][/a]

I agree, I would take less resolution for better accuracy at the price of complexity and computing time.  You mention you code your own demosaicing program.  Are you aware of any raw converter that address those issues?

[Graeme Nattress]

There are lots of raw converters out there and you've just got to try them and see if they fit in with your idea of what an image should look like. To a great extent it's a personal choice.

I work on the demosaic for the RED camera, which is a digital movie camera rather than a stills camera. You can get the RedAlert (mac) or RedCine (mac and pc) software at www.red.com and www.redrelay.net has some raw R3D files that you could try with the software if you're interested in seeing what I do. http://reduser.net/Demosaic%20Comparison.pdf shows some demosaic examples where I've traded absolute resolution for lack of artifacting. I also have to deal with the fact that the  RAW files are actually compressed RAW at about 12:1 so that we can record 4096x2304 movies onto a compact flash card, rather than still images.

Graeme

[Samotano]

There are lots of raw converters out there and you've just got to try them and see if they fit in with your idea of what an image should look like. To a great extent it's a personal choice.

I work on the demosaic for the RED camera, which is a digital movie camera rather than a stills camera. You can get the RedAlert (mac) or RedCine (mac and pc) software at www.red.com and www.redrelay.net has some raw R3D files that you could try with the software if you're interested in seeing what I do. http://reduser.net/Demosaic%20Comparison.pdf shows some demosaic examples where I've traded absolute resolution for lack of artifacting. I also have to deal with the fact that the  RAW files are actually compressed RAW at about 12:1 so that we can record 4096x2304 movies onto a compact flash card, rather than still images.

Graeme
[a href=\"index.php?act=findpost&pid=170130\"][{POST_SNAPBACK}][/a]

Thanks for sharing your experience.

[papa v2.0]

might be of interest

sorry cant up load file wiil try again

[tjhanlon]

I own an SD14, and had previously owned an SD9. What the Sigmas do well they do very, very well indeed. And what they don't do well - well, they don't.

There is a clarity and sharpness that X3F (the RAW format) images have. There is no AA filter, so there is no blurring. Details can be sharp and crisp without mush or smoothing. Only in rare and exceptional circumstances is moire to be found, and then there's no multi-colored rainbow effect. Colors are accurate and vibrant, and not exaggerated or over saturated. Because of the stacked pixels in the sensor, B/W conversions can be stunning. The SD10 has a removable IR filter, and it's even easier with the SD14 to get IR imagery. Layout of controls is simple, without a lot of different "modes" such as landscape or portrait - It is very much like a film SLR.

On the other hand, they produce huge files, and the Sigma's don't have a huge buffer. Additionally, images can get to be pretty noisy when underexposed and pushed, or at high ISO - especially in the blue channel. Autofocus can hunt, especially in low light.

When combined with good glass, you can get eyepoppers. Probably not the easiest for low-light action or weddings, but there are a number of really talented photographers who use them in weddings and get great results.

If you search pbase for sample SDxx images, you will see that most are of the landscape/macro variety.

Depending on the RAW converter, yellows and blues can get electric, but using Sigma's PhotoPro 2.3 or Iridient's Raw Developer does a better job than ACR; Raw Developer is fast, too. Yvind Strom did a tutorial on learning about the camera. It can be accessed here:

http://www.foto.nordjylland.biz/SD14/SD-usertips.htm

As far as why are the Foveon sensors not universally used, a lot also has to do with the uniqueness of the fabrication process, as well as the processing power in camera needed.

Hope this helps.

tjh

[papa v2.0]

hi

[Samotano]

I own an SD14, and had previously owned an SD9. What the Sigmas do well they do very, very well indeed. And what they don't do well - well, they don't.

[...]
Depending on the RAW converter, yellows and blues can get electric, but using Sigma's PhotoPro 2.3 or Iridient's Raw Developer does a better job than ACR; Raw Developer is fast, too. Yvind Strom did a tutorial on learning about the camera. It can be accessed here:

[
tjh
[a href=\"index.php?act=findpost&pid=170337\"][{POST_SNAPBACK}][/a]

So there is no demosaizing at all in the conversion of Sigma SD right? For curiosity, how big are the files?

[tjhanlon]

So there is no demosaizing at all in the conversion of Sigma SD right? For curiosity, how big are the files?
[{POST_SNAPBACK}][/a]

At highest resolution, and depending on the subject matter, the RAW files are between 10 and 13 MB. I really don't know the ins and outs of how the raw converters work with the SDxx files - it's a little too much like making sausage for my taste. I think the reason why the file size if variable in the SD14 is because of the embedded jpeg, something the SD9 and SD10 don't have - they are RAW only.

If you are interested, Carl Rytterfalk has some raw files to play with at [a href=\"http://www.rytterfalk.com]http://www.rytterfalk.com[/url] they can be accessed off his home page. If you have a Mac, Raw Developer is free, but it watermarks images in the center. But at least you can see how it works, and the latest version of ACR can process the X3F files as well. You can also have Sigma USA send you a cd with Sigma PhotoPro on it and some RAW test files. If you really want to play with SPP, drop me an email: tjhanlon at mac dot com. There are some others, such as ArcSoft's Darkroom, but I would avoid that, especially if you are on a Mac.

Regards,

tjh