Pages: [1]   Go Down

Author Topic: De-bayer: Single pixel shift  (Read 3699 times)

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
De-bayer: Single pixel shift
« on: November 29, 2013, 05:50:08 pm »

Thinking about the Sony Hasselblad camera anouncement yesterday made me think of a new method of debayering. Sony (as well as other companies) already have sensor shift technology. If you can make very fast exposures with pixel shutter, why not also shift the chip one sideways then one vertical outputting a new raw format with 3 images? This gives you full color sampling at every point eliminating the whole debayer mess. This would be the lowest cost way to get to full color sampling at every pixel.

Any photographer could try to approximate this by minor random shifts of the tripod, then aligning raws, then stacking. Success would be a crapshoot of how well the camera movement works. It would be better to have precise stepper motors moving the sensor known distances.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: De-bayer: Single pixel shift
« Reply #1 on: November 30, 2013, 02:54:18 am »

Hi,

Pentax does something similar on one of it's new DSLRs, they have variable aliasing reduction by shifting the sensor under exposure.

But I would say the Bayer bashing is a bit overblown. If pixel size is reduced the both color and monochrome aliasing goes away.

The enclosed images are shot with a P45+ and a Sony Alpha 77 at 3.5m distance with a 150 mm lens on both (Zeiss resp Sony lens). P45+ on the left Alpha 77 on the right.

One of the images is at actual pixels, in the other one the P45+ image is enlarged using bicubic.

Don't you think small pixels go a long way of solving all aliasing problems?

Another observation, shown in the third enclosed picture: Even the 3.9 micron image shows broken strains in the marked area, so 3.9 microns still produce fake details.

Please note: this is not about MFD, FF 135 or APS-C, it is simply about demonstrating large pixels versus small pixel with stuff I happen to have in my wardrobe.

Best regards
Erik



Thinking about the Sony Hasselblad camera anouncement yesterday made me think of a new method of debayering. Sony (as well as other companies) already have sensor shift technology. If you can make very fast exposures with pixel shutter, why not also shift the chip one sideways then one vertical outputting a new raw format with 3 images? This gives you full color sampling at every point eliminating the whole debayer mess. This would be the lowest cost way to get to full color sampling at every pixel.

Any photographer could try to approximate this by minor random shifts of the tripod, then aligning raws, then stacking. Success would be a crapshoot of how well the camera movement works. It would be better to have precise stepper motors moving the sensor known distances.
« Last Edit: November 30, 2013, 04:11:37 am by ErikKaffehr »
Logged
Erik Kaffehr
 

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: De-bayer: Single pixel shift
« Reply #2 on: November 30, 2013, 11:56:25 am »

I do not see the biggest problem with bayer as aliasing. I see it as sampling what is really transmitted by the lens at every point, which shows up most in color accuracy.

Here is an interesting article comparing a few different programs. The most important are on the right. The RT Amaze method vs the Lightroom method.

http://renderingpipeline.com/2013/04/a-look-at-the-bayer-pattern/

The first thing that stands out is the color differences from the same raw file. The second thing that stands out is the choice of a smoothed soft look by lightroom vs the sharp blocky look of RT. What is the outcome of that choice? My guess is the Lightroom version is more accurate in color by doing some blend of the color sampling from each pixel. RT definitely has the advantage on image detail at the expense (my guess) of having to adjust colors for accuracy. I could be wrong, the fake flower could be properly captured by RT.

Here is another article showing that all debayer methods have problems with what you are interested in, moire.

http://www.libraw.org/articles/bayer-moire.html

Smaller pixels just have the same issue on a smaller scale until you hit diffraction issues. I agree with you that all else being equal, smaller pixels will give a better image. At the point of diffraction all else is no longer equal. The sizes of the red, green and blue diffraction spots are very different. So the perfect camera would be 3 sensors behind a dichroic prism with 6.5 microns for red, 5.5 for green, 4.5 for blue. The software engineering problem would be making the best image of those 3 layers. The camera companies could legitimately charge $4-5K for that.

As Bart mentioned a couple weeks ago, with the current design you can get away with pixels 1/2 the diffraction spot size due to the gap in between red to red, blue to blue, etc. As you go smaller you start throwing away light relying on just chopping the top off the center diffraction peak, then boosting contrast in software. By the time the first ring hits the adjacent same color you will have color shifts. Tiny P&S pixels should have poor color accuracy.

I would rather actually sample each color at every point with the small extra cost of the sensor motors and the 3 subshot data buffers. I expect there would be a gain in overall color accuracy first, then elimination of false color artifacts second. There should also be an improvement in noise.

Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: De-bayer: Single pixel shift
« Reply #3 on: November 30, 2013, 03:45:32 pm »

Hi,

I am actually not interested at all in Moiré. I am interested in aliasing as a general problem and Moiré is just a subclass of it. Bayer demosaic just adds colour to it, but I suggest that small pixels are the solution to all aliasing problems. With adequate pixel size the problems go away. What is adequate pixel size? I would say about two microns because that is about natural diffusion length of light in silicon, but need for decent DR taken into account 3-4 microns with todays technology.

Best regards
Erik


I do not see the biggest problem with bayer as aliasing. I see it as sampling what is really transmitted by the lens at every point, which shows up most in color accuracy.

Here is an interesting article comparing a few different programs. The most important are on the right. The RT Amaze method vs the Lightroom method.

http://renderingpipeline.com/2013/04/a-look-at-the-bayer-pattern/

The first thing that stands out is the color differences from the same raw file. The second thing that stands out is the choice of a smoothed soft look by lightroom vs the sharp blocky look of RT. What is the outcome of that choice? My guess is the Lightroom version is more accurate in color by doing some blend of the color sampling from each pixel. RT definitely has the advantage on image detail at the expense (my guess) of having to adjust colors for accuracy. I could be wrong, the fake flower could be properly captured by RT.

Here is another article showing that all debayer methods have problems with what you are interested in, moire.

http://www.libraw.org/articles/bayer-moire.html

Smaller pixels just have the same issue on a smaller scale until you hit diffraction issues. I agree with you that all else being equal, smaller pixels will give a better image. At the point of diffraction all else is no longer equal. The sizes of the red, green and blue diffraction spots are very different. So the perfect camera would be 3 sensors behind a dichroic prism with 6.5 microns for red, 5.5 for green, 4.5 for blue. The software engineering problem would be making the best image of those 3 layers. The camera companies could legitimately charge $4-5K for that.

As Bart mentioned a couple weeks ago, with the current design you can get away with pixels 1/2 the diffraction spot size due to the gap in between red to red, blue to blue, etc. As you go smaller you start throwing away light relying on just chopping the top off the center diffraction peak, then boosting contrast in software. By the time the first ring hits the adjacent same color you will have color shifts. Tiny P&S pixels should have poor color accuracy.

I would rather actually sample each color at every point with the small extra cost of the sensor motors and the 3 subshot data buffers. I expect there would be a gain in overall color accuracy first, then elimination of false color artifacts second. There should also be an improvement in noise.


« Last Edit: November 30, 2013, 03:50:01 pm by ErikKaffehr »
Logged
Erik Kaffehr
 

NikoJorj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1082
    • http://nikojorj.free.fr/
Re: De-bayer: Single pixel shift
« Reply #4 on: November 30, 2013, 03:56:16 pm »

Don't you think small pixels go a long way of solving all aliasing problems?
A nice reality check, thanks!
Logged
Nicolas from Grenoble
A small gallery

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: De-bayer: Single pixel shift
« Reply #5 on: November 30, 2013, 05:01:20 pm »

I just noticed today on OPF a post on multishot backs

http://www.openphotographyforums.com/forums/showthread.php?p=148678#post148678

which refers back to an article here on LL by Mark in  March 2010.

http://www.luminous-landscape.com/reviews/cameras/h3d50ii.shtml




So this has already been done for years. Why don't we have this option on our DSLRs?
Logged

BJL

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6600
Sony Hasselblad camera announcement?
« Reply #6 on: December 01, 2013, 04:04:56 pm »

Thinking about the Sony Hasselblad camera anouncement yesterday ...
What announcement? All I have seen recently is this rumor: http://www.sonyalpharumors.com/sr4-hasselblad-and-sony-to-make-a-joint-sensor-announcement/

Back to the main topic: yes, multi-shot backs do something like this, so I also wonder whether any of the sensor stabilization technologies could be adapted to such a four-shot, high resolution mode. But it seems to me that the dominant industry trend is towards achieving all the resolution that a lens can deliver (including all the color resolution) by simply "over-sampling" -- reducing the pixel size and spacing until sensor resolution is beyond the lenses' limits due to diffraction plus aberrations.
« Last Edit: December 01, 2013, 04:06:52 pm by BJL »
Logged

NancyP

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2513
Re: De-bayer: Single pixel shift
« Reply #7 on: December 02, 2013, 11:32:57 am »

Don't the CFAs need to be bonded to the microlens and diode? If there is room to shift the sensor vis-a-vis the filter array, there is room for light scatter to occur. How much of an issue this would be is not clear to me - I am not an engineer or fabricator.
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: De-bayer: Single pixel shift
« Reply #8 on: December 02, 2013, 12:52:59 pm »

Don't the CFAs need to be bonded to the microlens and diode? If there is room to shift the sensor vis-a-vis the filter array, there is room for light scatter to occur. How much of an issue this would be is not clear to me - I am not an engineer or fabricator.

We are thinking of different implementations.

If you try to move the sensor under the CFA you would probably get wearing of the surfaces.

I am thinking of moving the whole sensor+CFA+microlenses relative to the image being formed by the camera lens. Yes, you would lose a pixel off each side. I think the sensors already have spare pixels on the sides that are used for other things like whibal.
Logged

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: De-bayer: Single pixel shift
« Reply #9 on: December 02, 2013, 01:35:44 pm »

I think that the idea of moving the sensor assembly 1 or 1/2 sensel in either dimension is not novel. Hasselblad have used this for their multi-shot backs. Super-resolution techniques rely on (random) movement for its magic.

-h
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: De-bayer: Single pixel shift
« Reply #10 on: December 02, 2013, 02:39:30 pm »

I think that the idea of moving the sensor assembly 1 or 1/2 sensel in either dimension is not novel. Hasselblad have used this for their multi-shot backs. Super-resolution techniques rely on (random) movement for its magic.

-h

Neither is your post, we have already posted that this was being done in early 2010. Can you add something to take it to the next stage?
Logged
Pages: [1]   Go Up