The endless dynamic range sensor of the future

[Der Oak]

Hello Forum,

I would like to share with you an idea that came to my mind about a future digital sensor with basically unlimited dynamic range (slight exaggeration here). I have no technical background whatsoever, so I don't know if what I am proposing is at all technically feasible. But it just feels as if it should be possible to develop such a system.

So, what am I talking about?! Basically I was thinking that bracketing exposures in order to expose different parts of a scene correctly in individual files does one thing: Making a composite where (just making up numbers here) the photosites that captured the sky were turned on for 1/125, whereas the rest of the photosites were switched on for 1/30 of a second. By this differential exposure, the sky is eposed correctly in one image, and the foreground is exposed correctly in the other one. Both are combined to have proper exposure across the whole scene.

I know, I know, this is trivial and everybody knows about it. But here comes the idea. Why can't we just tell the sensor to do exactly this differential exposure at the time of capture?

The setup goes like this: In a future camera we have iPhone-like touchscreens that basically cover most of the back of the camera, so that we have a nice and detailed preview of the scene (live-view enabled of course). And as we see the preview of the scene on the screen we take our touchpen and draw a line over the preview of the image. For example we mark exactly the edge between the bright sky and the dark foreground. Then we tell the in-camera software that everything above the drawn line shall be exposed at 1/125, whereas everything below the line must be exposed for 1/30.
At this moment what I want the camera to do is the following: All the photosites that I defined for exposing at 1/125 should "simply" be switched on only for this time and then turned off, whereas the photosites set for 1/30 will remain active for this longer time.

And suddenly we have >>1 file<< in which the sky has been exposed for 1/125, while the foreground was exposed for 1/30. I hope my English is good enough to get across what I mean.

Of course the photographic situation could be much more complicated than this and require much more precise adjustments. In that case one could do the same actions through a connected laptop, where the scene could be previewed at very high resolution and really large. Would be nice if the notebook screen was a touchscreen, but hey, it's the future I'm talking about.
In a complicated scenario one could define certain sectors of the frame and assign a specific exposure time individually for each one. The liveview function would give a preview of the settings right away, so that suddenly there would be an aweful lot of control possible at the moment of capture. This could significantly decrease time necessary for postprocessing, of course meaning that capturing an image takes longer. And quite obviously this wouldn't be any good for action photography or so.

I acknowledge that it may be complicated for an in-camera processor to individually address 15 million photosites or so, but one could compromise and make grids of 10x10 pixels, or 20x20, which should still give enough resolution for such a differential exposure treatment.

Now, I would really like to know what you folks think about this idea.

Do you think it could work? Or is this just technically impossible?

Would you welcome such a feature? Or would you think that it take the spirit out of photography?

Do you have other ideas that camera developers should work on in order to revolutionize the field?

I'm really looking forward to all your comments.

Thanks for listening
Heiko

[masl]

Fun...
Your idea is fine, but if it works you necessarily lose 1/2 of the photo sites for resolution.  My thought is to record the "time to saturation" for each pixel.  Considering short exposure times (1s and faster) the luminosity is linear with time and thus the intensity of the brightest pixel can always be normalized to be brought back into the "dynamic range" of the entire capture... problem is the image engine does massively more processing with much more data than currently.  I'm guessing these ideas have both died on the vine in every imaging R&D center in the world but it's still a fun game. ;^)
-Mark

[Slough]

Probably technically impossible, or at least such a niche product that it would not be worth the cost. An alternative is to have an auto bracket feature. At present cameras generally use only 12 bits to store data in a RAW file. There is not reason in principle why a camera could not take four pictures exposed 1 stop apart, and then the processor combines the image data to get an enhanced dynamic range true 16 bit file. But this will not happen, mainly because we can do it anyway on a PC using multiple images.

[BJL]

My thought is to record the "time to saturation" for each pixel.
[a href=\"index.php?act=findpost&pid=208948\"][{POST_SNAPBACK}][/a]

There are several "extreme subject brightness range" technologies around in special purpose sensors and research projects, one similar to your idea. Roughly, each photosite is checked at various times during the exposure (maybe 1/16th full exposure time, then 1/8 full exposure time, then 1/4 ...) When a photosite is close to full, its signal is output and the duration of exposure at that pixel is taken into account by a bit shift of the output level after A/D conversion: no shift for full exposure time, 1 bit shift left for 1/2 exposure time etc.

This approach is used in a sensor marketed for use in security cameras; the extra trick is doing A/D conversion at each pixel, so the bit shifts are also applied right there.


Apart from the details of various possible approaches, there seem to be no fundamental obstacles to greatly expanding highlight headroom and thus greatly expanding dynamic range if there is sufficient demand: after all, this is about measuring large amounts of light, a relatively easy task.

The greatest barrier to progress is at the other end: measuring low levels of light due to either dimly lit subject matter or high shutter speed needs. There sensors might be getting close to the situation where the main limit is the light gathering power of the camera, measured by the effective aperture area of the lens, often roughly the area of the front element of the lens.


P.S. As Slough has hinted, the demand for very extreme sensor DR is limited to cases when
1. Subject Brightness Range is extremely high; far higher than a print can show without extreme contrast compression, or other  manipulations,
and
2. One cannot use the HDR blending approach by working with several exposures, taken with the camera suitably stabilized by a tripod or whatever.

And maybe one day, that "whatever" could be in-camera stabilization good enough to allow the several exposures needed for HDR blending to be taken hand-held in a short burst when the subject is not moving much.

[Plekto]

Fuji already does this with their most advanced sensor, or close to it.  It's a really neat idea.  Yes, you lose half of your resolution, but the dynamic range is far closer to film, especially in overexposed parts(there's data to recover when you alter it in Photoshop or similar programs).

Overlapping grid of elements(technically in a hexagonal arrangement - not typical "Bayer" design).  Takes two shots at once and blends them together, using the best data  from each shot.  It really does amazing things.  Just... too low MP to replace film, due to the "12MP" sensor really working as 6MP.

[Der Oak]

Hi guys,

thanks a lot for your thoughts and input. There is something that I don't understand, possibly due to lack of technical knowledge:

Your idea is fine, but if it works you necessarily lose 1/2 of the photo sites for resolution.

Why would I use half the resolution? Let's think of an extremely simple scenario, where I am photographing a rectangle with a white upper half and a black lower half. My goal is to get an exposure, where both halfs are 50% grey.
Now I look at the liveview preview of my future camera and see what is actually in front of the lens: a rectangle, top white, bottom black. Now I work on my beautiful touchscreen display and define two sectors: top half and bottom half. From now on I want the camera to treat the upper half of the actual sensor individually from the lower half, but only in terms of how long each photosite will be active in order to record information.

In order to get overall 50% grey I have to expose the upper half quite short and the lower half quite long. Let's say the top for 1/500, and the bottom for 1/4. If I now take the picture the shutter will be open for the longest exposure that I have defined, i.e. 1/4 of a second. But only the bottom half photosites will actually be "on" for that long time, so that they can nicely receive the information I want them to receive (so that the black is recorded as grey (overexposed so to say)). The top half of the sensor will be turned off after 1/500 and for the rest of the exposure will not be able to record any further information (as if putting on a black mask). This way it is ensured that  the white area of my subject is underexposed and will look grey instead of white.
As all this information is recorded into the same file, I should end up with a fully grey photo.

I'm not sure why I should lose half my resolution because of this, but I would be happy for an explanation.

For me this functionality would also be clearly made for very controlled setups where the user controls the settings. The camera cannot know how I want my shot to look like. Possibly I want certain areas to overexpose and others to underexpose. I can't quite picture how an automated function could do that, so this is not what I intend to propose. In the above scenario I may want to revert the picture and get the top half black and the bottom half white (a bit of an extreme example, but just for illustration).

Now, of course all that work can be done in Photoshop by blending different exposures etc. But in complicated situations this may be difficult or at least laborious.
So in the end it would be up to personal taste in which way to get the desired shot. But personally I find trying to blend several exposures cumbersome and not so pleasent. I would rather compose my near-perfect exposure combination right when I'm on location. And at least here in Greece I frequently encounter scenarios where I would just love such a feature.

In any case, thanks a lot for your thoughts and input!
Heiko

[BJL]

There link below describes another idea planned for commercialization by Panasonic who claim a stunning 140dB, which is 10,000,000:1 or 23 stops! Once again, security video cameras seem to be the first target market.
To quote,

The new technology is based on a multiple exposure technique. A dynamic range of 140dB is compressed in each pixel by overlaying multiple shots of the same scene captured with different exposure times. Such an on-chip wide dynamic range technology provides high frame rates particularly suited for movies and reduces the system cost drastically by eliminating external frame memories. Panasonic intends to apply this new technology to the next-generation νMaicovicon® series, its major product line of MOS imagers, aiming at automotive and security systems.

http://panasonic.co.jp/corp/news/official....en080205-2.html

[dalethorn]

I think the way to go would be a direct scan camera with a small control board much like a recording console.  Then you can point to various areas in the image and increase the sensitivity and other parameters, and adjust the falloff or gamma for each selection, then when all is well, push the button and record the image to the media.

[telyt]

At present cameras generally use only 12 bits to store data in a RAW file.

The Leica DMR uses 16 bits.  I'm very pleased with its dynamic range.