Pages: [1] 2 3 ... 5   Go Down

Author Topic: Synthetic HDR  (Read 39952 times)

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« on: March 29, 2007, 08:39:48 pm »

Hi all,

I have been doing a bit of experimentation with HDR recently, and had an idea about synthesizing extra dynamic range (particularly in shadow areas) from a single image rather than a bracketed sequence of images.

I've written it up here:

Synthetic HDR at findatlantis.com

in some detail, but to cut a long story short, it works by using convolution to simulate binning, then generating a series of simulated greater exposures (that are guaranteed not to introduce any quantisation effects), then doing a standard HDR merge and tone mapping of the resulting image. The initial results are encouraging -- I'll include an example here, but there's more detail on my web site.

Here's the original test image:



Base image

It was shot with a Bronica ETRS, 100mm macro on an auto bellows, Megavision E4 Monochrome back (16 megapixel, 4096x4096, 12-bit).



Synthetic HDR image

This is an effectively 20-bit synthetic HDR version of the same image (no other images were used), tone mapped back to 8 bits for web purposes.

For comparison, here's a 'real' HDR image made the conventional way by doing a Photoshop HDR merge on a series of 7 separate exposures, 1 stop apart:



Real HDR image

There's not much to choose between the two, really -- most of the difference is down to different tone mapping settings, which could probably be overcome with a bit of dodging and burning.

Rather more telling is comparing (slightly out-of-focus) 100% crops from the bottom right hand corner:



Original image, 100% crop, auto-levels applied

This is the original image, cropped and adjusted with Levels. Compare this with the same area of the synthetic HDR image:



There's surprisingly little to choose between the two in terms of perceived resolution, but in the synthetic HDR version there is no sign of the horrible pixellation or posterisation, just something looking not unlike film grain. In this example there are some slight artifacts from my having used rectangular convolution kernels, but this could easily be avoided by using circular kernels instead.

I don't think this technique can necessarily replace HDR, but it does appear to be able to regain shadow detail in single frames surprisingly well. I'm considering implementing this algorithm as a Photoshop plugin, but in the mean time, the web page I linked above tells you enough to be able to reproduce my results.
Logged

laughingbear

  • Full Member
  • ***
  • Offline Offline
  • Posts: 232
Synthetic HDR
« Reply #1 on: March 29, 2007, 09:28:49 pm »

Quote
Here comes the big idea: perhaps we could simulate different exposures with binning, then use established HDR techniques to recombine them into a final image, such that the inevitable loss of resolution is just confined to shadow areas?

Hello Sarah,

Wow, it appears as if you are onto something there, most impressive!

I was just thinking about this "limitation" of shooting exposure brackets as base material for HDR landscapes. I mean, nice cloud patterns on a perfectly windstill day work fine, but how often do we find that?

Thanks a lot for sharing your findings. A photoshop plugin would sure be a very handy solution.

Best whishes
~^..^~
Bear
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #2 on: March 29, 2007, 10:32:12 pm »

Quote
Hello Sarah,

Wow, it appears as if you are onto something there, most impressive!

I was just thinking about this "limitation" of shooting exposure brackets as base material for HDR landscapes. I mean, nice cloud patterns on a perfectly windstill day work fine, but how often do we find that?

Thanks a lot for sharing your findings. A photoshop plugin would sure be a very handy solution.

Best whishes
~^..^~
Bear
[a href=\"index.php?act=findpost&pid=109514\"][{POST_SNAPBACK}][/a]

I think a PS plugin could potentially do a much better job, without any of the artifacts associated with the HDR merge or tone mapping, because I can figure out all of the maths to get exact solutions. My ideal would be something like, put PS into 16-bit mode (if it isn't already), then have the plugin synthesize any missing low-order precision, whilst keeping the image bit-for-bit exactly the same as the original except for shadow data. This should mean that you'd get essentially no artifacts whatsoever, just much more (cleaner) shadow information to play with in dodging and burning. It's a little different in philosophy to traditional HDR, I suppose, but I can see this being really good for dealing with printing difficult images. I'm not so concerned with getting into the more stylised (dare I say it a bit cliched?) applications of HDR -- my thing is fine art photography, and my motivation for this is trying to make better prints (better performances of the score, to misquote Ansel Adams), if you see what I mean.
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Synthetic HDR
« Reply #3 on: March 29, 2007, 10:40:47 pm »

Is it just me, or do both real HDR and simulated HDR images show very large areas of completely burned highlights?

Shadow detail is important, but IMHO HDR should focus on preserving the highlights while keeping some details in the shadows.

What I see as a problem with the current samples is probably just a matter of choice when mapping back from high bit to 8 bits, but I would personnaly find the current images unusable.

Regards,
Bernard

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #4 on: March 29, 2007, 10:58:13 pm »

Quote
Is it just me, or do both real HDR and simulated HDR images show very large areas of completely burned highlights?

Shadow detail is important, but IMHO HDR should focus on preserving the highlights while keeping some details in the shadows.

What I see as a problem with the current samples is probably just a matter of choice when mapping back from high bit to 8 bits, but I would personnaly find the current images unusable.

Regards,
Bernard
[a href=\"index.php?act=findpost&pid=109520\"][{POST_SNAPBACK}][/a]

I'm not so keen on that either, but I was concentrating on shadow detail here. I'm not that convinced of how good Photoshop's Merge-to-HDR feature actually is, to be honest -- one of the reasons I want to implement this as a plugin is that I could bypass that entirely. This article is really a proof-of-concept based on simulating the concept with manual Photoshop-prodding rather than anything definitive. With a plugin, it should be possible to keep the most-significant bits of the brightness information bit-for-bit identical to the original image, and just synthesize extra bits within shadow areas. As for tonality, it would then just basically be up to you to do your thing with levels and curves as normal -- the technique itself wouldn't affect that in any way. It should be very fast, too, because it won't need to do any significant maths outside shadow areas.

Also -- an important caveat: none of these images are intended to be great art. It's just that they happened to have the right kinds of properties, and it happened to be the only image sequence across a full 7 stops that I had to hand for comparison. Where I think great art *could* come out of this is in coaxing more and better shadow detail out of difficult images in printing -- my gut feeling is that a plugin could be a godsend for this. It's also potentially the kind of thing that you could build into a camera's onboard DSP firmware.
« Last Edit: March 29, 2007, 11:07:22 pm by st326 »
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Synthetic HDR
« Reply #5 on: March 29, 2007, 11:12:31 pm »

Good point, Bernard. There appears to be a few specral blown highlights in the original and one larger than specral blown highlight.

All of these seem to have been significantly enlarged in both the synthetic and real HDR images.
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #6 on: March 29, 2007, 11:14:04 pm »

Quote
Good point, Bernard. There appears to be a few specral blown highlights in the original and one larger than specral blown highlight.

All of these seem to have been significantly enlarged in both the synthetic and real HDR images.
[a href=\"index.php?act=findpost&pid=109526\"][{POST_SNAPBACK}][/a]

As I mentioned previously, this is really due to Photoshop's crappy HDR merge. The plugin won't have this problem, if for no other reason that it'll only actually affect shadow areas anyway.
« Last Edit: March 29, 2007, 11:15:56 pm by st326 »
Logged

Tom.D.Arch

  • Newbie
  • *
  • Offline Offline
  • Posts: 24
Synthetic HDR
« Reply #7 on: March 30, 2007, 12:57:09 am »

Quote
<clip> ...  then have the plugin synthesize any missing low-order precision, whilst keeping the image bit-for-bit exactly the same as the original except for shadow data. This should mean that you'd get essentially no artifacts whatsoever, just much more (cleaner) shadow information to play with in dodging and burning. <clip>
[a href=\"index.php?act=findpost&pid=109519\"][{POST_SNAPBACK}][/a]

Visually, your example looks great - no amazing.  Somehow, though, you should be running up against some law of 'information theory'.  Either the information is there in the original capture, or it isn't.  In theory you shouldn't be able to "synthesize any missing low-order precision", particularly in comparison with 'normal' HDR merge, where there really is more information to draw from.  Nonetheless, the proof seems to be in the pudding - this is really interesting!
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #8 on: March 30, 2007, 01:35:54 am »

Quote
Visually, your example looks great - no amazing.  Somehow, though, you should be running up against some law of 'information theory'.  Either the information is there in the original capture, or it isn't.  In theory you shouldn't be able to "synthesize any missing low-order precision", particularly in comparison with 'normal' HDR merge, where there really is more information to draw from.  Nonetheless, the proof seems to be in the pudding - this is really interesting!
[a href=\"index.php?act=findpost&pid=109538\"][{POST_SNAPBACK}][/a]

There's no violation of information theory here. I'm not peddling perpetual motion. ;-) There's simply a tradeoff between resolution and bit depth -- normally, this would result in a poor quality image, but by restricting the effect to deep shadow areas, there should be no visible degradation to brighter parts of the image, whilst retaining more shadow detail due to the greater dynamic range. It's not inventing any information -- just kind-of spreading it out in a way that looks better to human visual perception, if that makes any sense.
« Last Edit: March 30, 2007, 01:36:23 am by st326 »
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Synthetic HDR
« Reply #9 on: March 30, 2007, 01:44:29 am »

Quote
Either the information is there in the original capture, or it isn't.  In theory you shouldn't be able to "synthesize any missing low-order precision", particularly in comparison with 'normal' HDR merge, where there really is more information to draw from.  [a href=\"index.php?act=findpost&pid=109538\"][{POST_SNAPBACK}][/a]

That's my view also. There are many ways to skin a cat. This is obviously one of them, but we need more variations of the pudding, using different processing techniques, to determine if any ultimate advantage has been acheived.
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #10 on: March 30, 2007, 02:05:43 am »

Quote
That's my view also. There are many ways to skin a cat. This is obviously one of them, but we need more variations of the pudding, using different processing techniques, to determine if any ultimate advantage has been acheived.
[a href=\"index.php?act=findpost&pid=109545\"][{POST_SNAPBACK}][/a]

It's not opinion on my part, it's mathematics.

I'm not particularly surprised that some people here reacted negatively to the idea -- having lurked here for quite a while now, it seems to be pretty much par for the course. Nevertheless, I will go ahead and write the plugin. I intend to give it away for free, so people can use it or not, it's entirely up to them.

"Don't worry about people stealing an idea. If it's original, you will have to ram it down their throats." -- Howard Aiken
« Last Edit: March 30, 2007, 02:10:28 am by st326 »
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Synthetic HDR
« Reply #11 on: March 30, 2007, 02:23:29 am »

Quote
It's not opinion on my part, it's mathematics.

I'm not particularly surprised that some people here reacted negatively to the idea -- having lurked here for quite a while now, it seems to be pretty much par for the course. Nevertheless, I will go ahead and write the plugin. I intend to give it away for free, so people can use it or not, it's entirely up to them.

"Don't worry about people stealing an idea. If it's original, you will have to ram it down their throats." -- Howard Aiken
[a href=\"index.php?act=findpost&pid=109548\"][{POST_SNAPBACK}][/a]

Is your name really Sarah? Do a full job on any particular image of your choice. Send me the RAW file via PM, but not too large a RAW file, and let's compare your synthetic HDR with my photoshopped version. Okay?
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Synthetic HDR
« Reply #12 on: March 30, 2007, 02:29:22 am »

Better still. Make the RAW image available to everyone through a link, so we can all give your theory a good test and comparison.
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #13 on: March 30, 2007, 02:33:25 am »

Quote
Is your name really Sarah? Do a full job on any particular image of your choice. Send me the RAW file via PM, but not too large a RAW file, and let's compare your synthetic HDR with my photoshopped version. Okay?
[a href=\"index.php?act=findpost&pid=109550\"][{POST_SNAPBACK}][/a]

As it stands, I don't think an approach based on Photoshop's HDR merge is worth doing any serious comparisons with -- I'm not really happy with its results. Its results seem a bit off somehow -- I think it is including too much signal from the higher exposure frames, which is what's washing out the highlights. It happens on traditional HDR images too, it's not just mine. It's enough to demonstrate proof of concept, but that's all, which is why I intend to release a plugin that implements the algorithm optimally. Once that's available, you are welcome to compare it however you like.

And, to the rest of your question, yes, my name is Sarah, by day job I'm a rocket scientist (yes, really) at NASA Ames. I have a Cambridge PhD in space electronics, and a couple of decades experience in software engineering and electronic design, including a fair bit of signal processing, so I'm not just making this stuff up!
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #14 on: March 30, 2007, 02:41:30 am »

Quote
Better still. Make the RAW image available to everyone through a link, so we can all give your theory a good test and comparison.
[a href=\"index.php?act=findpost&pid=109551\"][{POST_SNAPBACK}][/a]

I'm uploading it, and a full res version of the result I got. It'll take a while (the files aren't small), so I'll post links once they are up there.
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Synthetic HDR
« Reply #15 on: March 30, 2007, 02:41:47 am »

Quote
As it stands, I don't think an approach based on Photoshop's HDR merge is worth doing any serious comparisons with -- I'm not really happy with its results. Its results seem a bit off somehow -- I think it is including too much signal from the higher exposure frames, which is what's washing out the highlights. It happens on traditional HDR images too, it's not just mine. It's enough to demonstrate proof of concept, but that's all, which is why I intend to release a plugin that implements the algorithm optimally. Once that's available, you are welcome to compare it however you like.

And, to the rest of your question, yes, my name is Sarah, by day job I'm a rocket scientist (yes, really) at NASA Ames. I have a Cambridge PhD in space electronics, and a couple of decades experience in software engineering and electronic design, including a fair bit of signal processing, so I'm not just making this stuff up!
[a href=\"index.php?act=findpost&pid=109553\"][{POST_SNAPBACK}][/a]

I'm impressed (are you still single? - okay ignore that if you wish). What you are saying is, at this point in time you don't have more than a substandard prototype image. If that's the case, we can't really comment further, can we?
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Synthetic HDR
« Reply #16 on: March 30, 2007, 02:50:38 am »

Quote
I'm uploading it, and a full res version of the result I got. It'll take a while (the files aren't small), so I'll post links once they are up there.
[a href=\"index.php?act=findpost&pid=109559\"][{POST_SNAPBACK}][/a]

Okay! Thanks! I Look forward to it.
Logged

kal

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 63
    • http://fotonordest.blogspot.com
Synthetic HDR
« Reply #17 on: March 30, 2007, 03:05:58 am »

Quote
There's no violation of information theory here. I'm not peddling perpetual motion. ;-) There's simply a tradeoff between resolution and bit depth -- normally, this would result in a poor quality image, but by restricting the effect to deep shadow areas, there should be no visible degradation to brighter parts of the image, whilst retaining more shadow detail due to the greater dynamic range. It's not inventing any information -- just kind-of spreading it out in a way that looks better to human visual perception, if that makes any sense.
[a href=\"index.php?act=findpost&pid=109540\"][{POST_SNAPBACK}][/a]

This makes a lot of sense. In fact it is quite similar to what some point-and-shoot cameras do when they offer iso 1600 or higher, but only at reduced resolution. You are doing the same thing, but selectively on dark areas.

The only real problem I see here is that this approach will work when the interesting part of the image is "correctly" exposed, and you only need to bring out some detail from possibly out-of-focus shadows. If the interesting part of the image is in the shadows, we are going to have a much harder time, since as you said we are trading resolution for bit depth.

Great work anyway, I wish I had thought it before!
Logged
Piero
[url=http://fotonordest.blogspot.c

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #18 on: March 30, 2007, 03:41:59 am »

OK, here are the links to the files (warning, moderately big!)

http://findatlantis.com/base-image.tif

http://findatlantis.com/tonemapped16bitversion.psd

I uploaded a 16-bit TIFF for the source image, partly because that's the exact same image I started with when I was doing the synthetic HDR experiments, and partly because most third-party software makes a mess of decoding monochrome DNG files (this is the Megavision's native format, but Adobe's own DNG decoder insists on trying to decode them as colour images (it tries to interpolate for a Bayer matrix that actually isn't there), so you lose quality and gain weirdness unless you process them with Megavision's software).
« Last Edit: March 30, 2007, 03:53:12 am by st326 »
Logged

st326

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
Synthetic HDR
« Reply #19 on: March 30, 2007, 03:43:58 am »

Quote
This makes a lot of sense. In fact it is quite similar to what some point-and-shoot cameras do when they offer iso 1600 or higher, but only at reduced resolution. You are doing the same thing, but selectively on dark areas.

The only real problem I see here is that this approach will work when the interesting part of the image is "correctly" exposed, and you only need to bring out some detail from possibly out-of-focus shadows. If the interesting part of the image is in the shadows, we are going to have a much harder time, since as you said we are trading resolution for bit depth.

Great work anyway, I wish I had thought it before!
[a href=\"index.php?act=findpost&pid=109564\"][{POST_SNAPBACK}][/a]

Thanks. :-)

I'm not expecting this to be something you'd necessarily use on every image, but for the ones where it is relevant (i.e. where you want to pull out a bit more shadow detail and normal dodging and burning or contrast manipulation is giving you too much noise, posterisation and/or Mach banding) I think it will probably be pretty useful.
Logged
Pages: [1] 2 3 ... 5   Go Up