Dark Frame Subtraction - how it works

[NigelC]

Just got a couple of queries on this. As I understand it, assuming this i swhat long exposure NR on 5D/5D2 is, camera takes a second exposure of equal length, sequentially with the shutter closed. Therefore , deliberate subject movement. e.g. water/clouds, will not be affected (talking about exposures of say a few minutes with ND filter). However, how would this work with say star trails where shutter would be open for a long time (hours) - would the camera lock up until finished?

Hasten to say haven't done very long exposures with a digital camera - last time I had the urge it was with film where only had to think about reciprocity.

[michael]

Digital cameras are just not suitable for really long exposure astro photography. Film is a much better choice for multi-hour star trails.

For normal astro work digital is great, but the way people do it is by stacking multiple short exposures and using a tracking system.

Michael

[NigelC]

Thanks, also heard somewhere that v. long exposure, i.e. hours, could trigger (irreversibly) hot pixels.

[RFPhotography]

There are digital cameras specifically made for astrophotography that incorporate active cooling.  They're very expensive.

Some will take multiple exposures using a tracking telescope then stack those together after the fact.  

There are some software programs that will do a good job of arranging and stacking a set of images taken with the camera on a tripod.  You end up with a smaller field of view because only the overlapping areas end up with the best result.  Deep Sky Stacker is a free option to play around with.  You can take multiple exposures over a period of time with a short break in between to allow the camera to cool itself.  That won't create the star trail effect though.

Stacking individual images, as Michael said, can generate the star trails, as in the attached.  Note the gaps between the trail segments.  That's because my 'break' between exposures was too long.

You can also make your own dark frame images, create a library of them, then do the dark frame subtraction in Photoshop rather than doubling your shooting time in the field.

EDIT:  Just as a quick example, the second image attached is a result from Deep Sky Stacker.  I can't remember how many shots I loaded into the program, but not that many.  I didn't get far enough out of the city so there's some obvious light pollution.  You can see the result of the program rotating the images to stack them in the tree on the right side.  The upper middle section is the Milky Way galaxy.  In the upper right, it's difficult to see on this small image, but there's another small galaxy (I forget the name at the moment).  These were 30 second exposures at f4/ISO 800.  Another time I'd open up to f2.8 and shoot at ISO 1600.  And get further away from the lights of the city, of course.  This is considered 'deep sky' photography. 

If you want to see some really cool deep sky photography incorporated into video, check these links out.

[andyptak]

Check out Harold Davis from San Franciscio. He does this all of the time, has books and courses about it.

[ejmartin]

I don't think you want to do dark frame subtraction in PS.  It should be done in linear gamma, preferably on the raw data prior to demosaic.

[RFPhotography]

It's been done for some time now.  JP Caponigro also wrote about it recently in DPP, http://www.digitalphotopro.com/technique/revolution/noise-lose-it.html.  Can't be that bad. 

[ejmartin]

My point was that subtraction is most accurate when done in linear gamma, since that is where dark current noise is additive.  In other words

(signal + dark current) - (dark current) = signal

but if you are subtracting after gamma compensation,

(signal + dark current)^(1/gamma) - (dark current)^(1/gamma) ≠ signal^(1/gamma)

Dedicated astrophotography software does the subtraction in linear gamma.  RawTherapee 3.0 (alpha) does the subtraction in linear gamma.

[RFPhotography]

I do understand your point.  The point I was making was that it could still be done on non-linear images and that it is being done with success.  Caponigro thinks doing it after the fact is better than doing it in camera.  Not everyone has or wants to have dedicated astrophotography software or add a different piece of software to their workflow. 

Perhaps you could post some examples to show the differences.

[bjanes]

I do understand your point.  The point I was making was that it could still be done on non-linear images and that it is being done with success.  Caponigro thinks doing it after the fact is better than doing it in camera.  Not everyone has or wants to have dedicated astrophotography software or add a different piece of software to their workflow. 

Some astrophotography programs (such as the freeware Iris) have demosaicing algorithms more suitable for astrophotography than terrestrial photography. One way to obtain linear data with Photoshop is the method outlined by the ICC. However, this is after demosaicing.

Regards,

Bill

[RFPhotography]

OK, I'll ask again.  Can someone post some examples to show differences in practical situations?

[Guillermo Luijk]

OK, I'll ask again.  Can someone post some examples to show differences in practical situations?

This is not actually a comparision of darkframe substraction in undemosaiced linear vs demosaiced gamma, but it makes it easy to understand why these operations are better done in the undemosaiced domain (Emil explained well why linearity is also desired):

This is the effect of a single hot pixel (the kind of undesired phenomena we intend to minimise with darkframe substractions):




What does this hot pixel look like prior to demosaicing?




After demosaicing, the effect of the hot pixel spreaded around the abnormal photocaptor, affecting as many as 8 pixels with the AHD interpolation algorithm. It's easy to imagine that minimising the effect of the hot pixel for the single photocaptor which created the problem, will always have a better result than substracting as many as 9 pixels that got wrong colours because of a single original abnormal photocaptor.

Regards

[fdisilvestro]

The advantage of doing dark frame substraction in postprocessing is that you need only one dark frame for every shutter speed /ISO/temperature combination.
I also think it is better to do i before demosaicing. A program like dcRaw allows to substract a dark frame at the Raw level
For few images, it is probably better to use in camera long exposure NR.

[RFPhotography]

Guillermo, thanks.  I do understand that.  You're pixel peeping though.  I'm looking for practical examples.  And maybe this should be a separate discussion because it's beginning to hijack Nigel's original question.

When I say 'practical', I'm talking about images viewed in real world conditions, not at extreme magnifications on a computer screen.  Will a client see the difference in a small image used in a brochure or on a website?  Will the difference be seen in a full HD video clip?  Will the difference be evident in a print hanging on a wall behind glass at normal viewing distances? 

Francisco, that's what I was getting at with the concept of a 'library' of dark frame images.  How important are exposure time and temperature from, again, a practical standpoint?  Can, for example, you make a series of images at, say, 10 or 15 or even 20 degree temperature differences?  20 degrees may be stretching it a bit.  Or can you have a set at, say, 30 sec, 60 sec, 120, sec, 5 min, 10 min, 30 min, 60 min as opposed to every shutter speed in between? 

[fdisilvestro]

Dark frames should be taken with conditions as close as possible to the image we want to reduce noise. I don´t think having a permanent set of dark frames would be the best practice.

Noise is affected by temperature, (that´s why they use active cooling in astrophotography).

Another issue is that dark noise not necessarily translates in "saturated pixels". The level of noise then will be a function of exposure time and that´s why the dark frame should be taken with the same shutter speed.

If you have saturated pixels, then this procedure will not recover the signal, it will just leave the final image without the colored or white spots from hot pixels.

Regarding the discussion about doing the dark frame subtraction at the raw level or in PS, my take is that if you want to recover the "signal" as much as possible (a priority for astrophotography) then you should do it at raw level. If you want to get a "pleasing" image (without apparent noise), then doing it in PS should be fine. After reading Caponigro´s article, I think that´s what he is after, especially when he mentions the possibility of adjusting opacity until you get the best result (BTW, I consider Caponigro a master of the medium)

[RFPhotography]

And that 'pleasing image' is what I want to achieve.  And it's what I assumed Nigel is trying to achieve.  If I'm wrong, that's OK.  As opposed to the hardcore astrophotographers who, yes, have a whole slew of dedicated hardware and software.  Which is why I think of the idea of a range of temperatures at certain intervals rather than at every degree.  Same with shutter speeds.  Is it better done at the RAW level?  Sure.  Better done using in camera LENR?  Sure.  But for reasons of, here's that word again, practicality doing it other ways may work.  If getting n^nth degree of image quality improvement doesn't have practical application then while it may be an interesting intellectual exercise to do once, doing it all the time is simply wasting time.

Andy's example of Harold Davis' photography is a terrific one.  Because of the way he shoots, he has LENR turned off.  Don't think anyone can quibble about the exceptional quality of his photography.

[NigelC]

Interesting discussion on astrophotography.

My main interest arose from a realisation that since switched from film to DSLR, hadn't done any exposures beyond  a couple of minutes. Recent acquisition of 10 stop ND filter for landscapes, mainly, indicated I could theoretically end up with extremely long exposures, but more likely to be not exceeding 20 minutes and generally in the 2-5 minutes range. What I think I take from the answers is that DSLRs OK in this range and in camera long exposure noise reduction takes second exposure with the shutter effectively capped and therefore would not overwrite the deliberate subject movement in the first exposure.  (although I understand there may be more optimum approaches involving entirely post capture). I suppose the only question then is if clearly DSLRs in standard form not suitable for very long exposures, e.g hours, where is the effective threshold when the sensor would heat up unacceptably, possibly damaging itself. 5 minutes, 15, 30, an hour? Camera is 5D2, if this is relevant.

[Sussex Landscapes]

 Note the gaps between the trail segments.  That's because my 'break' between exposures was too long.

The gaps are not generated by the exp time. they are produced by using the `lighten blend` mode that is used for basic stacking of images in PS. also used in the startrailes.de software and others i am sure. this method actually softness and fades out the end of the star streak (trail) at both ends, so when put against the next file, you get two faded ends, giving the gap appearance.
There is a work arounds for this, though it takes more pp skills,time and if a lot of images used the final file size.

essentially you use the Lighten Screen Blending (LSB) – Advanced Multi-exposure blending. using linear curves and and doubleing of the files in a stack in PS.
you can find more info on my website, and theres a link to the full article.
http://www.sussexlandscapephotography.co.uk/star_photography_tutorial.html

simon www.sussexlandscapephotgraphy.co.uk

[RFPhotography]

If I have a 30 second exposure then wait 20 or 30 seconds to start the next exposure, there's going to be a gap between the trail segments.  It may be exacerbated by the blend mode but it will be there nonetheless.  Nice tutorial as well.  Not sure I'd say it requires 'advanced' PS skills but it's definitely a good workaround.

[ejmartin]

Why wait so long before starting the next exposure?