Pages: [1]   Go Down

Author Topic: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack  (Read 11826 times)

fdisilvestro

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1852
    • Frank Disilvestro

Thanks to the work by Nikon Hackers, it is possible to get the original unprocessed raw data from selected Nikon cameras. This hack is available for the previous generation of Nikon cameras (sorry, no D810 or D7200) apparently due to a recent hardware change in the new bodies.

There are three known manipulation performed to NEF files which could not be overridden by the user:

- Black level - truncation of negative values (not performed in new bodies such as the D810)
- White Balance Preconditioning
- Hot pixel suppression (usually for speeds below 1/4 sec)

The hack bypasses those processes, storing basically a raw dump from the sensor

There are two versions of this:
1) Temporary change which stores a hack in the camera RAM and it is reset when the camera turns off, and
2) Hacked firmware

I have tried option 1, since I thought it would be safer

I took a couple of samples with and without the hack with a Nikon D800:

a) Dark frame, 30 Sec @ ISO 100
b) Intentional blow out of all channels
c) Regular exposure at 1/5 sec @ ISO 100

Here are the resulting histograms using rawdigger

Dark Frame
These two images are with the hack, note how the lower values are centered around 600 and negative (relative) values are present. The difference between the images is just the level of detail




The following set of images show the effect of the Hot pixel suppression (HPS) algorithm. Without the hack, the HPS has suppressed any value above 349, while in the hacked file there are values up to maximum (16383)




Blown channels:
In regular NEF files, the channels max out at different values, due to the WB Preconditioning, that multiplies the R and B channels. In the case of the hacked file, all channels max out at exactly the same level





WB Preconditioning
The following set shows first the effect of the WB Preconditioning, with empty slots at regular intervals in the R and B channels while in the hacked file this does not happen





Is it worth to use this? For general photography maybe not (further testing is necessary) but for scientific purposes or astrophotography it is definitely useful. The guys at Nikon Hacker have some impressive samples of astrophotograpies using these hacks.

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog

Very interesting Francisco, I'll take a closer look when I have some time.  Is the ram hack complicated?
Logged

fdisilvestro

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1852
    • Frank Disilvestro

Is the ram hack complicated?

Hi,

The hack is very easy but you need to connect the camera to a computer thru the USB port to load the hack. Just turn off Noise reduction and Long exposure noise reduction in the camera, connect the USB cable and run the program.

Since the hack loads in the RAM, you may want to increase the "Auto meter off" so the camera will not turn off before you take the pictures.

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog

Ok, thanks.
Logged

jejv

  • Newbie
  • *
  • Offline Offline
  • Posts: 13
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #4 on: June 16, 2015, 02:50:34 pm »

Oh dear, Nikon-San.

Nikon3 seems to have taken upon itself to scale fixed-point data.

The problem with scaling fixed-point (e.g. integer) data is that it introduces non-linear distortion.

I am confident that Nikon3 does employ at least some intelligent engineers, who understand that scaling fixed-point data (DAC output) without a significiant increase in precision, or a change to floating point, introduces non-linear distortion.  Which, while insignificant in the highlights, can cause or exagerate pattern noise in the shadows.

Such engineers have my great sympathy.  It must be frustratating to work in an environment where your comments are ignored, or you fear to comment.

Even as analysts suggest that Nikon has less than 5 years to survive, it may be best to keep your head down - at least until you get taken over.  At a smaller company, it may be easier to tell the CEO that the approach is wrong, and needs fixing.
« Last Edit: June 16, 2015, 02:53:22 pm by jejv »
Logged

bernie west

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 323
    • Wild Photo Australia
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #5 on: June 17, 2015, 03:21:03 am »

Mental note:  Upgrade my D810 firmware before 2020..
Logged

jejv

  • Newbie
  • *
  • Offline Offline
  • Posts: 13
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #6 on: June 17, 2015, 07:42:10 am »

@ FranciscoDisilvestro (or @ anyone else who has tried this hack):
- Can you still get sensible white balance in LR or similar ?
- Do you find you can pull up shadows more without patterning effects ?

I had a poke arround in a couple of D7200 NEF files.
They both seemed to have a constant scaling applied across the whole intensity range, with the scaling differing between R/G/B channels.   The scaling seems to differ from image to image.

Image 192:
R/G/B Scaling: 1.206/1.025/1.196

Image 195:
R/G/B Scaling: 1.548/1.321/1.550

Both of these with AWB2 - perhaps that makes a difference.

In image 195, it sort of looks like we could be losing ~0.4EV of dynamic range to the 1.32x green scaling.

The thing that first raised my eyebrows was the fringy histograms from rawdigger - which suggested some numerical problem to me.  I think that the fringes on the full-range, binned, histograms are caused by a kind of "beat" effect between the rawdigger histogram  bin spacing and the underlying histogram gaps caused by Nikon3's scaling.

Logged

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #7 on: June 17, 2015, 10:28:17 am »

Oh dear, Nikon-San.

Nikon3 seems to have taken upon itself to scale fixed-point data.

May I ask what's Nikon3?

As per the scaling, Nikon has been doing what I have heard referred to as WB pre-conditioning (watch the spelling;-) forever or so, as Francisco suggests above.  I don't know why or what they pre-condition to.  As far as I know it is constant in between images at the same ISO and can be calculated off a PTC curve by noting the different full well capacities of the 4 channels.  Most recent Nikon FF DSLRs seem to use 16-bit integer arithmetic internally.

Jack
« Last Edit: June 17, 2015, 10:33:37 am by Jack Hogan »
Logged

jejv

  • Newbie
  • *
  • Offline Offline
  • Posts: 13
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #8 on: June 18, 2015, 06:59:09 pm »

Quote
May I ask what's Nikon3?
Sorry, I was being a bit cranky.  "three" and "Sir" sound pretty much the same in Japanese.

Quote
As far as I know it is constant in between images at the same ISO and can be calculated off a PTC curve by noting the different full well capacities of the 4 channels.
PTC Curve?  A plot of noise against signal level ?  I may be being dim or ignorant, but I don't see why a PTC curve would make one want to scale the raw data.

Quote
As far as I know it is constant in between images at the same ISO
Yes, the two D7200 images I quoted scalings for are at ISO100 and ISO220.
- However, other ISO100 images from the same camera show different - but similar - scalings to the first image I examined.
I'll have  a look at D7100 images and maybe D800 images.

Quote
Most recent Nikon FF DSLRs seem to use 16-bit integer arithmetic internally.
Working with 14bit data, and doing multiplies, would tend to suggest datapaths / registers of >28 bits.
The scaling factors I find in the histograms imply at least 20 bits.

I don't know if RAW converters seek to undo the distortion caused by this strange scaling,
but - except for some pathological images, where removing distortion probably wouldn't affect the final result anyway - the scaling could be inferred from the "RAW" histogram, undone (removing the distortion), and the scaling moved to elsewhere in the processing pipeline, where it can be done at greater precision, without introducing significant distortion.



Logged

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #9 on: June 19, 2015, 06:05:53 pm »

PTC Curve?  A plot of noise against signal level ?  I may be being dim or ignorant, but I don't see why a PTC curve would make one want to scale the raw data.

I don't know why they scale it but if you want to know what the scaling factors are simply divide the green channel FWC by the red and blue and compare to what you see in the raw histograms (I've checked, they are the same).

I don't know if RAW converters seek to undo the distortion caused by this strange scaling,
but - except for some pathological images, where removing distortion probably wouldn't affect the final result anyway - the scaling could be inferred from the "RAW" histogram, undone (removing the distortion), and the scaling moved to elsewhere in the processing pipeline, where it can be done at greater precision, without introducing significant distortion.

Sure.  I am not sure I would call it a distortion as it is just linear multiplication, so it really does not affect the data or IQ any more than any other 16-bit integer operation (the standard in raw conversion these days, with a few notable exceptions like RT).  Think of it as getting your white balance half way there.  Not needed but doesn't hurt either imo.

Jack
Logged

mouse

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 260
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #10 on: June 19, 2015, 07:03:14 pm »

Sorry, I was being a bit cranky.  "three" and "Sir" sound pretty much the same in Japanese.

 :D I get it.  Nikon3 = Nikonsan.
Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #11 on: June 24, 2015, 04:48:08 pm »

Make pix not graphs.
Logged

jejv

  • Newbie
  • *
  • Offline Offline
  • Posts: 13
Re: Un-manipulated RAW from some Nikon cameras thanks to a firmware hack
« Reply #12 on: June 29, 2015, 02:42:24 pm »

Quote
Make pix not graphs.
Absolutely.

If you have captured a striking and significant image, no-one is going to complain that you should have exposed it +/- 1/3EV differently.
It might only be significant - say - to your own family, or even only to yourself - but that is still significance.

I have quite a few social snaps taken on Canon G2 or G3 (4Mpix) at ~1/5 s which I am very happy with: A blurred hand gesture at a dinner table can bring an image to life.   And the prints look fine at A4, and sometimes A3.

OTOH, we seek to make the best of the equipment we have available to us.

Not working in a studio with controlled - limited dynamic range - lighting, sometimes this means pulling up the shadows a lot.
What Nikon is doing can interfere with that.

I had forgotten, but this has been discussed before, here:  http://forum.luminous-landscape.com/index.php?topic=74178.msg604604#msg604604

I guess what I need to do now is make a little program that tries to de-scale - remove the gaps in - "uncompressed" NEF histograms, and let y'all see if it makes a difference in the shadows.


« Last Edit: June 29, 2015, 02:44:24 pm by jejv »
Logged
Pages: [1]   Go Up