Gamma corrected output in DCRAW

[Guillermo Luijk]

DCRAW is known for being a linear-output RAW developer. David Coffin claims there is no improvement in applying gamma before the final 16-bit integer rounding. Images produced by DCRAW can be de-linearized in PS just by converting from the output colour profile into the same one but with gamma, and it works.

However this creates the expected holes in the histogram which mean lower tonal richness in the left end. Another fellow and me have introduced a -g option into DCRAW so that -g 2.2 (or any value) can be used to apply a gamma curve to the output right before the final rounding to 16-bit integer.

I wonder if the improvement will be useful in the end or not, but looking at the histograms they are simply superb, all levels get nicely filled.

Moreover we have just applied a perfect gamma curve (OUT=IN^(1/2.2)) which in sRGB means more expansion in the deep shadows of the histogram so it's easier to handle them preventing black aggregation. Of course a specific sRGB-like profile but with the exact gamma parameter has to be assigned so that the image looks the same as developed and assigned in the normalized sRGB.

This is the comparision between the exact and sRGB 2.2 gamma:



I will offer the dcraw.exe for download soon.

The changes applied in the code don't go very far from just doing:
FORC3 img[c] = CLIP((int) 65535.0*pow(out[c]/65535.0,1/user_gamma));

instead the original DCRAW code:
FORC3 img[c] = CLIP((int) out[c]);


These are the histograms when developing linear sRGB and converting to non-linear sRGB in PS:
(top:general view, bottom: 1:2 zoom view, maximum zoom for 15-bit PS images)
dcraw -o 1 -4


And these are the same areas of the histogram when using the new non-linear DCRAW:
dcraw -o 1 -g 2.2 -4


Increased tonal richness and precision, and easier to handle shadows with PS curves (since they don't remain so close to 0) in sRGB.

This is what the image looks like:



Left: developed in linear sRGB and converted again into sRGB in PS (de-linearized)
Centre: developed in pure 2.2 gamma with DCRAW and assigned to sRGB (shadows look lifted)
Right: same as centre but assigned to a specific sRGB-like profile but using exact 2.2 gamma. Looks identical as left but with the higher quality histogram.

Do you think this is useful?

BTW looking at ACR originated histograms, they also contain big holes in the shadows what makes me think ACR does not stay floating point up to the very last stage, but rounds to integer before the gamma.

[Samotano]

As usual, great scientifically stimulating work.
Adobe, are you going to hire this fellow?!  

[Guillermo Luijk]

As usual, great scientifically stimulating work.
Adobe, are you going to hire this fellow?! 

It is Coffin who Adobe should hire  

The coder is adding a -g srgb option so that the specific sRGB gamma curve is applied. In that way:

dcraw -o 1 -g 2.2: output sRGB with standard 2.2 gamma
dcraw -o 1 -g srgb: output sRGB with sRGB gamma (can be directly assigned to sRGB in PS)
dcraw -o 2 -g 2.2: output Adobe RGB with standard 2.2 gamma (can be directly assigned to Adobe RGB in PS)
dcraw -o 4 -g 1.8: output ProPhoto RGB with standard 1.8 gamma (can be directly assigned to ProPhoto RGB in PS)

[Samotano]

It is Coffin who Adobe should hire 

The coder is adding a -g srgb option so that the specific sRGB gamma curve is applied. In that way:

dcraw -o 1 -g 2.2: output sRGB with standard 2.2 gamma
dcraw -o 1 -g srgb: output sRGB with sRGB gamma (can be directly assigned to sRGB in PS)
dcraw -o 2 -g 2.2: output Adobe RGB with standard 2.2 gamma (can be directly assigned to Adobe RGB in PS)
dcraw -o 4 -g 1.8: output ProPhoto RGB with standard 1.8 gamma (can be directly assigned to ProPhoto RGB in PS)
[a href=\"index.php?act=findpost&pid=193471\"][{POST_SNAPBACK}][/a]

Actually, "apparently" Thomas Knoll did try, but he wasn't interested...

[ostonica]

Thanks very much for this most interesting information Guillermo. Will you be posting a patch to the dcraw source for this? when you post the changed dcraw.exe?

From what you say, it appears that the WB multipliers are applied before the demosaicing takes place. In a different thread, Eric Chan indicated that it came after in ACR?

Thanks again

Jim

[Guillermo Luijk]

Thanks very much for this most interesting information Guillermo. Will you be posting a patch to the dcraw source for this? when you post the changed dcraw.exe?

From what you say, it appears that the WB multipliers are applied before the demosaicing takes place. In a different thread, Eric Chan indicated that it came after in ACR?

At least in DCRAW white balancing is one of the first operations performed, it comes before demosaicing. In fact if you white balance the linear image in PS AFTER demosaicing (this can be done with straight curves over the linear TIFF), colour artifacts appear:

WB before demosaicing:


WB after demosaicing (green artifacts appear):


It's logical to think all RAW developers work the same way, doing white balance before interpolation.


I will offer the new dcraw.exe for download as soon as it is 100% ready (we have some minor issue with the sRGB gamma function which is a bit different to the standard 1/2.2 function).

It will also include a -l option that pre-balances the G1 and G2 channels. Usually those channels are already balanced and that's why m2 and m4 multipliers in the WB trend to be the same. This is very important to avoid maze artifacts very common in the AHD algorithm (which aside this issue is a great interpolation algorithm), and this is specially common in Olympus cameras.

This is a simulation of maze artifacts when G1 and G2 channels are unbalanced:

• Automaitc WB: DCRAW balances G1 and G2 -> some noise but rarely mazes
• We force 1% deviation -> some mazes appear
• We force 5% deviation -> a lot of maze artifacts

Mazes are inherent to the way AHD works for being basically a horizontal and vertical pattern enhancer, so in noisy areas it can confuse noise with vertical or horizontal elements of the scene reproducing them. The mazes can be avoided with the -f option which considers G1 and G2 channels separately, but -f forces to use the VNG algorithm which has much less quality than AHD.

I will try to convince Coffin to introduce these changes in the original dcraw.c code, at least the -g option for gamma (I estimate just 1% probability of success hehe).

I have to say all the implementation work is being greatly done by another forum member, Manuel Llorens. I just give him ideas.

BR

[ostonica]

Many thanks for these explanations, Guillermo. I find them most interesting.

One reason I started to look at the new version of the DXO program was that the marketing blurb put so much stress on a new demosaicing algorithm that looked at 5 ? layers of surrounding pixels in determining the missing RGB values. The refer to it as a non-local algorithm. They implied that other converters used fewer surrounding pixels. But AHD also goes several layers out, doesn't it?

Have you compared the results of the DXO algorithms with those of the AHD algorithm. I have only recently started to do that and I like the results of the DXO converter better.

There are several comparisons on the DXO site,
http://www.dxo.com/intl/photo/dxo_optics_p...w_raw_converter
but those are all chosen to show up the DXO program

Thanks again

Jim

[ejmartin]

DCRAW is known for being a linear-output RAW developer. David Coffin claims there is no improvement in applying gamma before the final 16-bit integer rounding. Images produced by DCRAW can be de-linearized in PS just by converting from the output colour profile into the same one but with gamma, and it works.

However this creates the expected holes in the histogram which mean lower tonal richness in the left end. Another fellow and me have introduced a -g option into DCRAW so that -g 2.2 (or any value) can be used to apply a gamma curve to the output right before the final rounding to 16-bit integer.

I wonder if the improvement will be useful in the end or not, but looking at the histograms they are simply superb, all levels get nicely filled.

Left: developed in linear sRGB and converted again into sRGB in PS (de-linearized)
Centre: developed in pure 2.2 gamma with DCRAW and assigned to sRGB (shadows look lifted)
Right: same as centre but assigned to a specific sRGB-like profile but using exact 2.2 gamma. Looks identical as left but with the higher quality histogram.

Do you think this is useful?

The holes in the histogram are not going to have an effect, since the noise exceeds the quantization step and will dither the level quantization.  So there should be no advantage to your modification.

You may be seeing a difference in shadows because sRGB gamma transformation is not a pure power law there;  see
http://brucelindbloom.com/
go to "Math" section and click on XYZ to RGB to get the formula, the transformation is linear in deepest shadows rather than the gamma curve.

So no, I don't think it's useful, but I'm open to a quantitative demonstration that the filled-in levels can make a difference.  I'm just skeptical that such a demonstration is possible due to the dithering effect of the noise.  When one stretches the levels by a gamma transformation, one stretches the noise histogram too, and so long as the noise was greater than the quantization step before the gamma transformation, it will be greater than the quantization step after gamma correction and will still dither the jumps.

[Guillermo Luijk]

So no, I don't think it's useful, but I'm open to a quantitative demonstration that the filled-in levels can make a difference.  I'm just skeptical that such a demonstration is possible due to the dithering effect of the noise.

Mi idea is: if you can apply gamma before the final 16-bit rounding in DCRAW, why not do it? it will never be worse, and perhaps can benefit us some way (just to start the histogram is much more pleasant). Maybe we are increasing the robustness of the histogram against posterization or gain some detail in shadows gradation.

For instance when my program Zero Noise in the first version didn't apply gamma on output, the resulting 16-bit TIFF files could simply not be used in Gimp (which is an 8-bit editor) since they were rounded at fist to 8-bit linear on opening and that made the image unusable due to posterization in the shadows. A gamma DCRAW makes DCRAW's output straight usable on Gimp without needing to pipe output through any gamma app.

The same applies to use of DCRAW's TIFFs in PS: they can now be assigned straight to PS work spaces. So many people have complaint about the 'underexposed' images from DCRAW just for not knowing they were linear.

I think that even if noise can be greater than digital quantization due to not applying the gamma, to apply it will fill nicely all levels and if there is any precision or information to be preserved, it will be. We are enriching the histogram the same way as when defocussing the image or adding noise, but without damaging the image at all.


Yes we are using Bruce's site formula.

[Guillermo Luijk]

Find here the gamma versión of DCRAW: DCRAW with Gamma output.

To make it work you will need to copy the included DLL file into the C:\Windows\system32\ or whatever path your DLL files are located.

The new command for gamma is -g. If used, beware that now output will be gamma corrected but metadata will stay saying the file is gamma 1.0, so when opening the resulting non-linear TIFF file in PS just ignore the embedded profile and directly assign (not convert) the profile chosen in the RAW development:

dcraw ... -o 1 -g srgb -> assign in PS to sRGB
dcraw ... -o 2 -g 2.2 -> assign in PS to Adobe RGB
dcraw ... -o 4 -g 1.8 -> assign in PS to ProPhoto RGB


Histogram with DCRAW's linear output and gamma conversion into PS:




Histogram straight from gamma DCRAW:

[sarangiman]

Hi everyone,

I also wondered if it were possible to get 16-bit gamma corrected output. Thanks for doing this. Though I'm not yet convinced it matters, b/c when I take gamma-uncorrected 16-bit files into Photoshop and then apply appropriate camera ICC profiles that themselves do the gamma correction (ICC profiles can do this, yes?), I don't see any holes in the histogram in Photoshop (I do see holes in the histogram when 'previewing' what the image will look like after application of said profile, but not after the actual application & then conversion to a standard color space is performed).

At any rate, though, I would find this useful. I don't know why Dave Coffin doesn't just implement it into dcraw... doesn't seem like the option could hurt!

I don't know much C, but when I try to compile dcraw.c with your added line of code, I get the error that 'user_gamma' is undeclared. I must be missing something... what? Or, could you just provide the dcraw.c source with your modification? I'm compiling on Mac OS X... hence the dcraw.exe file doesn't help me.

Many thanks in advance,
Rishi

[Guillermo Luijk]

At any rate, though, I would find this useful. I don't know what Dave Coffin doesn't just implement it into dcraw... doesn't seem like the option could hurt!

Last versions of DCRAW by David Coffin already implement output gamma in 16-bit, with the -6 and -g options. Now -4 is a particular case of -6 to keep compatibility. The histograms are plenty of holes though, the gamma doesn't seem to be applied in floating point to the output.

From DCRAW manpage:

-g power toe_slope
Set the gamma curve, by default BT.709 (-g 2.222 4.5). If you prefer sRGB gamma, use -g 2.4 12.92. For a simple power curve, set the toe slope to zero.

-6
Write sixteen bits per sample instead of eight.

-4
Linear 16-bit, same as -6 -W -g 1 1.

Regards.