Pages: [1] 2   Go Down

Author Topic: Saturation level in DCRAW  (Read 49719 times)

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« on: February 23, 2008, 03:47:10 pm »

Some time ago I reported David Coffin I was experiencing problems when developing RAW files I borrowed from a 40D user: there was a magenta cast in the highlights. He told me this was surely because of the saturation point (level at which RGB channels clip in the RAW file) of that 40D was lower than the default value DCRAW used, and provided a somewhat uncomfortable solution by introducing a new scaling factor in DCRAW's source code and compile again.

Luckily that is not necessary anymore since David has included the -S option in DCRAW v8.82 that allows to arbitrarily set the saturation point of your particular camera.

I will show an example: by developing a 40D RAW file with standard parameters:

C:\>dcraw -v -w -H 2 -T -4 test40d.cr2
Loading Canon EOS 40D image from test40d.cr2 ...
Scaling with darkness 1024, saturation 16224

this was achieved:



there is a strong magenta cast in the blown areas because of channel  wrong alignment, even if we told DCRAW we wanted neutral (R=G=B ) highlights with -H 2. The reason is DCRAW used level 16224 (14-bit range) as the saturation point while this unit has a lower clipping point.

Let's find out our unit's saturation level by extracting the RAW file with no demosaicing or scaling at all:

C:\>dcraw -v -D -T -4 test40d.cr2



The saturation point of this 40D at ISO100 is 13283<16224. So now we develop again making use of the -S option:

C:\>dcraw -v -w -H 2 -S 13823 -T -4 test40d.cr2

and the result is neutral highlights with no information loss at all:




The reason why the histogram does not reach the right end is because we used white balance multipliers less or equal to 1.0 which guarantees not to loose any information. The -H 2 option told DCRAW to ensure neutral highlights which was done correctly this time thanks to having let DCRAW know the right saturation point.

Hope you find it interesting.


QUESTION: my 350D clips at 4095, maximum level in the 12-bit scale. I have a feeling the reason for this is not that the 350D is such a perfect camera that matches the saturation level of the sensor to the maximum input level of the ADC. Instead I have a feeling the the ADC actually saturates before the analogue signal fed into it reaches its maximum output voltage.

This means no magenta cast issues, but some loss of information in the RAW with respect to the real sensor output. Can anyone confirm this?


Regards.
« Last Edit: February 24, 2008, 07:59:28 am by GLuijk »
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Saturation level in DCRAW
« Reply #1 on: February 23, 2008, 06:14:10 pm »

Quote
QUESTION: my 350D clips at 4095, maximum level in the 12-bit scale. I have a feeling the reason for this is not that the 350D is such a perfect camera that matches the saturation level of the sensor to the maximum input level of the ADC. Instead I have a feeling the the ADC actually saturates before the analogue signal fed into it reaches its maximum output voltage.
[a href=\"index.php?act=findpost&pid=176927\"][{POST_SNAPBACK}][/a]

There's no way to tell where actual saturation would be, if you've clipped before the signal runs into non-linearity.  That's just a black box you'll never see, unless you find a way to get into the camera and measure or attenuate the signal before it hits the ADC.

If the sensor starts to get non-linear before RAW clipping, then you're going to run into color casts unless the converter is calibrated for the particular camera, and the gain is consistent over time.  The Canon 10D goes non-linear below RAW saturation at ISO 100.  It also has color casts in the extreme highlights at ISO 100, because converters don't account for it.  Speaking of the 10D, it does something that the 40D also does.  The 10D RAW clipping point at low ISOs varies with the vertical column of pixels.  This means that the new "-S" switch should be used in DCRAW to clip at the lowest clipping point of any line.  The 40D gets a little more complex, with clipping offsets superimposed both horizontally and negatively.  If you want to use the correct clipping point for the 40D, it would be wise to take a shot where every pixel clips, and then see the histogram, and draw your clipping point at the left edge of it (except for dead pixels, which you should ignore).
« Last Edit: February 23, 2008, 10:52:13 pm by John Sheehy »
Logged

bernie west

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 323
    • Wild Photo Australia
Saturation level in DCRAW
« Reply #2 on: February 23, 2008, 08:08:50 pm »

Guillermo, how do we find out the saturation points for the different cameras?  Is dcraw spot on for all the other canons?
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #3 on: February 24, 2008, 06:39:04 am »

Quote
it would be wise to take a shot where every pixel clips, and then see the histogram, and draw your clipping point at the left edge of it (except for dead pixels, which you should ignore)

Very interesting John. However, looking at the RAW histogram I plotted before, don't you think we can suspect that all pixels clip exactly at the same point in the 14-bit range? nevertheless pixel columns or other considerations. I said 13823, but there were also pixels (or channels) clipping at 13824; so I took 13823 being conservative in the same way as you suggest.

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #4 on: February 24, 2008, 06:45:07 am »

Quote
Guillermo, how do we find out the saturation points for the different cameras?  Is dcraw spot on for all the other canons?
[{POST_SNAPBACK}][/a]

DCRAW has a standard table of clipping points for every camera model (I guess), but Coffin could have made a mistake or taken some too high or too low values, that's why finding out where your camera clips could be interesting.

- To find out which saturation level is DCRAW using for your particular camera at a given ISO, just take a shot making sure you have blown pixels in the 3 channels in a wide enough area. Developd the RAW file using something like:
C:\>dcraw -v -w -H 2 -T -4 test40d.cr2
Loading Canon EOS 40D image from test40d.cr2 ...
Scaling with darkness 1024, saturation 16224

The figure after 'saturation' is the value DCRAW is using. For my 350D it provides 4095 for instance.


- To find out where your camera's real saturation point is at a given ISO, just develop the same RAW file with no demosaicing nor scaling:
C:\>dcraw -v -D -T -4 test40d.cr2

and find the value at which the blown highlights peak appears (to do this you can use [a href=\"http://www.guillermoluijk.com/tutorial/histogrammar/index_en.htm]Histogrammar[/url] or even PS in 16-bit info mode): it was 13823 in the 40D under test.



Regards.
« Last Edit: February 24, 2008, 07:32:07 am by GLuijk »
Logged

bernie west

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 323
    • Wild Photo Australia
Saturation level in DCRAW
« Reply #5 on: February 24, 2008, 07:04:53 am »

Quote
- To find out which saturation level is DCRAW using for your particular camera at a given ISO, just take a shot making sure you have blown pixels in the 3 channels in a wide enough area. Developd the RAW file using something like:
C:\>dcraw -v -w -H 2 -T -4 test40d.cr2
Loading Canon EOS 40D image from test40d.cr2 ...
Scaling with darkness 1024, saturation 16224

[a href=\"index.php?act=findpost&pid=177042\"][{POST_SNAPBACK}][/a]

Thanks Guillermo.  My version of dcraw doesn't show the saturation, so I guess it is time for an upgrade.

I'm interested to know what is happening after black and saturation clipping is performed?  Is the histogram stretched to fill the full 16-bit range, or will there be empty levels at each end of the histogram?
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #6 on: February 24, 2008, 07:26:28 am »

Quote
Thanks Guillermo.  My version of dcraw doesn't show the saturation, so I guess it is time for an upgrade.

I'm interested to know what is happening after black and saturation clipping is performed?  Is the histogram stretched to fill the full 16-bit range, or will there be empty levels at each end of the histogram?
Yes, before the demosaicing process takes place all levels are scaled to fit the 16-bit range. Taking the 40D example which had black and saturation levels 1024 and 16224 respectively:

Scaling with darkness 1024, saturation 16224

the entire undemosaiced RAW data is first linearly scaled to fit:
1024 -> 0
16224 -> 65535

and then white balance, demosaicing, highlight recovery,... is applied. So the interpolated levels will completely fill the empty gaps between captured levels.

Note that using less or equal to 1.0 WB multipliers (-H 2-9 in DCRAW) can lead to some little empty gap in the right end of the histogram as I showed in the 40D sample. This is normal and desirable, since it is the behaviour of the -H 2 feature that ensures white highlights but no loss of information at all. All RAW developers perform this kind of non-linear operation in the highlights to preserve them neutral after WB is applied.

The -S feature is new in DCRAW v8.82, just download it from Pukkita's, he automatically maintains updated builds of DCRAW as soon as Coffin makes the source code available.

Just curiorus: other camera vendors (like Nikon) don't provide a black offset level in their RAW files so the user gets RAW data which has already been clipped in the shadows. For these RAW files DCRAW will always provide a 0 black level.
This was clearly explained by Panopeeper in some thread and is a good point for Canon RAW files to get the best of the captured untouched information. Most of you can find below the black offset is just noise, but it is good to let the user have the possibility to choose what to do with it.
« Last Edit: February 24, 2008, 08:04:06 am by GLuijk »
Logged

papa v2.0

  • Full Member
  • ***
  • Offline Offline
  • Posts: 206
Saturation level in DCRAW
« Reply #7 on: February 24, 2008, 09:35:56 am »

hi

just a couple of questions

when i do

C:\>dcraw -v -D -T -4 myfilename.nef


i get a undemosaiced linear 16bit tiff very dark as expected. found the saturation point as suggested and entered into dc raw using (-S 3855 for my D70s) (is there a histogrammer for mac ?)

dcrawG4 -v -w -H 2 -S 3855 -T -4  myfilename.nef    as suggested but get a sRGB file out

is this correct?

this is the message that DCRAW returns

Scaling with darkness 0, saturation 3855, and
multipliers 1.000000 0.506931 0.881188 0.506931
AHD interpolation...
Blending highlights...
Converting to sRGB colorspace...


And to note as you explained Nikon have allready subtracted the dark frame from thier RAW file.
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #8 on: February 24, 2008, 09:58:50 am »

Quote
is there a histogrammer for mac ?
Nope, I don't have a Mac.


Quote
dcrawG4 -v -w -H 2 -S 3855 -T -4  myfilename.nef    as suggested but get a sRGB file out
is this correct?
Yes, sRGB is default output colour space in DCRAW. To change it just add: -o X
where X is 0=none, 1=sRGB, 2=AdobeRGB, 3=WideGamut, 4=ProPhoto, 5=XYZ


Quote
this is the message that DCRAW returns

Scaling with darkness 0, saturation 3855, and
multipliers 1.000000 0.506931 0.881188 0.506931
AHD interpolation...
Blending highlights...
Converting to sRGB colorspace...
And to note as you explained Nikon have allready subtracted the dark frame from thier RAW file.
I didn't say Nikon already substracted the dark frame, but the black offset level, a totally different thing. If you see the "darkness 0" message, it means your Nikon RAW data started from level 0 while the Canon's start in some higher level (128, 255,1024,...). That is what I meant with 'black offset', a black value that is added to the real pixel level and has to be substracted from it in the RAW file.
« Last Edit: February 24, 2008, 10:01:19 am by GLuijk »
Logged

papa v2.0

  • Full Member
  • ***
  • Offline Offline
  • Posts: 206
Saturation level in DCRAW
« Reply #9 on: February 24, 2008, 10:09:00 am »

Sorry my mistake

So why would cannon apply an offset and nikon dont?
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Saturation level in DCRAW
« Reply #10 on: February 24, 2008, 10:58:55 am »

Quote
Sorry my mistake

So why would cannon apply an offset and nikon dont?
[a href=\"index.php?act=findpost&pid=177066\"][{POST_SNAPBACK}][/a]

I would think the offset is applied in all cameras, electrically.  Canon just keeps it in the RAW file.

Leaving the bias in allows more accurate mean levels in signals near black (not that any given converter is bound to take advantage of this - they seem mostly to clip the data themselves at black before doing anything, and at least one RAW developer on another forum recently stated that this is what his products do.  The noise below the true black level actually contributes slightly to the S/N ratio if used in the conversion process, as weird as that may at first seem.  I always leave Canon's sub-black noise in for my manual RAW conversions, by subtracting the offset in a program like Iris that has negative image values.  You can do resampling, binning, white balance, demosaicing, and all with the negative noise still there, and present in the calculations.  Rendered blacks come out closer to true black, with slightly less chromatic noise when the negative noise is left in.

For example, if you clip at black first, and then WB for tungsten light, the blue channel may be scaled by 4x.  This means that all the noise in the blue channel is scaled 4x, in a positive direction, so the demosaicing pulls the blue levels up near black.  If you leave the negative noise in, however, the negative noise scales further below zero, and the mean color near black is closer to black, with 4x blue noise pulling in both directions away from zero.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Saturation level in DCRAW
« Reply #11 on: February 24, 2008, 11:54:03 am »

Quote
This was clearly explained by Panopeeper in some thread and is a good point for Canon RAW files to get the best of the captured untouched information. Most of you can find below the black offset is just noise, but it is good to let the user have the possibility to choose what to do with it.
[a href=\"index.php?act=findpost&pid=177048\"][{POST_SNAPBACK}][/a]

At least one RAW developer I know of (who probably knows a lot about what is going on in the commercial industry) recently asked why bother having this data, so I suspect that the state of commercial RAW development is such that the first thing most programs do is clip black themselves.  If the program is then going to do something like algorithmic removal of line noise, however, it has less info to work with.

Perhaps only the astrophotography programs use sub-black noise (which of course, is partly signal, too, if it isn't a blackframe).

I really don't know why it would be anyone's first instinct to clip at black; demosaicing, wb, resampling, etc, should all be best done before blackpoint clipping the image.  Almost everything before the final application of transfer curve to RGB and color space can be done with negative values.  The benefits are small, but they are there.
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #12 on: February 24, 2008, 12:04:25 pm »

Quote
I really don't know why it would be anyone's first instinct to clip at black; demosaicing, wb, resampling, etc, should all be best done before blackpoint clipping the image.  Almost everything before the final application of transfer curve to RGB and color space can be done with negative values.  The benefits are small, but they are there.

For me the reason is the ease of operation, speed and needed memory resources. Operating in 16-bit positive integer values in the range 0..65535 is less complicated and demanding than floating point with negative values.

I let Coffin know about the possibility of a floating point version of DCRAW, but he insisted that there would be no real advantage in the end. He even doesn't consider the possibility of applying the gamma curve before the final 16-bit rounding. And I don't think he says that just because he is lazy to translate his code into floating point; he is pragmatic and thinks 99% of applications are well served with 16-bit unsigned int operation.
« Last Edit: February 24, 2008, 12:05:08 pm by GLuijk »
Logged

Prof. Jewell

  • Newbie
  • *
  • Offline Offline
  • Posts: 3
Saturation level in DCRAW
« Reply #13 on: August 12, 2009, 03:00:41 pm »

Quote from: GLuijk
For me the reason is the ease of operation, speed and needed memory resources. Operating in 16-bit positive integer values in the range 0..65535 is less complicated and demanding than floating point with negative values.

I let Coffin know about the possibility of a floating point version of DCRAW, but he insisted that there would be no real advantage in the end. He even doesn't consider the possibility of applying the gamma curve before the final 16-bit rounding. And I don't think he says that just because he is lazy to translate his code into floating point; he is pragmatic and thinks 99% of applications are well served with 16-bit unsigned int operation.


There is another strong reason for '32 bit' floating output from dcraw:

        Photoshop rounds all '16 bit' input down to '14bits' (in order to speed up processing). http://luminous-landscape.com/forum/style_...ault/unsure.gif

With the much more aggressive post processing being used today coupled with popularity of wide color spaces such as ProPhoto 14 bit is near--or at--the end of its useful life. If dcraw is to retain its cutting edge, now is the time to add a '32 bit' switch. [NB it' already there in SAR.]
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #14 on: August 12, 2009, 03:27:30 pm »

Quote from: Prof. Jewell
Photoshop rounds all '16 bit' input down to '14bits'[/i] (in order to speed up processing)

(...)

If dcraw is to retain its cutting edge, now is the time to add a '32 bit' switch.
It's 15 bit actualy. PS encodes even values ranging 0..32768 and odd values from 32769..65535

A 32-bit switch would mean rewriting a good amount of DCRAW's code, pretty sure Coffin doesn't agree.

BR

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Saturation level in DCRAW
« Reply #15 on: February 09, 2010, 05:32:30 pm »

Quote from: Prof. Jewell
[NB it' already there in SAR.]

Please what is SAR in this context?

Samotano

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 88
    • http://flickr.com/photos/97538742@N00/sets/72157603392490862/
Saturation level in DCRAW
« Reply #16 on: February 09, 2010, 08:27:55 pm »

Quote from: Hening Bettermann
Please what is SAR in this context?
I believe he was referring to sar image processor.  It has a number of quite novel and/or intensive algorithms
Logged

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Saturation level in DCRAW
« Reply #17 on: February 09, 2010, 10:32:31 pm »

Thank you, Samotano!

jbrembat

  • Full Member
  • ***
  • Offline Offline
  • Posts: 177
Saturation level in DCRAW
« Reply #18 on: February 10, 2010, 05:21:06 am »

Guillermo,
do you know  XYZ values  scaling for 16 bit dcraw?

Jacopo
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Saturation level in DCRAW
« Reply #19 on: February 11, 2010, 04:08:15 pm »

what do you mean by XYZ scaling? XYZ is an intermediate colour space to which images are converted prior to final conversion to sRGB, Adobe RGB. In fact I think DCRAw multiplies the RGB -> XYZ and XYZ -> output colour profile matrices to do the conversion in a single step.

The only scaling applied is when white balancing, and that is donde in the RGB Bayer domain, even before demosaicing.
Pages: [1] 2   Go Up