Pages: 1 [2] 3   Go Down

Author Topic: Noise About Noise  (Read 23594 times)

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #20 on: June 05, 2008, 12:33:30 pm »

Quote
With ACR at default settings, the above image appeared slightly overexposed according to the ACR histogram, due in large part to the +0.5 EV baseline exposure compensation that ACR uses for the D3. This demonstrates the importance of checking the raw histogram, which is easily done with Rawanalyze. If one wants the camera RGB histogram to give a better picture of the status of the RGB channels, one can upload to the camera a white balance where the WB multipliers are all unity (1.0). Search UniWB for details.
[a href=\"index.php?act=findpost&pid=199887\"][{POST_SNAPBACK}][/a]

Bill, as usual an informative post - but I have a question re the para I retained above: When you say there is a +0.5 exposure compensation which ACR uses for the D3, is that "under the hood" or is it a function of the slider and tone curve settings which come packaged as "ACR defaults" for that camera?
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Noise About Noise
« Reply #21 on: June 05, 2008, 01:27:48 pm »

Quote
Bill, as usual an informative post - but I have a question re the para I retained above: When you say there is a +0.5 exposure compensation which ACR uses for the D3, is that "under the hood" or is it a function of the slider and tone curve settings which come packaged as "ACR defaults" for that camera?
[a href=\"index.php?act=findpost&pid=199897\"][{POST_SNAPBACK}][/a]

Mark,

The ExposureOffset is not well documented, but I understand that it is packaged under the "ACR defaults" for that camera, and ACR applies +0.5 EV of compensation to all D3 NEFs. If you convert the NEF to DNG, you can use an EXIF reader and read the ExposureOffset that the DNG converter writes to the ExposureOffset tag of the metadata. I have not figured out how to do that with my available software, but perhaps someone can tell us how it is done. You could then find out the offset for your 1DsMIII. From what you have reported, I suspect that it is closer to zero than +0.5.


Bill
Logged

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #22 on: June 05, 2008, 02:26:51 pm »

Yup, for the Canons that would be my suspicion too. Still, as you say, it would be interesting to get some hard info on how it's done. From what you are saying, it seems this is done "under the hood" - not by slider defaults. Even if so, and if one finds this for some reason (say related to personal preferences or imaging environment) systematically bothersome, one can always set a counteracting Exposure reduction as a personal ACR preset for the camera. But before going there, undoubtedly a lot of research went into those ACR defaults for each supported camera, so what's more interesting is to know how/why the ACR team would have arrived at the judgment to set the exposure default the way they may have done for the D3.
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #23 on: June 05, 2008, 05:10:44 pm »

Some notes to issues touched above.

1. Careful with interpreting the in-camera histogram. As long as it is not purely raw-based, the histogram even with neutralised WB and other settings will not even resemble the raw histogram for following reasons:

a. de-mosaicing
b. color space conversion from camera to sRGB or whatever

These steps mix the raw channels into the RGB channels.

c. non-linear encoding ("gamma"). This step causes the distribution to be "compressed".

The effects of a. and b. are not very bad, except for very special illumination.

The effect of c. is very large; see the different histograms below from three images with different exposures: one is the raw, then the linear luminance displayed by Canon's DPP, and the RGB, again by DPP.

Consequently, the amount of clipping or close to clipping can not be judged from the in-camera histogram, but the indication, that probably clipping occured is reliable - and that's the point. If the camera displays clipping on the LCD (I guess this is the norm these days), that is more reliable.

I have been using the neutral setting for months, and I am absolutely satisfied; the clipping indication (blinking on the LCD) matches exactly with that based on the raw analysis (as far as it is discernable on the LCD). However, apparent "underexposure" (from the perfect ETTR) is much more, than the histogram indicates.

2. The automatic exposure adjustment is recorded in the tag BaselineExposure. It can be seen only in DNG format. The adjustment is not visible on the slider.

3. Images from the Canon 1DsMkIII will be automatic adjusted by +0.35 EV.

Note, that not only the exposure, but noise reduction and sharpening too are recorded in DNG format (BaselineNoise and BaselineSharpness). I really do not know, how far these affect the ACR processing, but I guess they do. In other words, the noise reduction is unavoidable. Furthermore, I don't know, if this noise reduction and the one specified are additive. Plus, this specification does not distinguish between luminance and chrominance NR.

4. The "recovery" slider appears to have nothing to do with true recovery of clipping; that is apparently automatic. It works well on uniform or quasy-uniform areas, for example on skies, but not on very fine details - this is understandable.

5. *All* TIFF tags can be extracted by http://www.cryptobola.com/PhotoBola/free/RawnalyzeTL.exe
This program creates a text file named "original file name.taglist.txt" in the same folder as the input was. It can be used on any TIFF. When started in Windows, it will stop running after having created the text file. It can be started from command line as well (one can specify a function for example "Taglist" in Windows Explorer, and create the tag list by right-clicking on the file name). If someone wants to do that, I post the details.

[attachment=6917:attachment][attachment=6918:attachment][attachment=6919:attachm
ent][attachment=6920:attachment][attachment=6921:attachment][attachment=6922:atta
chment][attachment=6923:attachment][attachment=6924:attachment][attachment=6928:a
ttachment]
« Last Edit: June 06, 2008, 05:10:41 pm by Panopeeper »
Logged
Gabor

bernie west

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 323
    • Wild Photo Australia
Noise About Noise
« Reply #24 on: June 05, 2008, 06:40:51 pm »

Quote
Bernie raises a good point about in camera histograms. They are said without much proof to represent luminance histograms where the RGB channels are weighted. Sean McHugh has a good tutorial in his Cambridge in Color site, where he discusses luminance and RGB histograms. The RGB weighting factors he gives are 59%, 30%, 11% for GRB respectively. Since the factor for blue is only 11%, luminance histograms are not sensitive for detecting clipped blues.

[a href=\"index.php?act=findpost&pid=199887\"][{POST_SNAPBACK}][/a]
This is what I have found with skies in particular.  No clipping on the camera histogram, but when analysing the raw data, blue has clipped.
Logged

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #25 on: June 05, 2008, 07:29:39 pm »

Quote
This is what I have found with skies in particular.  No clipping on the camera histogram, but when analysing the raw data, blue has clipped.
[a href=\"index.php?act=findpost&pid=199956\"][{POST_SNAPBACK}][/a]

Yes. and this is precisely what happened with our Nikon D3 ETTR exposures; it happened to the Canon 1Ds3 exposures only when I implemented "ETFR" (Expose to the Further Right) - which was a bit more EC. as explained in the article But in this case, the sky clipping did show on my Canon LCD as well as in Camera Raw.
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #26 on: June 05, 2008, 07:33:42 pm »

Quote
Some notes to issues touched above.

I have been using the neutral setting for months, and I am absolutely satisfied; the clipping indication (blinking on the LCD) matches exactly with that based on the raw analysis (as far as it is discernable on the LCD). However, apparent "underexposure" (from the perfect ETTR) is much more, than the histogram indicates.

[a href=\"index.php?act=findpost&pid=199946\"][{POST_SNAPBACK}][/a]

Hi Pano - very interesting post and thanks for contributing. I'm particularly interested in your comment that I retained above. When you speak of "neutral settings" could you please clarify where and what these are? Are you talking about the JPEG settings for the in-camera histogram, or something for Camera Raw, (or both) and for whichever, what specifically are you using that you would define as "neutral"?
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #27 on: June 05, 2008, 07:36:33 pm »

Weighting of the RGB channels is part of the JPEG encoding. Most JPEG images are JFIF compliant; that means among others, that the data is stored in YCbCr format, not in RGB. The calculation of Y depends on the applied standard; according to CCIR 601,

Y = 0.299R + 0.587G + 0.114B
Logged
Gabor

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #28 on: June 05, 2008, 07:44:19 pm »

Looking at your images in the above post, am I correct to assume that by "neutral" you mean the illustrated "neutral" settings in Rawanalyze?
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #29 on: June 05, 2008, 07:44:33 pm »

Quote
When you speak of "neutral settings" could you please clarify where and what these are? Are you talking about the JPEG settings for the in-camera histogram
I meant only the sharpness, contrast, saturation, tone, and, of course the white balance.

Quote
this is precisely what happened with our Nikon D3 ETTR exposures
I don't understand this. The problem Bernie mentioned is characteristic to luminance histograms, but the D3 has color histogram, and that should not be much different from the raw histograms with a neutral setting.

Do you mind posting such a raw file?
Logged
Gabor

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Noise About Noise
« Reply #30 on: June 05, 2008, 10:19:26 pm »

Quote
Consequently, the amount of clipping or close to clipping can not be judged from the in-camera histogram, but the indication, that probably clipping occured is reliable - and that's the point. If the camera displays clipping on the LCD (I guess this is the norm these days), that is more reliable.

I have been using the neutral setting for months, and I am absolutely satisfied; the clipping indication (blinking on the LCD) matches exactly with that based on the raw analysis (as far as it is discernable on the LCD). However, apparent "underexposure" (from the perfect ETTR) is much more, than the histogram indicates.
[{POST_SNAPBACK}][/a]

With my Nikon D3 with neutral settings, I have been quite satisfied with the histogram to indicate ETTR. The accuracy of the camera histogram for ETTR reportedly varies among cameras. The blinking highlights also work well on my D3.

Quote
2. The automatic exposure adjustment is recorded in the tag BaselineExposure. It can be seen only in DNG format. The adjustment is not visible on the slider.

3. Images from the Canon 1DsMkIII will be automatic adjusted by +0.35 EV.
[a href=\"index.php?act=findpost&pid=199946\"][{POST_SNAPBACK}][/a]

From the above, I infer that the BaselineExposure for the 1DsMkIII is +0.35 EV as compared to +0.5 for the D3.

Quote
5. *All* TIFF tags can be extracted by [a href=\"http://www.cryptobola.com/PhotoBola/free/RawnalyzeTL.exe]http://www.cryptobola.com/PhotoBola/free/RawnalyzeTL.exe[/url]
This program creates a text file named "original file name.taglist.txt" in the same folder as the input was. It can be used on any TIFF. When started in Windows, it will stop running after having created the text file. It can be started from command line as well (one can specify a function for example "Taglist" in Windows Explorer, and create the tag list by right-clicking on the file name). If someone wants to do that, I post the details.
[a href=\"index.php?act=findpost&pid=199946\"][{POST_SNAPBACK}][/a]

I downloaded and used the Rawnalyze program and obtained the following data from the DNG. The BaselineExposure is shown as negative, whereas I expected it to be positive. Thanks very much for the information and program!

1000000/1000000        651399/1000000    
       50730 BaselineExposure            SRATIONAL        -50/100        
       50731 BaselineNoise                  RATIONAL          60/100        
       50732 BaselineSharpness           RATIONAL          100/100        
       50734 LinearResponseLimit         RATIONAL          100/100        
       50736 LensInfo                          RATIONAL          1050/10
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #31 on: June 05, 2008, 10:57:13 pm »

Hold on Bill, something is fishy!

I verified in a dozen D3 images with different ISOs, and the BaselineExposure is everywhere PLUS 0.5.

You must have applied some in-camera setting, which caused the DNG converter to act differently. It is important to figure out, what that was.

First I thought of Active Daylight; however, I have a D300 image with ADL High (allegedly), and there the exposure adjustment is not different from Off.
« Last Edit: June 06, 2008, 03:10:09 am by Panopeeper »
Logged
Gabor

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #32 on: June 06, 2008, 09:33:13 am »

Quote
I meant only the sharpness, contrast, saturation, tone, and, of course the white balance.
I don't understand this. The problem Bernie mentioned is characteristic to luminance histograms, but the D3 has color histogram, and that should not be much different from the raw histograms with a neutral setting.

Do you mind posting such a raw file?
[a href=\"index.php?act=findpost&pid=199973\"][{POST_SNAPBACK}][/a]

Pano - I don't mind, but I'll have to get back to you on that.
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Noise About Noise
« Reply #33 on: June 06, 2008, 04:40:01 pm »

Quote
Hold on Bill, something is fishy!

I verified in a dozen D3 images with different ISOs, and the BaselineExposure is everywhere PLUS 0.5.

You must have applied some in-camera setting, which caused the DNG converter to act differently. It is important to figure out, what that was.

First I thought of Active Daylight; however, I have a D300 image with ADL High (allegedly), and there the exposure adjustment is not different from Off.
[{POST_SNAPBACK}][/a]

Gabor,

I have 20 NEFs from the D3 taken as part of an Experiment with a Stouffer wedge with manual exposure and the same camera settings. Some have a baseline exposure tag as shown by your program of +0.5 EV as expected, but a large number towards the end of the shoot have a baseline exposure of -0.5 EV. The camera settings were not changed except for shutter speed. The preview in ACR does not show an unexpected darkening.

I deleted the original DNGs and reconverted the NEFs to DNG with the same results. Here is a link to one NEF that gives the wrong baselineExposure.

[a href=\"https://download.yousendit.com/50DB929F6D40432A]Link[/url]

Bill
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #34 on: June 06, 2008, 05:04:07 pm »

Bill, please post one from the very same serie, which gets +0.5 EV.

Thanks
Gabor
Logged
Gabor

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Noise About Noise
« Reply #35 on: June 06, 2008, 05:57:19 pm »

Quote
Bill, please post one from the very same serie, which gets +0.5 EV.

Thanks
Gabor
[{POST_SNAPBACK}][/a]

Gabor,

This file has +0.5 EV  baselineExposure

[a href=\"https://download.yousendit.com/CA67B86200F440CB]Positive[/url]

And the very next in the series has a negative value:

Negative

Bill
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #36 on: June 06, 2008, 06:55:14 pm »

Bill,

ISO 100 is the key. I do not have any other ISO 100 shot. I verified shots wityh perhaps all other ISOs, and all get +0.5.

I suggest you to make two shots of the same scenery (perhaps the Stouffer), with the same exposure, but one with ISO 100, the other with ISO 200, and compare their histograms. There should be some clipping as well.


Gabor
Logged
Gabor

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #37 on: June 06, 2008, 07:09:43 pm »

Quote
I meant only the sharpness, contrast, saturation, tone, and, of course the white balance.
I don't understand this. The problem Bernie mentioned is characteristic to luminance histograms, but the D3 has color histogram, and that should not be much different from the raw histograms with a neutral setting.

Do you mind posting such a raw file?
[a href=\"index.php?act=findpost&pid=199973\"][{POST_SNAPBACK}][/a]

Gabor,

OK, I have permission to share a Nikon D3 file. You can let me know which you want. These files are about 12MB each. I would have to up-load it to a private website address from which you could download it. If you send me a private email we'll set this up offf-line, then you can analyse and bring the discussion back on-line.

Mark
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Noise About Noise
« Reply #38 on: June 06, 2008, 07:20:25 pm »

Quote
You can let me know which you want. These files are about 12MB each. I would have to up-load it to a private website address from which you could download it. If you send me a private email we'll set this up offf-line, then you can analyse and bring the discussion back on-line.
Mark,

the simplest way is to upload it/them to yousendit.com. You don't need to register, you don't even need to use your email address, nor mine, it can be a fictional one (though you can use my email address from the profile if you want to). In response you receive a URL for the uploading. You can post that, or if you don't want to "publish" them, then pls email me. You can use the forum's messaging, or my email address; look at my profile.

Regarding which one(s): you mentioned, that a blue cipping has not been indicated with the neutral profile. That one, which you deem the strongest case, i.e. you think the clipping was stong, but you don't see any clipping indication on the in-camera histogram.

I do not have Nikon Capture, so I don't see the original histogram, although if I extract the JPEG, then ACR should display the same one.


Thanks
Gabor
Logged
Gabor

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Noise About Noise
« Reply #39 on: June 06, 2008, 07:32:48 pm »

Quote
Mark,

the simplest way is to upload it/them to yousendit.com. You don't need to register, you don't even need to use your email address, nor mine, it can be a fictional one (though you can use my email address from the profile if you want to). In response you receive a URL for the uploading. You can post that, or if you don't want to "publish" them, then pls email me. You can use the forum's messaging, or my email address; look at my profile.

Regarding which one(s): you mentioned, that a blue cipping has not been indicated with the neutral profile. That one, which you deem the strongest case, i.e. you think the clipping was stong, but you don't see any clipping indication on the in-camera histogram.

I do not have Nikon Capture, so I don't see the original histogram, although if I extract the JPEG, then ACR should display the same one.
Thanks
Gabor
[a href=\"index.php?act=findpost&pid=200155\"][{POST_SNAPBACK}][/a]

Gabor,

I didn't use Nikon Capture either - only ACR.

On the issue of the blue clipping and the histogram - indeed we were looking at the Luminance histogram on the D3. The D3 does have the capability to show either the aggregate or the individual R,G,B tonal distributions.

Re your email address - if your ISP allows you to receive 12 MB attachments we're in business and that's by far the easiest thing to do, except your email address is not in your L-L profile. Please send me a P-M with your email address and I'll ship you the file.
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."
Pages: 1 [2] 3   Go Up