Blowing out whites too soon with Nikon D800E.

[Some Guy]

Playing with gray zones in a Nikon D800E w/ Nikon 24-70mm f/2.8.

Set to "Standard" for image settings.
ISO 100.
ExpoDisc on lens pointed at blue clear sunlight sky.  WB checked with Sekonic C-500 and WB manually set in camera to 8330K.

NEF files converted to TIFF in Nikon's current version of ViewNX-i (Using its "Export to Contact Sheet" in TIFF format.).  Straight conversion from NEF to TIFF.

Shutter speeds shown on images.
Aperture shown on images.

First black change shows in PS between frame DSC8439 and DSC8440.
White goes full RGB 255 values in frame DSC8449.

So from DSC8439 (Approx. Full black @ RGB 3 ea.) to DSC8449 (Full white @ RGB 255) is 11 stops coverage (1/4000 to 1/4 sec @ f/8.).  Does the same if I Export in ViewNX-i to a JPEG contact sheet too.

Question:
What puzzles me is why it shows little gray between DSC8448 and DSC8449, just drops or blows out to white quickly?  I would think there would be couple of tones in there, but it goes to white fast.

Any ideas?  Maybe "Portrait" mode instead of "Standard" would retain some highlight tones?

SG

[Redcrown]

Take your RGB values (just use the Green value):

6/8/10/14/30/56/94/135/177/223/255

Enter them in a spreadsheet and then plot a line graph. What you will see is a typical photo sensitive curve. Notice that it is not linear (straight line). It has a "toe" on the left and a "shoulder" on the right. It's the same curve us old folk used to analyze films. Different films and even different chemical processing of the same film would give different curves.

Now assume you plotted only through the 223 value. What would you predict for the next point on the curve?

That said, your method for measuring dynamic range is somewhat flawed because you are processing the raw. The processing steps and the viewing method (monitor profile) alter the "effective" dynamic range.

To know the true "scientific" dynamic range, go here:
http://www.dxomark.com/Reviews/Nikon-D800-Review/Nikon-D800-Sensor-performance

To get more detail out of your highlights, look to different raw processors and the different options within those processors. Things like camera profiles and highlight recovery can make a big difference.

[Some Guy]

Take your RGB values (just use the Green value):

6/8/10/14/30/56/94/135/177/223/255

Enter them in a spreadsheet and then plot a line graph. What you will see is a typical photo sensitive curve. Notice that it is not linear (straight line). It has a "toe" on the left and a "shoulder" on the right. It's the same curve us old folk used to analyze films. Different films and even different chemical processing of the same film would give different curves.

Now assume you plotted only through the 223 value. What would you predict for the next point on the curve?

That said, your method for measuring dynamic range is somewhat flawed because you are processing the raw. The processing steps and the viewing method (monitor profile) alter the "effective" dynamic range.

To know the true "scientific" dynamic range, go here:
http://www.dxomark.com/Reviews/Nikon-D800-Review/Nikon-D800-Sensor-performance

To get more detail out of your highlights, look to different raw processors and the different options within those processors. Things like camera profiles and highlight recovery can make a big difference.

Thanks.

I just plotted the RGB off the TIFF file and not the attached JPG.  What I got is in the screenshot below.

There really is no shoulder in the curve or very little, just a lot of gradual toe in the black values.  Might explain why the camera is prone to blowing out the red so easy too.  I would hoped to see some like a 240 value in the next to last 255 square rather than the steep jump from 224 to 255 in the Green channel.

There is some "Brightness" (-1, 0, +1)  in the camera menu that might shift the entire curve to the left a bit.  I don't know what effect "Contrast" has in the camera menu on the NEF curve over the Brightness one.  Same for "Portrait" over "Standard" although Nikon demos Portrait to keep from overexposing and washing out skin in sunlight.

I was hoping to use the old spotmeter that was setup for the Zone System and using one-stop increments, but once the Nikon gets to the brights it blows it off too easy.  Maybe 1/3 increments might keep detail there.  Zones 8-10 or 11 will get very tricky to manage via metering without finer delineation.

As to RAW converters, RawTherapee has some info on the D800E and the shoulder, but it looks like a lot of work to manage it in post using a spotmeter and one-stop for each Zone.  I was hoping Nikon ViewNX-i would do it, but seems it goes to the white too soon in conversion.  Link: http://www.russellcottrell.com/photo/BTDZS/D800Ecalibration.htm


Right now, I can see little texture in K7 prints unless I add a shoulder to the output curve.  Then that compresses the rest of the curve too so it gets complex without going into it with a decent image with a better shoulder and toe, or even linear.

SG

[Redcrown]

Remember that raw data is linear, but that human visual perception is not. Thus every raw converter applies a base tone curve to move the raw data closer to human perception. That base tone curve changes all pixel values, before you get a shot at them with your own adjustments.

I'm not familiar with Nikon raw converters, and only have a brief experience with RawTherapee. But in Adobe Camera Raw, each "profile" has a base tone curve, and they vary a lot. They vary between profiles for the same camera. They vary across camera models.

Long ago I discovered that the "Camera Faithful" profile for my Canon cameras preserved highlights better than the other profiles, including Adobe Standard. When I looked at their base tone curves I saw why.

Then, I was helping out a Nikon guy once and went to show him this difference on his raw files. To my surprise, the opposite was true for Nikon. The Nikon "camera standard" profiles for his camera created brighter images than the Adobe Standard profile. An inspection of the Nikon base tone curves showed why. Unlike Canon, the Adobe Nikon profiles had a stronger base tone curve which raised all the tones considerably more, including highlights.

The Adobe DNG Profile Editor is the only tool I'm aware of that will display base tone curves. And if you use the DNG Profile Editor to create a custom profile, you can make the base tone curve anything you want.

Take a look at the RawDigger program (www.rawdigger.com). It will let you see the true linear raw values for your exposures, and thus help explain what's happening to those values as they pass through your raw converter. With a little curves adjustment in your raw conversion I'd bet you can get more separation in the highlights.

I'd bet that RawDigger will show that the raw data for your "255" exposure is not really clipped, but the raw processor (and it's base tone curve) is forcing those values to 255.

[Guillermo Luijk]

You are not measuring D800's sensor dynamic range (that BTW is the largest in the market), but the JPEG tone curves Nikon decided to implement in the camera engine:

http://forum.luminous-landscape.com/index.php?topic=96758.0



If you are not satisfied with camera JPEGs just shoot RAW and apply your own processing.

Regards

[Some Guy]

Remember that raw data is linear, but that human visual perception is not. Thus every raw converter applies a base tone curve to move the raw data closer to human perception. That base tone curve changes all pixel values, before you get a shot at them with your own adjustments.

I'm not familiar with Nikon raw converters, and only have a brief experience with RawTherapee. But in Adobe Camera Raw, each "profile" has a base tone curve, and they vary a lot. They vary between profiles for the same camera. They vary across camera models.

Long ago I discovered that the "Camera Faithful" profile for my Canon cameras preserved highlights better than the other profiles, including Adobe Standard. When I looked at their base tone curves I saw why.

Then, I was helping out a Nikon guy once and went to show him this difference on his raw files. To my surprise, the opposite was true for Nikon. The Nikon "camera standard" profiles for his camera created brighter images than the Adobe Standard profile. An inspection of the Nikon base tone curves showed why. Unlike Canon, the Adobe Nikon profiles had a stronger base tone curve which raised all the tones considerably more, including highlights.

The Adobe DNG Profile Editor is the only tool I'm aware of that will display base tone curves. And if you use the DNG Profile Editor to create a custom profile, you can make the base tone curve anything you want.

Take a look at the RawDigger program (www.rawdigger.com). It will let you see the true linear raw values for your exposures, and thus help explain what's happening to those values as they pass through your raw converter. With a little curves adjustment in your raw conversion I'd bet you can get more separation in the highlights.

I'd bet that RawDigger will show that the raw data for your "255" exposure is not really clipped, but the raw processor (and it's base tone curve) is forcing those values to 255.

Regarding Adobe Camera Raw, then I'll assume it assigns a base to each camera make and model prior to it opening?  Correct?

This matter arose when trying to come up with some sort of Zone System which seems it cannot be done unless the RAW converter applies a Zone to each stop exposure that I meter with the spotmeter.  In other words, the old Ansel Adam's Zone System cannot be used with digital unless the RAW converter also uses one-stop along the sensor's range? (i.e. Zone VIII to IX was accepted as one stop per Adam's and film, but my digital Zone VIII to IX may be only 1/3 stop unless the RAW converter is more linear to his way of thinking.).

Seems like RawTherapee may be the only answer is it is highly tunable and maybe a base can be applied somehow from it, but don't know if it will do it for "digital" B&W Zone System using a spotmeter and subsequent pre-visualization.

I don't have a local bookstore that carries the few digital zone books out there either.  I suspect they may be in error if "One stop does not equal one Zone" in the highlights.  Actually, the old Pentax Spotmeter I have had a old stick-on Zone VI Studios scale that was one stop per Zone.  Possibly one could be made that would be finer for the highlight Zone ZII to X end (e.g. 1/3 stop)?

Fwiw, I did play around a bit with the RawDigger today with the 14 bit NEF files.  Sort of confused me when I saw the RGB values shoot up to 5,000+ until I got it in my head it was 14 bit files, and not what I normally see as 0-255 RGB in PS.

SG

[kirkt]

Here is a plot for my Canon 5DIII which I think is more what you are trying to make for your camera.  I shot a gray target through the entire range of exposures from Zones -IV to XI.  I metered the gray target and assigned that reading Zone V, and then adjusted the ISO and aperture so that I could span the entire range of zones with reasonable shutter speeds.  Each zone is 1 stop exposure difference, as you would expect.  The plot depicts the average values of a patch of the gray target, as sampled by Raw Digger, for the green channel (G1) versus Zone.  The camera sensor is pretty linear between Zones -II and IX.

The next question is, how much of this linear region is useful, in terms of what your display can display and your printer can print?  This has to be answered in the context of how these raw Zones get translated (compressed, expanded) into RGB zones with a particular raw converter and, possibly, after additional manual Zone compression in post-conversion editing.  Also attached is a plot of L* values across Zones for a raw conversion done in Raw Photo Processor (RPP) - I can dial in the highlight rolloff to create the shoulder I want, etc.  This is one way to convert the linear data to something useful.

But, at least in the sensor data, Zones raw levels are linearly related to exposure, like one would expect.  

kirk

[Some Guy]

Here is a plot for my Canon 5DIII which I think is more what you are trying to make for your camera.  I shot a gray target through the entire range of exposures from Zones -IV to XI.  I metered the gray target and assigned that reading Zone V, and then adjusted the ISO and aperture so that I could span the entire range of zones with reasonable shutter speeds.  Each zone is 1 stop exposure difference, as you would expect.  The plot depicts the average values of a patch of the gray target, as sampled by Raw Digger, for the green channel (G1) versus Zone.  The camera sensor is pretty linear between Zones -II and IX.

The next question is, how much of this linear region is useful, in terms of what your display can display and your printer can print?  This has to be answered in the context of how these raw Zones get translated (compressed, expanded) into RGB zones with a particular raw converter and, possibly, after additional manual Zone compression in post-conversion editing.  Also attached is a plot of L* values across Zones for a raw conversion done in Raw Photo Processor (RPP) - I can dial in the highlight rolloff to create the shoulder I want, etc.  This is one way to convert the linear data to something useful.

But, at least in the sensor data, Zones raw levels are linearly related to exposure, like one would expect.  

kirk

Interesting data off the Canon.

If I call frame DSC8447 above as Zone V gray (visually on screen), then I only have two stops left towards white:  DSC8448 and DSC8448 which is full RGB 255.  The monitor is calibrated and I can see from abut 4 to 255 on the Piezography test on how good is your monitor so not a monitor issue for now (Link: http://www.piezography.com/PiezoPress/blog/calibration/calibration-do-i-need-it/ ), more like a converter matter someplace or the camera settings.

I got a whole lot of black to play with though.  

Somehow, either in camera settings or the RAW converter, I need to push that curve to the left somehow and get a shoulder back in the highlights.  Hopefully without resorting to small exposure increments which doesn't play well with the spotmeter's Zone scale.

Today, since I've been messing with this, the camera seems to be getting better highlight details.  I've wondered if it being in the dark so long (Case and lens cap.) if it takes some light and exercise to get the sensor up to snuff for the highlights?  Doesn't seem to be an issue with the darks, just the lights right out of the case.

SG

Additional:

Tried setting the camera's menu to Standard and "Brightness +1" and seemed to move Zone V down a bit and gave me a bit more highlight coverage.  I'll attach the RAW to TIFF (JPG here) which seems to show the curve shifting to the left and opening up the shadows as well as giving me a bit of a shoulder as well in the highlights.

SG
 

[kirkt]

It sounds like you are acquiring raw image data and then converting it to an RGB image with Nikon's raw converter, possibly using some sort of picture style, or whatever Nikon calls their JPEG schemes.  

You will likely have much more control over the process if you use a raw converter in a way that does not apply a tone curve that is so drastically compressing your highlights.  I have found that, when I reflectance meter a "middle gray" target, the meter reading puts me in Zone IV instead of Zone V - the old 12.5% versus 18% gray issue I suppose.  My limited understanding is that the metering is set so that the sensor has a certain amount of headroom above the metered reading (3 stops, for example) for highlights so that highlight clipping does not occur too early.  Some raw converters apply a behind-the-scenes exposure "correction" to make up for this - RPP does not.  To make my raw data, upon conversion, such that the metered gray becomes L*50, I need to add about a stop of exposure increase - this will elevate the curve but at the expense of compressing the highlights.  At least I can select the rolloff for how this highlight compression occurs.

I am not sure I understand how you are determining what the reference is for your Zone scheme ("Zone V") - you stated that you determined it "visually" - how about what your meter reading tells you?  If, visually, the middle zone is one patch lower in gray intensity, then you have three highlight patches before clipping.  That seems sort of arbitrary.

In order for you to get the raw conversion you are looking for, you probably need to re-revaluate what your metered readings mean and then take more manual control over the raw conversion process, as Guillermo suggested above.  Look at the raw data in your file first (use Raw Digger to sample your gray patches) to see just exactly where clipping occurs in the raw data - then you can figure out what the raw conversion process is doing to that raw data and how you can alter the raw conversion process to suit your needs.  If you shoot in daylight, the green channel is a good proxy for the file because it usually is the first channel to reach sensor saturation in a daylit scene.  

Examining the raw data is also helpful when devising ETTR strategies to overexpose during capture and then pull back the exposure in post.  Once you get a better idea of where clipping really occurs in the raw file, you know that that point is where Zone IX, for example, occurs during capture.  If you choose to use highlight reconstruction in your raw conversion, you may be able to tweak your Zones during "development" and shoot with the idea that raw conversion will permit you to expand your highlight Zones a bit.  I do not use a Nikon, so there may be Nikon-specific issues that also contribute to this analysis or which I am not aware.

kirk

[Jack Hogan]

Some Guy,

So the question for most camera/raw converter designers/photographers is how do you show 11+ stops of linear information from the camera with only 8- stops available for display, pleasingly.

We have a lot of latitude while rendering an image (in-camera or in-converter) because our visual system works relatively, not absolutely.  By shifting linear raw data up or down (for instance Exposure/Brightness sliders in a raw converter) what we are really doing is deciding what portion of the 11+ stops are shown in the 8- stop output medium 'window'.  Are we ok with clipping some of those from the final displayed photograph?  Or would we like to bend the linear curve and try to squeeze in more information non-linearly towards the ends (for instance contrast/curve sliders in PP)?  The answer is always a compromise and almost never the same: it is subjective and scene dependent.

What cameras and converters give you is typically a first generic guess at what their designers/focus groups think is a 'pleasing' way to squeeze those 11+ stops into 8- in a typical given situation (Landscape, Portrait, etc).  The more sophisticated the hardware and software, the more sophisticated the guess.  They work well some of the time, not others.  Options increase the more linear stops of DR your camera produces.

By now you've figured out that there is the objective, quantitative, linear information captured in the raw data (shown by Kirk); and then there is the subjective, qualitative, non-linear processing that squeezes it into the output medium (shown by Guillermo).  To achieve that objective your job as a photographer is mainly A) to make sure that desirable highlights are just not clipped in the raw data; and then B) to process that raw data for a pleasing result.

When you say that you are blowing out whites too soon, are you doing A)?  And what is your question about B)?

Jack

[Some Guy]

Still trying to wrap my head around this B&W digital zone system.

With film, I remember the teacher's "Expose for the shadows, and develop for the highlights" mantra.  Digital seems to be the reverse:  Expose for the highlights and process (in some RAW converter) for the shadows.  Just where and how one gets the middle Zone V to fall between them is the tricky part and likely due to the RAW converter - so far.

I think I got it to work with Nikon's ViewNX-i as it batch processes the NEFs and also does a nice contact sheet as above.  Seems though I have to fiddle with the camera's internal menu a bit though with the Brightness setting (+1 so far).

RawTherapee 'may' be the better of the RAW converters out there (I was sort of surprised by the Sharpness I got out of it in my first learning attempts.), but seems you need a PhD to run it or spend a lot of time to learn it.  ViewNX-i is quick.  Didn't try ACR since it doesn't seem to work with camera settings a well as with the Nikon software, it just does its own curve thing which seems to punch blues up and a bit too much vibrancy and contrast too so maybe not a good idea if one wants to play with the Zone System in B&W.  I need to check deeper into Nikon's Auto Dynamic Range settings (By default) too as I fear that may be causing havoc being set at Auto.

I did try the Raw Digger and it seemed the Nikon D800E of mine would go from Underexposed Blue to Overexposed Red as such in the one-stop increments I shot at:  I began the count at Blue as one, and then stopped at full Red which occurred at 10.  So I will guess that is Zone 0 to Zone IX.

Subsequent "Zonal" prints yesterday seem to be pretty good off a 18% gray card with the one stop increments.  My new gray card seems a bit darker than what I recall it was visually too (So much for 'remembering' what it looked like since the ColorChecker Passport is what I often use and Zone V or 18% isn't clearly marked on it.).  At least enough to use a spotmeter and play Ansel Adams for a day.  I tossed a color scene into the mix at the end and that seemed a little flat maybe due to the Brightness +1 too, but that could be fixed in post.

I need to play around a bit more with RawTherapee too and see if I can tune this curve a bit for B&W.  I could alter the printer curve a bit in Qimage Ultimate, but would rather do it in camera first since if the highlight detail is gone, no sense trying to make a shoulder for the image in the printer to get it back.

SG

[TonyW]

I think the advice offered so far is pretty much spot on.  

IMHO it is quite possible to apply what you have learned relating to the Zone System and there is no reason why you cannot use your existing spot meter.  Just think of your camera sensor as a new type of film (with quite a bit of exposure/processing latitude  ) compared to the old analogue stuff.  

FWIW I have also recently acquired a D800E and hardly had a chance to play but some thoughts that may be helpful and hopefully not seen as trying to teach granny to suck eggs  

....With film, I remember the teacher's "Expose for the shadows, and develop for the highlights" mantra.  Digital seems to be the reverse:  Expose for the highlights and process (in some RAW converter) for the shadows.

Seems pretty much spot on and I suppose you could think of digital as transparency but capable of holding a much wider scene dynamic range  

Just where and how one gets the middle Zone V to fall between them is the tricky part and likely due to the RAW converter - so far.

Your Zone V value would be L*50 as shown in Lightroom or in RGB values would be around 118 in a 2.2 gamma such as Adobe RGB and approx RGB=100 in ProPhoto

I think I got it to work with Nikon's ViewNX-i as it batch processes the NEFs and also does a nice contact sheet as above.  Seems though I have to fiddle with the camera's internal menu a bit though with the Brightness setting (+1 so far).

I do not use ViewNX so difficult to comment although I would say that it is not the way you should be doing this.  Your NEF's are being converted on the fly and all sorts of things happening to arrive at the contact sheet.  I really believe that this is skewing your perception and leading you to make incorrect assumptions.

It is not clear to me what you are referring to with Brightness setting of +1.  AFAIK there is only one thing that adjusts the overall exposure setting and that is in the Custom Settings Menu item b Metering/exposure b6 Fine Tune optimal exposure.  Selecting this menu allows you to adjust the Matrix, Centre weighted and Spot metering to +/- 1.  Be careful with this as if you adjust you will not see any information that you have made such adjustments showing anywhere in the camera menu system (unlike normal exposure compensation)
The only thing I can think you may be changing is the Picture control settings but these do not affect the raw image (my assumption is you are wanting to use raw?) and will be ignored in any raw editor other than Nikon.  Adobe has its own Nikon presets which are pretty close to Nikon in camera.
I have not had time to properly investigate but it seems to me that my D800 is a little hyperactive with exposures tending to overexpose and certainly disagrees with my newly acquired spot meter still I need to test a little more to confirm and rule out clumsy user error  .  Anyway shooting a flatly lit grey card and bringing the NEF into Lightroom shows a value of 56.9% with Camera Neutral (Adobe Std a little higher) a little higher (1/3 stop?) than the magic 50% (if that zone placement is of particular importance!)  Now just trying Raw Therapee and confess not being used to navigating or even RTFM it seems to me that the default setting shows the L value of 50% and the custom setting with all zeroed an L value of 39.  See example attached

Didn't try ACR since it doesn't seem to work with camera settings a well as with the Nikon software, it just does its own curve thing which seems to punch blues up and a bit too much vibrancy and contrast too so maybe not a good idea if one wants to play with the Zone System in B&W.  I need to check deeper into Nikon's Auto Dynamic Range settings (By default) too as I fear that may be causing havoc being set at Auto.

I am not aware of any means of adjusting the cameras Dynamic range - am I missing something?

[Some Guy]

Tony, what I meant with the dynamic range is the Nikon Active D-Lighting (ADL) in the camera's menu - sorry if I wasn't clear on their acronyms.  It's in the camera icon in menu > Active D-Lighting where you have the Auto, Extra high, Normal, Low, and Off options.  Those appear in the sundry Nikon software converters too.  I believe Nikon's converters are the only ones that operate with those camera settings (Proprietary?).  I don't use Adobe or others as I don't know what their initial (made-up?) settings for the camera's NEF are per the post by Redcrown above.  They probably best guess or something since they all seem to differ someplace (e.g. My WB and tint never agrees between various maker's software either which is another matter.).

Also, I should have turned it (ADL) Off prior to getting into this mess.  I think the factory default is Auto and no doubt it is playing against me.

The "Brightness" setting does have some effect on moving the entire curve left or right.  It's in Camera icon > set Picture Control > Standard (Where mine is now, but Nikon seems to think it should be set to Neutral if I want to play with it as I am doing.) > with the three Brightness options of -1 , 0 or +1 in the D800E.  On mine, setting it to +1 seems to have moved the entire curve to the left a bit and opened up the shadows while giving me a bit more of a shoulder for the highlights.  Possibly the Contrast just steepens it or flattens it against the Brightness setting?  Haven't gotten there yet to see what happens to the Contrast curve.

Aside, I learned with the ExpoDisc that the narrower and sharper the spike in the camera's LCD at mid-screen, the more accurate the WB appears to be.  Seems the white spike on the LCD is the Luminance value of the combined RGB, and if the WB is off then the base of the spike widens a lot.  I can manually tune it for a sharper spike in CaptureNX-2 for the tint and WB.  Then apply it to a sequence if the same lighting existed.

SG

[TonyW]

SG, no problem in fact I should have recognised this as ADL.  

ADL may be a problem for you in two ways first it is applied in camera and adjust exposure slightly and secondly it is really meant for rendered files such as JPEG and TIFF.  The ADL instruction set is stored with the NEF and only those converters that are either Nikon own or use Nikon SDK are able to apply them to the NEF.  

Other raw converters will ignore these instructions (I assume this will also be the case for the exposure changes?).  ACR and LR presets for Nikon are fairly good at mimicking those in camera settings but of course cannot actually read them therefore any changes you have made will not be seen.

The Brightness setting in Picture Control again is really meant for JPEG, TIFF and although these settings will be honoured and applied to raw they will only be honoured in Nikon converters.

I think that you may be better starting with something like Camera Neutral or Standard with '0' settings and go from there with your testing.