Precise Digital Exposure

[ErikKaffehr]

Hi,

Most raw converters have some hue twists that may depend on exposure. ETTR may affect those "hue twists".

http://dcptool.sourceforge.net/Hue%20Twists.html

http://chromasoft.blogspot.se/2009/02/visualizing-dng-camera-profiles-part-1.html

Best regards
Erik

I've found that ETTR techniques usually result in subtle and undesirable color shifts in the finished result. I'd rather have better colors and tonalities and a bit more noise.

[ErikKaffehr]

Hi,

It is very simple, optimal capture is where as many photons as possible are collected, without clipping any channel.

The reason is simply that shot noise is minimised at maximum exposure, or more correctly Signal/Noise ratio is maximised with maximum possible exposures.

The issue is a bit muddled when we leave base ISO, as readout noise may be lower on some sensors at higher ISO. Other muddling factors are that raw converters cheat a lot with histograms. Lightroom always seems to do some highlight compression  and can be quite cavalier about highlight recovery. Capture One's "film curve" makes an ETTR image look horribly overexposed and the histogram goes with it. :-) We may ask Adobe and Capture one for correct raw histograms before asking camera vendors. :-)

Personally, I used to use a spot meter in film times. With slide film I exposed for the highlights and with B&W film I exposed for the shadows. The idea with the zone system was that exposure and development were interrelated. So, processing was taken into account at exposure and different processing was applied to individual exposures.

With digital all this makes little sense. Expose ETTR, or for mid tones if concerned about hue twists. Once capture has been made, we can apply any kind of processing.

Using a spot meter can be useful for finding optimal exposure. Personally I stopped spot metering a couple of weeks after going into digital. With film I still use the spot meter, as the film back has no histogram.

Best regards
Erik

Yes, I agree, of course it will. ETTR is just optimal exposure. The exposure is either such it produces optimal data or it isn’t and there are degrees in which sub optimal data affects our work. Now what the comment might imply is that less than optimal exposure (ETTR) will produce results no one can see and that I suppose is possible. This is much like the use of editing in high bit (16-bit) because we know rounding errors could, possibly result in data loss that is visible at some point on some output devices. It might not. But why take the chance?

[bernie west]

Yes that's right. It might not need them but certainly in LR if you pull the exposure around you can see non-linear changes to the channels in the histogram.

It will definitely be non-linear in demosaiced white-balanced images.  I'm more wondering if it's non-linear on the underlying raw data.  As Guillermo says, if you 'normalise' your image with linear raw adjustments, then it will by definition (more or less) turn out the same as a 'normal' jpg exposure.  That you and I and others are finding that they sometimes aren't the same would suggest that the Exposure slider in LR isn't working in a fully linear way with the underlying raw data.  I haven't played with raw data since back in 2008 when we where all trying to work out the optimal approach to ETTR, so I might shoot some test shots and see exactly what is happening with my current setup (Nikon with PV2012 vs Canon with some other earlier PV), when I get some time.

[bernie west]

The behavior of the exposure slider in ACR/LR depends on which process version is in use. With the current process, PV2012, the exposure slider as well as all the other sliders in the basic panel are image adaptive. See the post by Eric Chan on the Adobe forums. Auto highlight recovery is always in use, which can hide overexposure in the raw file. With the earlier process version, PV2010, I think that the exposure slider was not image adaptive (that is was linear) unless highlight recovery is taking place.

The newer Adobe profiles have hue twists, which can cause problems when exposure and recovery are in use. See SandyMc's post here. These are introduced with profiles that have lookup tables in addition to the matrix math. Sandy's DCPtool an address this problem. The whole topic is complicated and beyond the scope of my expertise. However, in my experience the hue shifts are not problematic if there are no blown highlights.

Hopefully, some of forum heavies will enter into discussion.

Bill

Interesting stuff!  If it's beyond your understanding, it will be beyond mine as well.  I've only just started to bother changing from the default Adobe profile to the camera specific profiles, so I'm hoping I might see some changes now.  I think I'll shoot some test shots and take a look at the raw data and do conversions in dcraw vs LR and see if I can get a picture of what's going on for my specific Nikon model.  Either way, I'm much more happy with adobe default renderings of my Nikon raws than I ever was with the raws off my Canon.

[bernie west]

Adode code does not do exposure correction on raw channels, the data are already after demosaick and then after color transform...

It has to be doing something with the raw data, or else it could never rescue blown highlights.

[bjanes]

How did you arrive at your +2½ to 3 ½ stops for highlights? Trial and error? Experience?
“I was able to establish… key values to spot meter, where… the location of extreme highlight and shadow values with RGB values maintained or with recovery”

It doesn't need to be this complicated! Simply expose a gray card to +5 stops in 1/3 stop increments. This will produce 16 exposures. View these 16 exposures in your digital Raw Processing software and see which exposure reads 99% brightness - the Optimum White Point [OWP] for your system (meter/camera/software) combination.

My experience with all my cameras shows the [OWP] falls between +3_2/3 stops to +4_1/3 stop (arrived at by the exposure just before I reach LR's red “Highlights Clipping” warning). This is the Exposure Bias [EB] that needs to be applied to your spot meter reading of the brightest area in your scene. This will produce 99% brightness in your raw software- the [OWP].

Bob,

I would not advise using LR/ACR to determine where the sensor saturates, especially with the current process version (PV2012) which applies automatic highlight recovery. In addition, LR/ACR have a baseline exposure adjustment. For example, the BaselineExposure for the Nikon D3 is +0.5 EV and the values for the D70 and D200 are -0.75 and -0.5 respectively. These adjustments increase or decrease the rendered values by the amount of the baseline adjustment.

Most light meters (including those built into our cameras) are calibrated to yield 12% saturation rather than the 18% expected for mid-gray (see Thom Hogan). This allows 0.5 EV of highlight headroom. The difference between 18% and 100% expressed in stops is log base 2(1.0/0.18) or 2.5 stops. For 12% the difference is 3 stops. To place a metered tone at saturation with a camera calibrated for 12%, one must increase the metered exposure by 3 stops. If you are willing to use highlight recovery of 0.5 to 1.0 EV in the raw converter, this could be increased to 3.5 or 4.0 stops, which is consistent with your empirically observed results. The rendered values with ACR/LR would be affected by the baseline offset.

Regards,

Bill

[Guillermo Luijk]

Adode code does not do exposure correction on raw channels, the data are already after demosaick and then after color transform...

If data is demosaiced and linearly (matrix) colour transformed, and even if it has been gamma transformed (this only applies to pure gamma lifting, not sRGB-like gammas), it should be possible to change exposure without any hue shift through scaling the RGB values by a constant value.

I think the point is that commercial RAW developers could apply more sophisticated non-linear processing (colour transform, gamma, highlight/shadow recovery strategies) either prior to the exposure adjustment, or after it but exposure dependent, that surely have other benefits but ruin the linear properties of the colorimetrics in RAW data. The differences are not big though, so only colour peepers could be interested in these hue shifts, not the regular user.

Regards!

[AlterEgo]

It has to be doing something with the raw data, or else it could never rescue blown highlights.

no, inventing the data in "blown" highlights does not need (it is not about what is better - but about what is possible) "raw" data... you have for example 2 channels in their internal RGB color space after the color transform w/o "clipping" - so you can use that information to invent the data in the 3rd channel...

[AlterEgo]

If data is demosaiced and linearly (matrix) colour transformed, and even if it has been gamma transformed (this only applies to pure gamma lifting, not sRGB-like gammas), it should be possible to change exposure without any hue shift through scaling the RGB values by a constant value.

which is exactly why it is no longer the raw data - it is already 1) demosaicked and 2) color transformed into some proper color space and 3) with new Adobe profiles a LUT (=non linear transform) can be applied before exposure slider operation (Adobe supplies 2.5D LUTs, but you can get a profile that does full 3D transform if you want, that is totally allowed - so there you go with hue shifts)...

[AlterEgo]

In addition, LR/ACR have a baseline exposure adjustment.

Adobe code has one hidden expocorrection inside it (hardcoded, might be zero) and one picked (if present) from DCP profiles (BaselineExposureOffset)... so the real hidden expocorrection is a sum of two components... convert a raw to DNG using Adobe's code (ACR, LR, DNG converter) and check "BaselineExposure" tag for what is in the code (vs what is in profiles)... that tag is "ISO" dependent... it can be different for raws shot with different nominal "ISO"s... for example Sony A7  = ISO 50 : BaselineExposure = -0.65 and ISO 100+ : BaselineExposure = +0.35 ... you can also check Fuji's x-trans cameras for high ISOs 

[bernie west]

no, inventing the data in "blown" highlights does not need (it is not about what is better - but about what is possible) "raw" data... you have for example 2 channels in their internal RGB color space after the color transform w/o "clipping" - so you can use that information to invent the data in the 3rd channel...

I'm talking about highlights blown in the jpg rgb, but not in the raw data.  LR absolutely works on the raw data.  The only question is: does the exposure slider work linearly on the underlying raw data?  And from what others have said, it appears it doesn't.  This verifies my feeling over the years from playing around with extreme ETTR (i.e. exposed just to the point of raw clipping).

[AlterEgo]

LR absolutely works on the raw data. 

no, it does not... as it was noted - demosaick and color transform (linear /matrix/ or non linear /matrix + LUT or dummy matrix + LUT/) before exposure correction in UI... that makes it a non-raw data... demosaick alone makes it a non-raw data... think about this - Adobe code works identically with both original raw files and with lossless linear DNG files converted from original raw files using that code, because it ( the code) is written to do this by design - that is the Adobe's strategy - no difference between the original raw files and lossless (linear or not) DNG when using LR/ACR...

do you have the original raw channels in linear DNG files ? but you still can "recover" (what a wrong word, as if the data is actually present !) clipped highlights there...

still think that "or read a good summary" is a good idea ?

[bernie west]

You're kind of missing the point.  The distinction under discussion was between raw and fully rendered rgb values (i.e. the histogram in LR - that is, post everything, including white balance and gamma correction).  Not between raw and post demosaic/color-transform.  I accept your point that it's not working directly on the raw data itself, but in the context of this debate, it's still working very close to the raw data itself, as opposed to after white balancing and gamma correction, as is the case when you work on an image in photoshop.  If there was no benefit to be gained in this distinction we wouldn't bother doing any image manipulation in LR.  We'd just do it all in photoshop after rendering with dcraw or something equally similar and fast.

[AlterEgo]

I accept your point that it's not working directly on the raw data itself... as opposed to after white balancing and gamma correction

consider this - exposure correction is done there (ACR/LR) after WB operations and after color transform which might be as well applying some "gamma" (or any other curve - whatever is in the 1st stage LUTs by their design)

[AlterEgo]

We'd just do it all in photoshop after rendering with dcraw or something equally similar and fast.

consider this - such raw converter as lightzone ( http://www.lightzoneproject.org/ ) was doing everything after running dcraw executable

[hjulenissen]

no, it does not... as it was noted - demosaick and color transform (linear /matrix/ or non linear /matrix + LUT or dummy matrix + LUT/) before exposure correction in UI... that makes it a non-raw data... demosaick alone makes it a non-raw data...

I'd be surprised if Adobe chose to do highlight recovery on data that has been processed by demosaic, and even more if it has been processed by (potentially nonlinear) color processing. After such processing, the saturation point is hard to define and a single blown sensel in a single channel can affect all 3 channels in a spatial neighborhood.

Not saying that it can't be done, only that it (from my POV) seems like making the task harder than it needs to be.

-h

[bernie west]

consider this - exposure correction is done there (ACR/LR) after WB operations and after color transform which might be as well applying some "gamma" (or any other curve - whatever is in the 1st stage LUTs by their design)

So now it's after demosaic, colour transform, AND white balance??  What's it going to change to next comment?

I find it hard to believe it would be operating on the data after WB, as it would recover stuff-all highlights that were blown as indicated on the LR histogram.  As I said, what would be the point of using LR if it only worked on essentially rendered data? 

[AlterEgo]

So now it's after demosaic, colour transform, AND white balance??  What's it going to change to next comment?

that was always the case - where did you see the changes ?

[AlterEgo]

I'd be surprised if Adobe chose to do highlight recovery on data that has been processed by demosaic

did you see the note about linear DNG files ? that is a fact - those have demosaicked data, yet there are no difference between them and original raw files if you use Adobe software and conversion to linear DNG was done by Adobe software (the same version for DNG conversion and raw conversion, lossless DNG, just to make sure)

[bernie west]

that was always the case - where did you see the changes ?

You originally said 'demosaic and colour transform'.  Twice.  Unless "colour transform" includes white balance.  I assumed colour transform means conversion to a colour space.  

But once again, this is all dodging around the point of the discussion.