Shooting Raw - Histograms & ETTR

[SolarPaul]

I want to get the best out of my photos and am really interested in ETTR.  There have been a number of discussions in this forum about 2 things which could better support this approach:

  - RAW Histograms
  - Auto exposure for ETTR (several approaches have been discussed)

Apart from the Magic Lantern software which supports ETTR for some Canon models (I have an Olympus OMD EM-1) I haven't been able to find much in the way of manufacturer support for either issue

There don't seem to have been any posts on this for a while (mea culpa if I have missed any) and it would be good to get an update on the latest situation. Does anybody have any news?

Thanks in Advance

[dwswager]

I want to get the best out of my photos and am really interested in ETTR.  There have been a number of discussions in this forum about 2 things which could better support this approach:

  - RAW Histograms
  - Auto exposure for ETTR (several approaches have been discussed)

Apart from the Magic Lantern software which supports ETTR for some Canon models (I have an Olympus OMD EM-1) I haven't been able to find much in the way of manufacturer support for either issue

There don't seem to have been any posts on this for a while (mea culpa if I have missed any) and it would be good to get an update on the latest situation. Does anybody have any news?

Thanks in Advance

Optimum Digital Exposure

There was a long forum thread about 2 months ago.  Basically, since there is no raw histogram, you can alter white balance and picture control properties on how the camera would convert to the raw data if you were shooting in an image format instead of Raw.

Bottom Line - Auto Exposure bracketing!  Would be nice if the there was an Auto Raw DR Bracketing mode, but alas, camera makers have not implemented that.

[SolarPaul]

Thanks.

It was logged under Optimum Exposure rather than ETTR and so I missed it. Wow, what a lot of info and differing viewpoints. I'm going to have to work hard to understand it! Certainly not as simple as I expected

Cheers

[bjanes]

Thanks.

It was logged under Optimum Exposure rather than ETTR and so I missed it. Wow, what a lot of info and differing viewpoints. I'm going to have to work hard to understand it! Certainly not as simple as I expected

Cheers

I think that approach is unnecessarily complicated and it would be simpler to revert to Michael's original concept of ETTR and that is to adjust the camera exposure so that the histogram is as far to the right without clipping as possible. IMHO, ETTR is best applied at base ISO and one moves the histogram to the right by using a wider aperture or slower shutter speed, increasing the number of captured photons and improving the signal to noise ratio. One can also move the histogram to the right by increasing the ISO, but this does not alter the number of photons collected and the SNR from shot noise is not affected. With some cameras such as most Canons, increasing the ISO will reduce the read noise and improve the shadows, but with newer Sony CMOS sensors, changing the ISO has a minimal effect. However, when shooting above base ISO, one should still adjust exposure so that the histogram is to the right so as to collect as many photons as possible.

The histogram on most cameras is somewhat conservative and will show clipping around 0.5 EV or slightly more before actual clipping in the sensor. The situation with the Nikon D800e is shown below. The exposure is from a Stouffer step wedge where the steps are in 0.3 EV increments. With the camera set to the camera standard picture control (normal contrast and saturation) the camera luminance histogram is just below clipping and the raw histogram shows the green channels about 2/3 EV from clipping. The red channel is another 1 EV to the left and a white balance multiplier of about 2.0 is needed for white balance. Since the luminance histogram reflects primarily the green channel, it gives a satisfactory indication of the green channels.



Increasing exposure by 0.3 EV causes clipping of the camera histogram, but the green channel is still slightly below clipping.



The signal:noise of a raw file varies as the square root of the number of photons captured. Underexposure by 1 EV will reduce the SNR only by a factor of 1.4. With modern sensors, one can still get excellent results with slight underexposure and the highlights will not be blown. If one fiddles around in an attempt for perfect ETTR exposure, he may miss the shot. With the D800e and most other modern cameras, one will get excellent results if exposure is adjusted so that the camera histogram is to the right without clipping and one need not bother with more exotic approaches for most practical work. When shooting red flowers, the situation is more complicated and beyond the scope of this post, but one should use the RGB histograms on the camera in this case.

Bill

[Isaac]

… that approach is unnecessarily complicated … With modern sensors, one can still get excellent results with slight underexposure and the highlights will not be blown.

Yes -- The perfect [best / better] is the enemy of the good. "Il meglio è l'inimico del bene."

[digitaldog]

I think that approach is unnecessarily complicated and it would be simpler to revert to Michael's original concept of ETTR and that is to adjust the camera exposure so that the histogram is as far to the right without clipping as possible.

Agreed, that’s pretty much what I do. Depending on the kind of shooting(lot’s of time to control the process so maybe even bracket) to fast and dirty (1/3 on exposure compensation), seems to work well once you have a good idea what tolerances you can get away with based on testing your sensor, camera-exposure system.

[SolarPaul]

Thanks very much bill. Your explanation makes sense and is easy to understand.

I am an absolute newbie at "serious" digital photography (hands on with RAW files) and I have  much to learn with just the basics of manipulating images and work flow. So I need to keep things in perspective and prioritise where I am going to apply my focus in order to balance the learning new basic skills with a vision of aiming for the best. I am reminded here of the old musical(?) aphorism "practice without theory is blind, theory without practice is sterile".

Are my thoughts below reasonable as a VERY basic summary and starting point for moving towards ETTR (Optimum Exposure)?

 * I will need to understand the dynamic range of the sensor of my camera (an Olympus OM-D E-M1)
 * Where the dynamic range of a scene to be photographed exactly matches the dynamic range of the sensor (let's assume no specular reflections or other areas where photon collection by pixel can be maximised or "blown") I would manage the exposure (ISO as low as possible and shutter speed and aperture as appropriate for my vision) to ensure no loss / minimise loss in highlights and shadows
 * Where the dynamic range of a scene to be photographed exceeds the dynamic range of the sensor I would have to make decisions about what I was prepared to lose (I would expect specular reflections etc to be a first choice for most situations)
 * Where the dynamic range of a scene to be photographed is less than the dynamic range of the sensor then I should try to "move the histogram to the right" but stopping before clipping highlights (except possibly again for specular highlights etc). The objective of this is to improve the signal to noise ratio and reduce noise in shadows. This moving to the right should be achieved by slower shutter speeds and or wider apertures (to get more photons onto the sensor) than by increasing ISO which will not result in capture of more photons
 * Given that Bayer sensors record colours separately (Red, Blue & Green x 2) the above matters really need to be addressed by individual colour rather than as a single dimension
 * Tools such as RawDigger can provide an accurate histogram by colour of the image trapped by the sensor (after the fact)
 * In camera histograms are based on JPEG previews (aiming for 18% grey balance) and may not be accurate for managing exposure for ETTR (Optimum Exposure)

I can see that different approaches and techniques are being championed to manage exposure for ETTR (Optimum Exposure) when taking photos but I don't want to pick up that hot potato now as it might distract from getting a basic understanding and starting point which will be enough for me for the present

Am I thinking straight?

[nma]

I am using the E-m1, incorporating ETTR in most shots.  I have verified this approach "post hoc," by examining the raw histogram in LR or PS. You have to keep your nerve, the camera jpg will flash burn-out warnings but the Raw is fine.  I find the E-m1 exposure to be very accurate, very seldom misses. I also use a lot of HDR bracketing because it is very convenient when the scene has too high DR for an LDR shot.

1. set the exposure flags in Menu 4, Gear G for highlights=251 and shadows=4
2. Select jpg settings for contrast, saturation, and sharpening to -1
3. wb = auto
4. Set exposure front dial so that EV is between +1 and +2 (example for Mode=A)

In step 4 I vary the EV setting by "experience". In very contrasty scenes EV=+1 or even EV=+0.7. In low contrast scenes EV=+2

[SolarPaul]

Thanks for the settings.

I'll try them out after my E-M1 arrives (its on order) and let you know how I go

Cheers

[bjanes]

Thanks very much bill. Your explanation makes sense and is easy to understand.

I can see that different approaches and techniques are being championed to manage exposure for ETTR (Optimum Exposure) when taking photos but I don't want to pick up that hot potato now as it might distract from getting a basic understanding and starting point which will be enough for me for the present

Am I thinking straight?

Paul,

I think your analysis is right on and you have a good foundation for ETTR and its pros and cons.

White balance is often not taken into account and it is particularly acute when photographing red flowers. A common rule of thumb is to decrease exposure by 1 EV when photographing saturated red flowers, but analysis with Rawdigger shows this to be bad advice. If you photograph a white target under daylight and look at the raw file, you will see that the red channel is about 1 EV to the left of green, but this varies somewhat among different cameras. DXO lists the WB multipliers for various cameras or you can do your own test.

Shown below are camera and raw histograms for the D800e with WB set to daylight on the left and UniWB on the right. The camera standard picture control with normal contrast and saturation were used and the color space was set to AdobeRGB (the widest space available on this camera). With the metered exposure, the red channel is two stops below saturation and the camera histogram shows strong clipping in the red channel, but the UniWB gives a more reasonable histogram and shows no need to reduce exposure.



To eliminate clipping in the red channel, it is necessary to reduce exposure by 1 full EV and the red channel is 3 EV short of clipping.


The camera can capture reds considerably more saturated than accommodated by AdobeRGB and this accounts for about 1 EV of clipping as can be shown be rendering the file into ProPhotoRGB.

To get an ETTR exposure in the red channel, it is necessary to increase exposure by 2 EV over metered. One would then have to use negative exposure in the raw converter and a color shift could occur if the raw converter shows a hue twist or if the response curve of the camera is non-linear. IMHO, the metered exposure or a +1EV exposure is reasonable. One could reduce saturation in the camera setting for the JPEG preview, but then the preview would looked washed out and live view focusing could be impaired. Other opinions are welcome.



Regards,

Bill

[digitaldog]

Am I thinking straight?

Yes! And it’s really not that much different than analog photography. We tested films+exposure+development and futzed around endlessly to get what we wanted.

[dwswager]

ETTR is an attempt to overcome the performance limitations of DSLR sensors.  As sensors systems (sensors, associated electronics, and firmware) get better, this technique provides diminishing returns.  Like all techniques, it is important to understand when the trick is useful and how to pull it off before you actually need it in the field. Obviously, the goal is to get the sensor systems to the point that the trick is no longer necessary, though it is unlikely we will actually get there.  Same is true for exposure stacking to extend DR.

Gratuitous blowviating...

What I find incomprehensible is that in this day and age, camera manufacturers have not tried to differentiate their products from their competitors by including these features in their upper end cameras.  For example, Auto Exposure Bracketing is still viewed by camera manufacturers as a tool to automatically vary exposure to get the single 'best' frame without regards to maybe you want them all.

How many of us would consider buying a new camera or even switching brands if you could buy cameras that had:

1. Auto ETTR where given the ISO, aperture, and shutter speed, you set 2 and it sets the 3rd to get you ETTR results.  Possibly with an option to equalize the channels with correction factors bringing other channels up.  I used to shoot my D70 with a cyan filter to retard the red channel.  Depending on the placement and quality of the exposure meter, it would probably take at least 2 shots to get the ETTR.

2. Auto DR Exposure Bracketing where you set the EV steps and hit the button and camera continues shooting varying shutter speed such that you end up with a complete set of images at the EV steps specified that cover the entire DR of the scene.  1 button press, 1 complete set of images.

3. Auto Focus Stacking set...Basically Helicon Remote built into the camera.  You set the parameters and near and far focus point and hit the remote release and the camera calculates the shots and foucs steps necessary to get you there.

4. Features we might not even know we want and stuff we haven't even thought of yet!  Think of things you do on your cell phone that you didn't even know you wanted to do until the feature was there or someone showed it to you.  This is the area where I think consumer electronics companies like Samsung and Sony can attack and kill traditional camera manufacturers.

[SolarPaul]

Thanks Bill & DWS

Well what a day I've had. I thought I would do a bit more research and see what others have been saying about ETTR etc. I feel like I stumbled into one of the major battles in Game of Thrones. Boy talk about lots of blood! I didn't realise that this was such a controversial topic.

It seems that the theory is good but hard to implement in practice with JPEG histograms and cameras exposing for JPEGs (optimising exposure for JPEGs). I can understand why the manufacturers do this but I find it hard to believe that none appear to be offering raw shooters a choice of optimising exposure for RAW - with RAW histograms and exposure settings.

I can't wait to get my hands on my new camera, learn some tools and try all this out. You bet I will be shooting scenes with low dynamic range to test for ETTR benefits in shadows and see what happens with colours. Red flowers and other high colour scenes are a must

Best Regards

[spidermike]

I can understand why the manufacturers do this but I find it hard to believe that none appear to be offering raw shooters a choice of optimising exposure for RAW - with RAW histograms and exposure settings.

I may be wrong but as far as I qam aware raw data is just that - raw data. Information from the little 1s and 0s in the digital circuitry. It is only when you interpret those 1s and 0s are converted to jpeg-compatible output that you get the LCD image and the knowledge that a photo is over exposed. As an example, if you set your camera to 'natural' or 'landscape' or 'portrait' you will get different histograms because the camera is applying different algorithms, and as a result will get subtly different highlight/shadow warnings. So if you were going to get auto-ETTR for raw data which interpretation would you base it on?
Add to that the jpeg is 8-bit but the raw data is 12 or even 14 bit and it is this that means the raw data has about 1-stop more headroom over the jpeg data so wihtout 'converting' to a jpeg-compatible output how would the camera would need to know '1-stop more headroom than what'.

[dwswager]

I may be wrong but as far as I qam aware raw data is just that - raw data. Information from the little 1s and 0s in the digital circuitry. It is only when you interpret those 1s and 0s are converted to jpeg-compatible output that you get the LCD image and the knowledge that a photo is over exposed. As an example, if you set your camera to 'natural' or 'landscape' or 'portrait' you will get different histograms because the camera is applying different algorithms, and as a result will get subtly different highlight/shadow warnings. So if you were going to get auto-ETTR for raw data which interpretation would you base it on?
Add to that the jpeg is 8-bit but the raw data is 12 or even 14 bit and it is this that means the raw data has about 1-stop more headroom over the jpeg data so wihtout 'converting' to a jpeg-compatible output how would the camera would need to know '1-stop more headroom than what'.

That is exactly the limited thinking that causes camera makers to do what they do.  First, they are all driven to get the best image, as they see it, out of the camera.  There is a big market for that because of the need for immediacy in a lot of market segments.  And yes, the raw data is just that:  Basically a voltage generated by a sensor well of RGB in the Bayer pattern.  And if total reality, what you would want is the exact representation of what the sensor sees.  But not even the sensor itself translate light intensities of varying colors into the 'real' values.

All the camera would need to know is what is the noise floor - where does an actual signal appear outside sensor noise.  And where is the sensor well threshold limit - that point at which the sensor well is producing all the voltage it can and additional light causes no more signal.  Those are the 2 clipping points and the spread defines the usable DR of the sensor.  The actual exposure is irrelevant because the data has to be converted in post processing and the technician will set it as they see fit.  The same is true of individual channel intensities and contrast (global and local).  Think of all the different parameters like WB, Contrast, Brightness, Etc that you can set to change a JPEG produced by the camera.  There really is no 'base' image and none is necessary.  This is precisely the advantage of RAW data...it can be manipulated.

[spidermike]

That is exactly the limited thinking that causes camera makers to do what they do.  First, they are all driven to get the best image, as they see it, out of the camera. 

That's not really relvant to the technical aspects of getting ETTR to work with raw data.

All the camera would need to know is what is the noise floor - where does an actual signal appear outside sensor noise.  And where is the sensor well threshold limit - that point at which the sensor well is producing all the voltage it can and additional light causes no more signal.  Those are the 2 clipping points and the spread defines the usable DR of the sensor. 

So if setting up a 'ETTR program' to work with raw data in camera, what is the limit - one pixel that excees the 'well threshold limit'? Or 1% of pixels? Or 10%? Do you base it on that 10% pixels being contiguous or scattered anywhere on the sensor? 
By the time you have sorted that out you may as well fire off a set of backeted exposures and pick the one that suits your desired output.

[bjanes]

I may be wrong but as far as I qam aware raw data is just that - raw data. Information from the little 1s and 0s in the digital circuitry. It is only when you interpret those 1s and 0s are converted to jpeg-compatible output that you get the LCD image and the knowledge that a photo is over exposed. As an example, if you set your camera to 'natural' or 'landscape' or 'portrait' you will get different histograms because the camera is applying different algorithms, and as a result will get subtly different highlight/shadow warnings. So if you were going to get auto-ETTR for raw data which interpretation would you base it on?
Add to that the jpeg is 8-bit but the raw data is 12 or even 14 bit and it is this that means the raw data has about 1-stop more headroom over the jpeg data so wihtout 'converting' to a jpeg-compatible output how would the camera would need to know '1-stop more headroom than what'.

Mike,

This is not correct. The JPEG is gamma encoded, and an 8 bit gamma 2.2 image has plenty of DR. See the chart by Norman Koren.

Regards,

Bill

[spidermike]

Mike,

This is not correct. The JPEG is gamma encoded, and an 8 bit gamma 2.2 image has plenty of DR. See the chart by Norman Koren.

Regards,

Bill

Can you explain what I said that is not correct?
I did not say a jpeg does not have sufficient DR, but what your point does suggest is that desiging an in-camera ETTR function does not need to be based on raw data.

[SolarPaul]

Wonderful photos, many on this site, are taken by really talented pros and amateurs who argue for optimising exposure for RAW and it is quality like theirs that I aspire to. Time will tell whether I have a good enough eye. I'm a newbie (to digital) and this is all theory to me at present. Very interesting, but still theory. I want to test it all out and make my own decisions about what technical skills - I imagine there will be a number - will work for me.

It is clear to me, however, that:

• Images produced from RAW files are superior to those produced from JPEGs
• Current cameras are designed to optimise exposure for JPEGs
• The RAW files available for post processing are based on those JPEG optimums

It seems logical that optimising exposure for JPEGs will often but not always produce the optimum exposure for RAW processing and Bill's example of the red flower above looks to be an excellent example of this. I expect that if one is to have the "Best" RAW file to work with then care needs to be taken to ensure that the image captured:

• Falls within the dynamic range of the sensor (by colour)
• Where the dynamic range of the scene is wider than the sensor then the photographer should be able to choose what parts are sacrificed (which areas should be allowed to clip or block)
• Where the dynamic range of the scene is lower than the sensor then proponents of ETTR argue that noise will be minimised if the RAW image maximises photon capture through increased aperture or slower shutter speed (move the histogram to the right with subsequent pull back in "development")

Opponents of optimum exposure for RAW (I now understand that ETTR is just one aspect of this) often argue, amongst other things, that shots are missed while practitioners fiddle around with exposure and that there is a real danger of "falling off the edge" (clipping highlights). This is probably true but it appears to me that that isn't because there is anything wrong with the objectives but with the tools which are set up to produce different things (JPEGs), thereby forcing practitioners who want the best "to fiddle around in the dark".

One objection to ETTR and its counterpart DTTL that does concern me, though, is the issue colours being affected and how much effort is required to "correct" or manage this in post processing. Bill alludes to this in his post above re the red flower. I would like to understand this better and any input or discussion on this would be much appreciated.

Regardless of any constraints on ETTR/DTTL from colour issues, arguing that exposure for JPEGs is "proper" exposure for RAW just doesn't make sense to me.  Even without any real hands on experience yet it seems sensible and logical to me that my camera should offer a choice of 2 exposure modes:

• Optimise exposure for JPEG - pretty much as is
• Optimise exposure for RAW - giving me a choice of parameters that I can choose / set so that I can realise my vision

[bjanes]

Can you explain what I said that is not correct?
I did not say a jpeg does not have sufficient DR, but what your point does suggest is that desiging an in-camera ETTR function does not need to be based on raw data.

Mike,

I beg your pardon. Since you mentioned 8 bit JPEG and then 12 or 14 bit raw, I assumed that you were talking about a DR limitation. So what was your point in discussing bit depth? Obviously, a raw histogram requires raw data.

Bill