Luminous Landscape Forum
Equipment & Techniques => Digital Cameras & Shooting Techniques => Topic started by: SolarPaul on January 01, 2015, 07:21:28 pm
-
I want to get the best out of my photos and am really interested in ETTR. There have been a number of discussions in this forum about 2 things which could better support this approach:
- RAW Histograms
- Auto exposure for ETTR (several approaches have been discussed)
Apart from the Magic Lantern software which supports ETTR for some Canon models (I have an Olympus OMD EM-1) I haven't been able to find much in the way of manufacturer support for either issue
There don't seem to have been any posts on this for a while (mea culpa if I have missed any) and it would be good to get an update on the latest situation. Does anybody have any news?
Thanks in Advance
-
I want to get the best out of my photos and am really interested in ETTR. There have been a number of discussions in this forum about 2 things which could better support this approach:
- RAW Histograms
- Auto exposure for ETTR (several approaches have been discussed)
Apart from the Magic Lantern software which supports ETTR for some Canon models (I have an Olympus OMD EM-1) I haven't been able to find much in the way of manufacturer support for either issue
There don't seem to have been any posts on this for a while (mea culpa if I have missed any) and it would be good to get an update on the latest situation. Does anybody have any news?
Thanks in Advance
Optimum Digital Exposure (http://www.luminous-landscape.com/essays/the_optimum_digital_exposure.shtml)
There was a long forum thread about 2 months ago. Basically, since there is no raw histogram, you can alter white balance and picture control properties on how the camera would convert to the raw data if you were shooting in an image format instead of Raw.
Bottom Line - Auto Exposure bracketing! Would be nice if the there was an Auto Raw DR Bracketing mode, but alas, camera makers have not implemented that.
-
Thanks.
It was logged under Optimum Exposure rather than ETTR and so I missed it. Wow, what a lot of info and differing viewpoints. I'm going to have to work hard to understand it! Certainly not as simple as I expected
Cheers
-
Thanks.
It was logged under Optimum Exposure rather than ETTR and so I missed it. Wow, what a lot of info and differing viewpoints. I'm going to have to work hard to understand it! Certainly not as simple as I expected
Cheers
I think that approach is unnecessarily complicated and it would be simpler to revert to Michael's original concept of ETTR and that is to adjust the camera exposure so that the histogram is as far to the right without clipping as possible. IMHO, ETTR is best applied at base ISO and one moves the histogram to the right by using a wider aperture or slower shutter speed, increasing the number of captured photons and improving the signal to noise ratio. One can also move the histogram to the right by increasing the ISO, but this does not alter the number of photons collected and the SNR from shot noise is not affected. With some cameras such as most Canons, increasing the ISO will reduce the read noise and improve the shadows, but with newer Sony CMOS sensors, changing the ISO has a minimal effect. However, when shooting above base ISO, one should still adjust exposure so that the histogram is to the right so as to collect as many photons as possible.
The histogram on most cameras is somewhat conservative and will show clipping around 0.5 EV or slightly more before actual clipping in the sensor. The situation with the Nikon D800e is shown below. The exposure is from a Stouffer step wedge where the steps are in 0.3 EV increments. With the camera set to the camera standard picture control (normal contrast and saturation) the camera luminance histogram is just below clipping and the raw histogram shows the green channels about 2/3 EV from clipping. The red channel is another 1 EV to the left and a white balance multiplier of about 2.0 is needed for white balance. Since the luminance histogram reflects primarily the green channel, it gives a satisfactory indication of the green channels.
(http://bjanes.smugmug.com/Photography/Red-Flower-III/i-wR3hXrV/0/XL/img2_comp-XL.png)
Increasing exposure by 0.3 EV causes clipping of the camera histogram, but the green channel is still slightly below clipping.
(http://bjanes.smugmug.com/Photography/Red-Flower-III/i-TTZBbgH/0/XL/img1_comp-XL.png)
The signal:noise of a raw file varies as the square root of the number of photons captured. Underexposure by 1 EV will reduce the SNR only by a factor of 1.4. With modern sensors, one can still get excellent results with slight underexposure and the highlights will not be blown. If one fiddles around in an attempt for perfect ETTR exposure, he may miss the shot. With the D800e and most other modern cameras, one will get excellent results if exposure is adjusted so that the camera histogram is to the right without clipping and one need not bother with more exotic approaches for most practical work. When shooting red flowers, the situation is more complicated and beyond the scope of this post, but one should use the RGB histograms on the camera in this case.
Bill
-
… that approach is unnecessarily complicated … With modern sensors, one can still get excellent results with slight underexposure and the highlights will not be blown.
Yes -- The perfect [best / better] is the enemy of the good. "Il meglio è l'inimico del bene."
-
I think that approach is unnecessarily complicated and it would be simpler to revert to Michael's original concept of ETTR and that is to adjust the camera exposure so that the histogram is as far to the right without clipping as possible.
Agreed, that’s pretty much what I do. Depending on the kind of shooting(lot’s of time to control the process so maybe even bracket) to fast and dirty (1/3 on exposure compensation), seems to work well once you have a good idea what tolerances you can get away with based on testing your sensor, camera-exposure system.
-
Thanks very much bill. Your explanation makes sense and is easy to understand.
I am an absolute newbie at "serious" digital photography (hands on with RAW files) and I have much to learn with just the basics of manipulating images and work flow. So I need to keep things in perspective and prioritise where I am going to apply my focus in order to balance the learning new basic skills with a vision of aiming for the best. I am reminded here of the old musical(?) aphorism "practice without theory is blind, theory without practice is sterile".
Are my thoughts below reasonable as a VERY basic summary and starting point for moving towards ETTR (Optimum Exposure)?
* I will need to understand the dynamic range of the sensor of my camera (an Olympus OM-D E-M1)
* Where the dynamic range of a scene to be photographed exactly matches the dynamic range of the sensor (let's assume no specular reflections or other areas where photon collection by pixel can be maximised or "blown") I would manage the exposure (ISO as low as possible and shutter speed and aperture as appropriate for my vision) to ensure no loss / minimise loss in highlights and shadows
* Where the dynamic range of a scene to be photographed exceeds the dynamic range of the sensor I would have to make decisions about what I was prepared to lose (I would expect specular reflections etc to be a first choice for most situations)
* Where the dynamic range of a scene to be photographed is less than the dynamic range of the sensor then I should try to "move the histogram to the right" but stopping before clipping highlights (except possibly again for specular highlights etc). The objective of this is to improve the signal to noise ratio and reduce noise in shadows. This moving to the right should be achieved by slower shutter speeds and or wider apertures (to get more photons onto the sensor) than by increasing ISO which will not result in capture of more photons
* Given that Bayer sensors record colours separately (Red, Blue & Green x 2) the above matters really need to be addressed by individual colour rather than as a single dimension
* Tools such as RawDigger can provide an accurate histogram by colour of the image trapped by the sensor (after the fact)
* In camera histograms are based on JPEG previews (aiming for 18% grey balance) and may not be accurate for managing exposure for ETTR (Optimum Exposure)
I can see that different approaches and techniques are being championed to manage exposure for ETTR (Optimum Exposure) when taking photos but I don't want to pick up that hot potato now as it might distract from getting a basic understanding and starting point which will be enough for me for the present
Am I thinking straight?
-
I am using the E-m1, incorporating ETTR in most shots. I have verified this approach "post hoc," by examining the raw histogram in LR or PS. You have to keep your nerve, the camera jpg will flash burn-out warnings but the Raw is fine. I find the E-m1 exposure to be very accurate, very seldom misses. I also use a lot of HDR bracketing because it is very convenient when the scene has too high DR for an LDR shot.
1. set the exposure flags in Menu 4, Gear G for highlights=251 and shadows=4
2. Select jpg settings for contrast, saturation, and sharpening to -1
3. wb = auto
4. Set exposure front dial so that EV is between +1 and +2 (example for Mode=A)
In step 4 I vary the EV setting by "experience". In very contrasty scenes EV=+1 or even EV=+0.7. In low contrast scenes EV=+2
-
Thanks for the settings.
I'll try them out after my E-M1 arrives (its on order) and let you know how I go
Cheers
-
Thanks very much bill. Your explanation makes sense and is easy to understand.
I can see that different approaches and techniques are being championed to manage exposure for ETTR (Optimum Exposure) when taking photos but I don't want to pick up that hot potato now as it might distract from getting a basic understanding and starting point which will be enough for me for the present
Am I thinking straight?
Paul,
I think your analysis is right on and you have a good foundation for ETTR and its pros and cons.
White balance is often not taken into account and it is particularly acute when photographing red flowers. A common rule of thumb is to decrease exposure by 1 EV when photographing saturated red flowers, but analysis with Rawdigger shows this to be bad advice. If you photograph a white target under daylight and look at the raw file, you will see that the red channel is about 1 EV to the left of green, but this varies somewhat among different cameras. DXO lists the WB multipliers for various cameras or you can do your own test.
Shown below are camera and raw histograms for the D800e with WB set to daylight on the left and UniWB (http://www.guillermoluijk.com/tutorial/uniwb/index_en.htm) on the right. The camera standard picture control with normal contrast and saturation were used and the color space was set to AdobeRGB (the widest space available on this camera). With the metered exposure, the red channel is two stops below saturation and the camera histogram shows strong clipping in the red channel, but the UniWB gives a more reasonable histogram and shows no need to reduce exposure.
(http://bjanes.smugmug.com/Photography/Red-Flower-III/i-CSZdgQ7/0/L/101-01-10-Histos-L.png)
To eliminate clipping in the red channel, it is necessary to reduce exposure by 1 full EV and the red channel is 3 EV short of clipping.
(http://bjanes.smugmug.com/Photography/Red-Flower-III/i-t9GmvWp/0/L/101-08-17-Histos-L.png)
The camera can capture reds considerably more saturated than accommodated by AdobeRGB and this accounts for about 1 EV of clipping as can be shown be rendering the file into ProPhotoRGB.
To get an ETTR exposure in the red channel, it is necessary to increase exposure by 2 EV over metered. One would then have to use negative exposure in the raw converter and a color shift could occur if the raw converter shows a hue twist or if the response curve of the camera is non-linear. IMHO, the metered exposure or a +1EV exposure is reasonable. One could reduce saturation in the camera setting for the JPEG preview, but then the preview would looked washed out and live view focusing could be impaired. Other opinions are welcome.
(http://bjanes.smugmug.com/Photography/Red-Flower-III/i-VMzXNQn/0/L/101-06-15-Histos-L.png)
Regards,
Bill
-
Am I thinking straight?
Yes! And it’s really not that much different than analog photography. We tested films+exposure+development and futzed around endlessly to get what we wanted.
-
ETTR is an attempt to overcome the performance limitations of DSLR sensors. As sensors systems (sensors, associated electronics, and firmware) get better, this technique provides diminishing returns. Like all techniques, it is important to understand when the trick is useful and how to pull it off before you actually need it in the field. Obviously, the goal is to get the sensor systems to the point that the trick is no longer necessary, though it is unlikely we will actually get there. Same is true for exposure stacking to extend DR.
Gratuitous blowviating...
What I find incomprehensible is that in this day and age, camera manufacturers have not tried to differentiate their products from their competitors by including these features in their upper end cameras. For example, Auto Exposure Bracketing is still viewed by camera manufacturers as a tool to automatically vary exposure to get the single 'best' frame without regards to maybe you want them all.
How many of us would consider buying a new camera or even switching brands if you could buy cameras that had:
1. Auto ETTR where given the ISO, aperture, and shutter speed, you set 2 and it sets the 3rd to get you ETTR results. Possibly with an option to equalize the channels with correction factors bringing other channels up. I used to shoot my D70 with a cyan filter to retard the red channel. Depending on the placement and quality of the exposure meter, it would probably take at least 2 shots to get the ETTR.
2. Auto DR Exposure Bracketing where you set the EV steps and hit the button and camera continues shooting varying shutter speed such that you end up with a complete set of images at the EV steps specified that cover the entire DR of the scene. 1 button press, 1 complete set of images.
3. Auto Focus Stacking set...Basically Helicon Remote built into the camera. You set the parameters and near and far focus point and hit the remote release and the camera calculates the shots and foucs steps necessary to get you there.
4. Features we might not even know we want and stuff we haven't even thought of yet! Think of things you do on your cell phone that you didn't even know you wanted to do until the feature was there or someone showed it to you. This is the area where I think consumer electronics companies like Samsung and Sony can attack and kill traditional camera manufacturers.
-
Thanks Bill & DWS
Well what a day I've had. I thought I would do a bit more research and see what others have been saying about ETTR etc. I feel like I stumbled into one of the major battles in Game of Thrones. Boy talk about lots of blood! I didn't realise that this was such a controversial topic.
It seems that the theory is good but hard to implement in practice with JPEG histograms and cameras exposing for JPEGs (optimising exposure for JPEGs). I can understand why the manufacturers do this but I find it hard to believe that none appear to be offering raw shooters a choice of optimising exposure for RAW - with RAW histograms and exposure settings.
I can't wait to get my hands on my new camera, learn some tools and try all this out. You bet I will be shooting scenes with low dynamic range to test for ETTR benefits in shadows and see what happens with colours. Red flowers and other high colour scenes are a must
Best Regards
-
I can understand why the manufacturers do this but I find it hard to believe that none appear to be offering raw shooters a choice of optimising exposure for RAW - with RAW histograms and exposure settings.
I may be wrong but as far as I qam aware raw data is just that - raw data. Information from the little 1s and 0s in the digital circuitry. It is only when you interpret those 1s and 0s are converted to jpeg-compatible output that you get the LCD image and the knowledge that a photo is over exposed. As an example, if you set your camera to 'natural' or 'landscape' or 'portrait' you will get different histograms because the camera is applying different algorithms, and as a result will get subtly different highlight/shadow warnings. So if you were going to get auto-ETTR for raw data which interpretation would you base it on?
Add to that the jpeg is 8-bit but the raw data is 12 or even 14 bit and it is this that means the raw data has about 1-stop more headroom over the jpeg data so wihtout 'converting' to a jpeg-compatible output how would the camera would need to know '1-stop more headroom than what'.
-
I may be wrong but as far as I qam aware raw data is just that - raw data. Information from the little 1s and 0s in the digital circuitry. It is only when you interpret those 1s and 0s are converted to jpeg-compatible output that you get the LCD image and the knowledge that a photo is over exposed. As an example, if you set your camera to 'natural' or 'landscape' or 'portrait' you will get different histograms because the camera is applying different algorithms, and as a result will get subtly different highlight/shadow warnings. So if you were going to get auto-ETTR for raw data which interpretation would you base it on?
Add to that the jpeg is 8-bit but the raw data is 12 or even 14 bit and it is this that means the raw data has about 1-stop more headroom over the jpeg data so wihtout 'converting' to a jpeg-compatible output how would the camera would need to know '1-stop more headroom than what'.
That is exactly the limited thinking that causes camera makers to do what they do. First, they are all driven to get the best image, as they see it, out of the camera. There is a big market for that because of the need for immediacy in a lot of market segments. And yes, the raw data is just that: Basically a voltage generated by a sensor well of RGB in the Bayer pattern. And if total reality, what you would want is the exact representation of what the sensor sees. But not even the sensor itself translate light intensities of varying colors into the 'real' values.
All the camera would need to know is what is the noise floor - where does an actual signal appear outside sensor noise. And where is the sensor well threshold limit - that point at which the sensor well is producing all the voltage it can and additional light causes no more signal. Those are the 2 clipping points and the spread defines the usable DR of the sensor. The actual exposure is irrelevant because the data has to be converted in post processing and the technician will set it as they see fit. The same is true of individual channel intensities and contrast (global and local). Think of all the different parameters like WB, Contrast, Brightness, Etc that you can set to change a JPEG produced by the camera. There really is no 'base' image and none is necessary. This is precisely the advantage of RAW data...it can be manipulated.
-
That is exactly the limited thinking that causes camera makers to do what they do. First, they are all driven to get the best image, as they see it, out of the camera.
That's not really relvant to the technical aspects of getting ETTR to work with raw data.
All the camera would need to know is what is the noise floor - where does an actual signal appear outside sensor noise. And where is the sensor well threshold limit - that point at which the sensor well is producing all the voltage it can and additional light causes no more signal. Those are the 2 clipping points and the spread defines the usable DR of the sensor.
So if setting up a 'ETTR program' to work with raw data in camera, what is the limit - one pixel that excees the 'well threshold limit'? Or 1% of pixels? Or 10%? Do you base it on that 10% pixels being contiguous or scattered anywhere on the sensor?
By the time you have sorted that out you may as well fire off a set of backeted exposures and pick the one that suits your desired output.
-
I may be wrong but as far as I qam aware raw data is just that - raw data. Information from the little 1s and 0s in the digital circuitry. It is only when you interpret those 1s and 0s are converted to jpeg-compatible output that you get the LCD image and the knowledge that a photo is over exposed. As an example, if you set your camera to 'natural' or 'landscape' or 'portrait' you will get different histograms because the camera is applying different algorithms, and as a result will get subtly different highlight/shadow warnings. So if you were going to get auto-ETTR for raw data which interpretation would you base it on?
Add to that the jpeg is 8-bit but the raw data is 12 or even 14 bit and it is this that means the raw data has about 1-stop more headroom over the jpeg data so wihtout 'converting' to a jpeg-compatible output how would the camera would need to know '1-stop more headroom than what'.
Mike,
This is not correct. The JPEG is gamma encoded, and an 8 bit gamma 2.2 image has plenty of DR. See the chart (http://www.normankoren.com/digital_tonality.html) by Norman Koren.
Regards,
Bill
-
Mike,
This is not correct. The JPEG is gamma encoded, and an 8 bit gamma 2.2 image has plenty of DR. See the chart (http://www.normankoren.com/digital_tonality.html) by Norman Koren.
Regards,
Bill
Can you explain what I said that is not correct?
I did not say a jpeg does not have sufficient DR, but what your point does suggest is that desiging an in-camera ETTR function does not need to be based on raw data.
-
Wonderful photos, many on this site, are taken by really talented pros and amateurs who argue for optimising exposure for RAW and it is quality like theirs that I aspire to. Time will tell whether I have a good enough eye. I'm a newbie (to digital) and this is all theory to me at present. Very interesting, but still theory. I want to test it all out and make my own decisions about what technical skills - I imagine there will be a number - will work for me.
It is clear to me, however, that:
- Images produced from RAW files are superior to those produced from JPEGs
- Current cameras are designed to optimise exposure for JPEGs
- The RAW files available for post processing are based on those JPEG optimums
It seems logical that optimising exposure for JPEGs will often but not always produce the optimum exposure for RAW processing and Bill's example of the red flower above looks to be an excellent example of this. I expect that if one is to have the "Best" RAW file to work with then care needs to be taken to ensure that the image captured:
- Falls within the dynamic range of the sensor (by colour)
- Where the dynamic range of the scene is wider than the sensor then the photographer should be able to choose what parts are sacrificed (which areas should be allowed to clip or block)
- Where the dynamic range of the scene is lower than the sensor then proponents of ETTR argue that noise will be minimised if the RAW image maximises photon capture through increased aperture or slower shutter speed (move the histogram to the right with subsequent pull back in "development")
Opponents of optimum exposure for RAW (I now understand that ETTR is just one aspect of this) often argue, amongst other things, that shots are missed while practitioners fiddle around with exposure and that there is a real danger of "falling off the edge" (clipping highlights). This is probably true but it appears to me that that isn't because there is anything wrong with the objectives but with the tools which are set up to produce different things (JPEGs), thereby forcing practitioners who want the best "to fiddle around in the dark".
One objection to ETTR and its counterpart DTTL that does concern me, though, is the issue colours being affected and how much effort is required to "correct" or manage this in post processing. Bill alludes to this in his post above re the red flower. I would like to understand this better and any input or discussion on this would be much appreciated.
Regardless of any constraints on ETTR/DTTL from colour issues, arguing that exposure for JPEGs is "proper" exposure for RAW just doesn't make sense to me. Even without any real hands on experience yet it seems sensible and logical to me that my camera should offer a choice of 2 exposure modes:
- Optimise exposure for JPEG - pretty much as is
- Optimise exposure for RAW - giving me a choice of parameters that I can choose / set so that I can realise my vision
-
Can you explain what I said that is not correct?
I did not say a jpeg does not have sufficient DR, but what your point does suggest is that desiging an in-camera ETTR function does not need to be based on raw data.
Mike,
I beg your pardon. Since you mentioned 8 bit JPEG and then 12 or 14 bit raw, I assumed that you were talking about a DR limitation. So what was your point in discussing bit depth? Obviously, a raw histogram requires raw data. :)
Bill
-
BJamnes - my reference to bit depth was that (or, more exactly, my assumption that) the greater bit depth in raw data led to the raw data having more headroom than the jpeg suggested: so when the jpeg data (and its associated histogram) showed the image was blown the raw data allowed you another stop or so before detail was unrecoverable.
But my first post was asking more that how can you have a 'raw histogram' - you need to interpret the raw data to have a histogram and that 'interpretatoin' is the jpeg file on which a histogram is based.
-
- Optimise exposure for JPEG - pretty much as is
- Optimise exposure for RAW - giving me a choice of parameters that I can choose / set so that I can realise my vision
Optimising or raw relies on you knowing from experience how the raw data relates to the jpeg data. On one camera model you will learn that you can expose so that the extreme of hte histogram si just touching the right hand edge, but on another camera you know that you can take it a tad further and still be OK with the raw data. But even then you still need to know how to interpret the histogram to know what you are looking at - for example if there is a small blip at the right hand ege is that specular highlights (reflections off turbulent water) or is that important detail (white feathers of a coot).
But I repeat my earlier question: is there anything such as a 'raw histogram'? How does a 'raw histogram' differ from a 'jpeg histogram'? Raw data needs to be interpreted into a picture to make any sense regards exposure and that is precisely what the jpeg picture does - and by changing the in-camera profile you change the histogram.
-
But I repeat my earlier question: is there anything such as a 'raw histogram'? How does a 'raw histogram' differ from a 'jpeg histogram'?
Hi,
A Raw histogram shows the Raw data, i.e. before White Balancing which might push some values into clipping (e.g. Red flowers). The Raw data histogram then tell us that the clipped values are purely the result of post-processing, and can therefore be completely restored with proper scaling in linear gamma space, before White Balancing. For example, RawTherapee does allow such corrections to be applied before demosaicing.
Cheers,
Bart
-
But I repeat my earlier question: is there anything such as a 'raw histogram'? How does a 'raw histogram' differ from a 'jpeg histogram'? Raw data needs to be interpreted into a picture to make any sense regards exposure and that is precisely what the jpeg picture does - and by changing the in-camera profile you change the histogram.
The raw data do not need to be demosaiced or otherwise interpreted in order to calculate a raw histogram. One can work directly with the raw channels, but frequently the values are binned into a number of levels which are shown in a frequency distribution (e.g. Rawdigger), but at least one program (Histogrammar (http://www.guillermoluijk.com/software/histogrammar/index.htm)) allows direct plotting (1:1) with no binning.
Bill
-
I tried uniwb once with my Canon but I think it's a bit messy to have a crazy white balance in the preview so I did not think it's worth it. Instead my approach is to make some tests and see how the histogram statistically relates to raw exposure. Getting the feel for the histogram and how much you can clip in various conditions one can expose well anyway.
ETTR may not be as important with cameras equipped with the low noise Sony Exmor, like the D800, but for my Canon and my MF backs is still rather important. In any case a raw histogram and raw highlight blinkies would not hurt to have as a setting for advanced users so I find it a bit frustrating that manufacturers are so clueless about this. I don't know of any camera with a properly designed raw histogram, out of the box.
We will probably never get it either, as sensors get less and less noisy it will be less and less important with ETTR and thus we'll care less about raw histograms, the exposure new standard will be like it was with slide film - if the preview looks good then the exposure is good (and that's been the thinking all along with the manufacturers).
In most cases the histogram is conservative, ie clips before the raw clips. But in some conditions it can be the opposite, ie the raw clips despite there's no clipping in the histogram and that's not good. For example if the histogram show luminance and there's a bright red color this can happen.
-
BJamnes - my reference to bit depth was that (or, more exactly, my assumption that) the greater bit depth in raw data led to the raw data having more headroom than the jpeg suggested: so when the jpeg data (and its associated histogram) showed the image was blown the raw data allowed you another stop or so before detail was unrecoverable.
Yes and no. Yes, there’s more bit depth, more data for editing the data before we may see damage (posterization) but no in terms of DR which is an attribute of the capture system. The JPEG looked blown out because the rendering from raw was incorrect. The raw allowed us to fix that issue but that data was there, no matter the bit depth. The JPEG is one (incorrect) answer from the raw data, bit depth not at that point being the factor.
-
Thank you again for clarifying it, guys.
-
So if setting up a 'ETTR program' to work with raw data in camera, what is the limit - one pixel that excees the 'well threshold limit'? Or 1% of pixels? Or 10%? Do you base it on that 10% pixels being contiguous or scattered anywhere on the sensor?
By the time you have sorted that out you may as well fire off a set of backeted exposures and pick the one that suits your desired output.
Yes, the ETTR case is the hard case. And all of what I think is missing is highly dependent on processing and onboard power; two very real constraints. And I certainly agree from a practical standpoint, shooting a bracketed set of exposures is probably going to be the best option in most cases. But the JPEG has a gamma correction applied while the RAW data is nominally linear. That is, the RAW data is already being manipulated by the processor. What I am asking a new mode of manipulation at the RAW level. And of course, some sort of setpoint needs to be decided. Since the case for ETTR is based on a situation where the scene is within the DR of the camera and over exposing to pull detail out of the shadow region where very few data bits are used up to a higher level where more data bits are utilized, we should be looking at no to very minimal highlight clipping. Maybe in ETTR mode, Expose Compensation on the camera acts as a 'gross' adjustment to the clipping setpoint.
-
The raw data do not need to be demosaiced or otherwise interpreted in order to calculate a raw histogram. One can work directly with the raw channels, but frequently the values are binned into a number of levels which are shown in a frequency distribution (e.g. Rawdigger), but at least one program (Histogrammar (http://www.guillermoluijk.com/software/histogrammar/index.htm)) allows direct plotting (1:1) with no binning.
Bill
If I understand the process correctly, analog data from the sensor must, in all cases, be "binned" at the point where the A/D converter writes the data to a (digitally encoded) file. I believe Rawdigger allows one to adjust the size of the bins, while Histogrammar uses bins at the maximum resolution of the bit depth used to write the raw file.
Please correct me if I am wrong.
-
RAW data are not binned in any way, and can be read 'unbinned', i.e. as a single value corresponding to one RAW channel for every sensel in the CFA (R, G1, G2, B).
In fact one day I started to improve Histogrammar to differentiate the G1 from the G2 green subchannels (some sensors like the old Panasonic's for some reason suffer from G unbalancing and I wanted to study that), but got a bit mad to build all the possible colour combinations needed for a beautiful colour histogram. For 3 channels this is trivial but 4 require quite a lot of extra effort.
BTW the RAW histogram + Auto ETTR story seems a lost battle. ETTR will loose all interest before camera makers decide to look at it:
http://www.luminous-landscape.com/forum/index.php?topic=33267
Regards!
-
Guillermo-
I think perhaps that I have a different (perhaps incorrect) understanding of "binning". I would like to resolve that issue but don't wish to hijack this thread in that direction. I have taken the liberty of sending you an e-mail.
Regards,
Mike.
-
Guillermo-
I think perhaps that I have a different (perhaps incorrect) understanding of "binning". I would like to resolve that issue but don't wish to hijack this thread in that direction. I have taken the liberty of sending you an e-mail.
Regards,
Mike.
'Binning' is exactly the process of building a histogram...how many in this interval tells the height of the histogram at that interval. Hence, I don't think it is hijacking, but clarifying.
-
Guillermo-
I think perhaps that I have a different (perhaps incorrect) understanding of "binning". I would like to resolve that issue but don't wish to hijack this thread in that direction. I have taken the liberty of sending you an e-mail.
Regards,
Mike.
Sorry Mike I thought by binning you meant the mixing of the G1 and G2 RAW channels. I wanted Histogrammar to have the possibility to go up to 1:1 zoom as Bill explains. You can produce histograms 65536 long.
I also kept all the 'bins' at every lower zoom level as power of two containers, so that the shape of the histogram never leads to confusion (in log mode some peaks can arise specially in the deep shadows though, but this is expected due to level quantization).
-
Hi,
A Raw histogram shows the Raw data, i.e. before White Balancing which might push some values into clipping (e.g. Red flowers). The Raw data histogram then tell us that the clipped values are purely the result of post-processing, and can therefore be completely restored with proper scaling in linear gamma space, before White Balancing. For example, RawTherapee does allow such corrections to be applied before demosaicing.
Cheers,
Bart
Hi,
White balance is not the only issue with non-Raw histograms which can show clipping. Color space encoding is another factor, responsible of the majority of the "Red blown channels" complaints.
RAW channels: luminance values of each photosite
JPeg or TIFF: RGB values (a coupled combination of luminance, hue and saturation)
If you have a clipped raw channel, it means you have exceeded the maximum value of luminance that the sensor can handle (excesive exposure)
If you have a clipped RGB channel, you don't know if it is because of excesive exposure or due to saturation, which could be caused by using a small color space such as sRGB.
If the issue is saturation, reducing exposure is the wrong approach to solve the problem. Additionally, you might have saturation clipping of some colors that do not show as clipped channels in RGB; for instance, a saturated cyan is represented as a zero value in the red channel. Now, how do you know if a zero value in the red channel is because of a deep shadow or a clipped cyan?
-
Additionally, you might have saturation clipping of some colors that do not show as clipped channels in RGB; for instance, a saturated cyan is represented as a zero value in the red channel. Now, how do you know if a zero value in the red channel is because of a deep shadow or a clipped cyan?
But these saturations (I mean the ones making some RGB channel get clipped to zero after the colour profile conversion), are irrelevant in the histogram accuracy exposure.
The dangerous is the opposite case: a RAW channel is clipped but after the colour profile conversion (typ. linear combination of all three channels) that particular RGB value is not clipped in the JPEG. I think this is the main source of false negatives when using UniWB.
Regards.
-
ETTR may not be as important with cameras equipped with the low noise Sony Exmor, like the D800, but for my Canon and my MF backs is still rather important. In any case a raw histogram and raw highlight blinkies would not hurt to have as a setting for advanced users so I find it a bit frustrating that manufacturers are so clueless about this. I don't know of any camera with a properly designed raw histogram, out of the box.
We will probably never get it either, as sensors get less and less noisy it will be less and less important with ETTR and thus we'll care less about raw histograms, the exposure new standard will be like it was with slide film - if the preview looks good then the exposure is good (and that's been the thinking all along with the manufacturers).
I agree that with the Exmor sensors ETTR is less critical than with older designs. However, shot noise predominates in all but darkest shadows, and it is still important to maximize SNR by collecting as many photons as possible. The main advance with the Exmor is in reducing read noise, but shot noise is inherent in the light before it hits the sensor and the Exmor can not do anything about this.
Bill
-
But these saturations (I mean the ones making some RGB channel get clipped to zero after the colour profile conversion), are irrelevant in the histogram accuracy exposure.
The dangerous is the opposite case: a RAW channel is clipped but after the colour profile conversion (typ. linear combination of all three channels) that particular RGB value is not clipped in the JPEG. I think this is the main source of false negatives when using UniWB.
Regards.
100% Agree.
The point I was trying to make is that making exposure decision based on RGB histograms might be wrong. In any internet photography forum there are people complaining about blown out reds and often the proposed solution is to reduce exposure, which I consider wrong. The same issue with cyan goes completely unnoticed (the majority of images with a blue sky in them in sRGB have clipped cyan) and nobody would even think to solve an issue where a channel is zero (Red in the case of clipped cyan) by reducing exposure.
A RGB histogram, even with uniWB, is an unreliable tool to determine exposure
-
I agree that with the Exmor sensors ETTR is less critical than with older designs. However, shot noise predominates in all but darkest shadows, and it is still important to maximize SNR by collecting as many photons as possible. The main advance with the Exmor is in reducing read noise, but shot noise is inherent in the light before it hits the sensor and the Exmor can not do anything about this.
I fully agree with Bill, ETTR is about optimizing the amount of captured photons. All the rest is post-processing, scaling the channel responses in Linear gamma space to avoid irreversible data loss as much as possible. Post-processing with floating point number math is IMHO indispensable for that. For example RawTherapee does that, and that immediately opens up lots of possibilities.
Cheers,
Bart
-
Sorry Mike I thought by binning you meant the mixing of the G1 and G2 RAW channels. I wanted Histogrammar to have the possibility to go up to 1:1 zoom as Bill explains. You can produce histograms 65536 long.
I also kept all the 'bins' at every lower zoom level as power of two containers, so that the shape of the histogram never leads to confusion (in log mode some peaks can arise specially in the deep shadows though, but this is expected due to level quantization).
Guillermo,
Thanks for your reply. My notion of "binning" is simply the process of converting the analog (voltage) values as they are read from the sensor to digital values which are written to the raw file. I assume this occurs in the A/D converter whereby the total range of analog values are divided (binned) into equally spaced voltage intervals; the number of such intervals being determined by the bit depth of the converter. Thus, for example, a 14 bit pipeline will allow the recording of 16683 discrete intervals (or bins) each of which is identified by an unique digital integer. The histogram then graphs the total number of pixels (sensels) falling within each of those intervals.
If I correctly understand your Histogrammar (a program I have happily used for several years), you elect to expand the number of bins to 65535 (regardless of the bit depth of the raw file); which means that it will display the data from every fourth bin from a raw (14 bit) file containing 16683 such intervals.
Any corrections to my admittedly very basic understanding will be gratefully received.
-
Histogrammar only plots histograms from 16-bit TIFF image files. How can it plot RAW histograms then? thanks to DCRAW's -d and -D commands, that put undemosaiced data values from any RAW file as gray levels into a 16-bit monochrome TIFF following the camera CFA:
(http://www.guillermoluijk.com/tutorial/dcraw/hotpixel.gif)
Histogrammar's RAW mode simply interprets each gray level as R, G1, G2 or B according to its position in the grayscale image and plots the histogram. The -D command specifically just takes the RAW numbers and puts them in the TIFF file. The -d command allows to obtain the RAW data at EV intervals since it normalizes the RAW data to the 0..65535 range.
E.g. this is a Canon 5D RAW histogram showing how the camera saturates at 3692. The histogram runs up to 65535 but all levels beyond 3692 are empty:
(http://www.guillermoluijk.com/tutorial/satlevel/hist5d.gif)
Of course I'd find Rawdigger today much more user friendly.
-
BTW the RAW histogram + Auto ETTR story seems a lost battle. ETTR will loose all interest before camera makers decide to look at it:
Looks like Nikon is going to be providing an in-camera RAW histogram (http://nikonrumors.com/2015/01/08/nikon-to-announce-a-new-firmware-download-program-on-january-19th-including-several-improvements-for-the-d750-d810-d800-d800e-d610-and-d600-cameras.aspx/) by the end of this month via a firmware update to its recent DSLRs. I had to pinch myself and double check the date (not April 1st, phew) to make sure I was not dreaming.
-
Looks like Nikon is going to be providing an in-camera RAW histogram (http://nikonrumors.com/2015/01/08/nikon-to-announce-a-new-firmware-download-program-on-january-19th-including-several-improvements-for-the-d750-d810-d800-d800e-d610-and-d600-cameras.aspx/) by the end of this month via a firmware update to its recent DSLRs. I had to pinch myself and double check the date (not April 1st, phew) to make sure I was not dreaming.
That would be really great news, if one make includes RAW histograms others will follow. I wonder if it will be a log (EV stops) or gamma lifted histogram.
Regards
-
I wonder if it will be a log (EV stops) or gamma lifted histogram.
"...will announce a new “I AM Advancing” program for Nikon DSLR..." - subscription based, $10/mo, stop paying fees and access to camera's menu blocked ;D
-
"...will announce a new “I AM Advancing” program for Nikon DSLR..." - subscription based, $10/mo, stop paying fees and access to camera's menu blocked ;D
I couldn't find this in any of the links I checked. Most just say that the downloader is free and that you can get firmware updates for 3 years.
-
I couldn't find this in any of the links I checked. Most just say that the downloader is free and that you can get firmware updates for 3 years.
it was a joke... however subscription model shall find its way eventually, why is it that Adobe needs it and Nikon doesn't ?
-
it was a joke... however subscription model shall find its way eventually, why is it that Adobe needs it and Nikon doesn't ?
Don't blame Adobe. Blame the customers that support it. As for me, I'm still on Photoshop CS6.
-
Don't blame Adobe. Blame the customers that support it.
And don't blame the customers who are quite happy to pay for the product. Seems like many are quite happy to do so.
-
Don't blame Adobe. Blame the customers that support it. As for me, I'm still on Photoshop CS6.
I beat you, mine is CS2 :D
-
I wonder if it will be a log (EV stops) or gamma lifted histogram.
Regards
I would prefer a log (EV stops) histogram, but since this is still a rumor and we are allowed to dream, what about a custom function that lets you decide between the two options?
-
Don't blame Adobe. Blame the customers that support it. As for me, I'm still on Photoshop CS6.
I don't feel the need to blame anyone. I too am still using CS6, but don't doubt that eventually Adobe will offer some very significant(can't do without) improvement that will force me to the cloud. Thus far, not tempted.
As for camera makers, it's an interesting (and frightening) idea that, in future they may offer cameras whose feature set is intentionally missing desirable refinements. They could then offer subscription firmware updates to add features. It would provide a revenue stream from owners who cannot justify the cost of moving up to the next generation camera, but who would gladly pay a fraction of that cost to upgrade their current camera.
-
They could then offer subscription firmware updates to add features.
nope, subscription to keep your camera running - much better... pay or face a brick.
-
I don't feel the need to blame anyone. I too am still using CS6, but don't doubt that eventually Adobe will offer some very significant(can't do without) improvement that will force me to the cloud. Thus far, not tempted.
As for camera makers, it's an interesting (and frightening) idea that, in future they may offer cameras whose feature set is intentionally missing desirable refinements. They could then offer subscription firmware updates to add features. It would provide a revenue stream from owners who cannot justify the cost of moving up to the next generation camera, but who would gladly pay a fraction of that cost to upgrade their current camera.
I won't go so far as to say intentionally missing features, but since I think, with appropriate processing capability and onboard power, that software would be the logical area for improvement, that they could charge for upgrades. Of course, people will be 'rooting' their cameras to put 3rd party apps on them.
-
And don't blame the customers who are quite happy to pay for the product. Seems like many are quite happy to do so.
I agree that lots of customers are happy to pay a monthly subscription fee. But it is equally true that the blame that subscription based software is with us is because of sentence number 1.
As for me, I paid for Photoshop and the upgrades and was happy to do so, but when subscription 1st came out adobe wanted $29 a month while my upgrade from CS5 to CS6 was $175. Subscription based is more expensive and always will be. It is the whole reason that 5 and 6 year car loans are very popular. Get the individuals payments low enough and people will 1) overpay for stuff, and 2) buy stuff they really can't afford. Reminds of the whole extended warranty scam.
-
Something new?
http://www.dpreview.com/articles/8189925268/what-s-that-noise-shedding-some-light-on-the-sources-of-noise?utm_source=newsletter&utm_medium=email&utm_campaign=generic
-
Something new?
http://www.dpreview.com/articles/8189925268/what-s-that-noise-shedding-some-light-on-the-sources-of-noise?utm_source=newsletter&utm_medium=email&utm_campaign=generic
That is a nice article, but Emil Martinec's post (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html) is still the best source of which I am aware on this topic. The current article is less technical and may be more easily comprehended by those who are less technically oriented.
Bill
-
Emil's article is the Bible of digital camera noise!