dynamic range and exposure

[victoraberdeen]

Jim,

Sorry to cause you such distress, however the article is just a rehash of standard transparency technique, and I am pleased you will gain from the technique. You can try this simply, on a sunny day pick a place with some good dark shadow area and then roll of the exposures. ( Good example - http://www.contaxg.com/document.php?id=10120 ). If the shadow is 60th f2 and the highlight 2000th  f16 begin shooting at a 15th and go as dark as you lens camera will go. Now take a look at the images and make your own assessment on the effect of the exposures. Repeat the exercise for different lighting like snow and water. I hope you will learn that there is only one correct exposure, for the subject your photographing. And the more subjects you include in the picture the bigger the compromise you make. Simple huh!

I have been on this site for a long time, have seen a lot of his work and think that Michael should know better! Maybe he needs to work for a good editor.

Victor  

[Ray]

As for your other example where the camera has more dynamic range than the scene, It seems that the "best" approach would be to overexpose the image, while being careful not to blow out the highlights. That way the majority of the pixels are up out of the noise. If I understand this correctly, more bits are used to represent the image data the farther to the right the data is in the histogram. This should be the case whether the histogram represents gamma-corrected data or linear data.

Jim,
Well this is precisely what Michael is recommending in his article and it certainly makes sense to me. But supposing the dynamic range of the scene is only 2 or 3 stops and the histogram shows the image to consist mainly of mid-tones, ie. the histogram is not close to either the left or right side, but slap in the middle. I presume in those circumstances noise is not a problem, yet the advice would still be to overexpose in order to achieve greater tonal subtlety. Right?

[Ray]

EC does not drop stuff off the left, but compresses it into the toe of the TRC.

There are so many acronyms in this thread I don't know whether I'm coming or going. I take it EC stands for exposure compensation, but not sure about TRC.

One thing I've noticed when converting RAW images in Breezebrowser (sorry! BB), is that exposure compensation of a seriously underexposed image (say 1.5 to 2 stops) at the time of conversion, results in an image with considerably less noise in the shadows than any levels adjustment in PS can achieve with a straight non-EC conversion. However[/i], the EC conversion has slightly MORE noise in the rest of the tonal range. There's a price to pay. Does this make sense?

[dlashier]

> You can read more here

Thanks Jim, thought I halfway understood this stuff but now I'm really confused  

:Victor
> First sample shows the sky going magenta in the top right corner!?

Looks like a channel blown. What processing/conversion? (Interesing photo btw)

> exposing to the left and see how the highlights have blown out!

Looks like good exposure to me. Sometimes you just have to choose and I think you chose right (whoops! meant 'think you chose correctly') in this case. BTW, this does show that you can sometimes blow highlights gracefully with digital.

> Finally a film example, so far I have failed to get this much  from a digital camera, I'll keep playing!

Oh I think digital will handle that, albeit possibly with a slightly different effect. Just watch out for artifacting in the highlights.

Victor, sounds like your experience mirrors mine in that I rarely have the luxury of deciding where to place the 'hump' but rather am pushing both ends and may have to choose which end to blow.

- DL

Here's what Thomas has to say on the issue...

Yes, the problem of only one or two of the channels being clipped is
sometimes a problem.  With raw capture it is not as bad as with JPEG
capture, since with raw what you care about is clipping in the native
camera color space, rather than in the working RGB space (since
conversion to working space happens after the raw converter's tone
adjustments).  Lots of colors clip in sRGB or even Adobe RGB that
don't clip in camera native space.

The ideal fix would be for the camera makers to change the
overexposure clipping warning to flash based on the native color
space channels, rather than on luminosity.   This is one of reasons
why Camera Raw's histogram is an overlay of the color channel
histograms, rather than a luminosity histogram.

In practice, this is usually not that big of an issue, since the
brightest objects in a scene tend to be nearly neutral (often white
clouds).  When in doubt, you can always bracket and check the
resulting color histograms to pick which frame to work with.

Thomas.

[atverd]

Actually this is well known fact, that digital doesn't handle traditional "underexposure" well and I'm using this "Right" approach with digital from the beginning. I even did some measures for this:
http://www.robgalbraith.com/ubbthre....pe=post

If you take a look at noise graph it's easy to see. Sorry for using stddev  as noise mark, dE would be better, but even this should be enough to show tendency. Don't ask me about shape of graph - I have no clue why it's like this

So, the article gives me theoretical base for my practical observations and this is very nice!

Thank you, Michael and Co for this!

Andrey

[d2frette]

Victor -

Quite a dream!  Forget 512 bit...Just get us to 32bit!  

At 32-bits:
- we'd have 2.1 billion unique levels in the 1st f/stop.
- we'd have 4.2 million unique levels in the 10th f/stop.

That'd be enough to work with, right?

- Dave

[BernardLanguillier]

Dear all,

I just read the very interesting article of Michael on the topic, and there are a few things I have a hard time understanding. Thanks for any explanation on the topic:

- I read recently (poster was Thom Hogan on dpreview I believe) that, on the D100 at least, the histogram that is displayed is computed based on the thumbnail image displayed on the screen of the camera, and not on the actual RAW image.

If this were correct, it would mean several things:

1. it is probably only a 8 bit projection of the 12 bit raw space,
2. the Y axis of the histogram only represents the amount of pixels of a given darkness in the thumbnail image. This means that the rightmost fifth of the histogram actually represents one fifth of the dynamic range, and therefore only a part of the brightmost stop.

The vertical lines draw on the screen are just misleading since they lead the user to think that that one fifth corresponds to one stop, but this is not the case since, as Michael points out, one stop should correspond to half of the screen.

I guess that whether I am right or wrong depends on what is actually displayed in the histogram.

At least, isn't there a need to confirm this on a per camera basis?

Thank you for your feedback on this.

Best regards,
Bernard

[jwarthman]

Victor,
You're new here. Perhaps, rather than repeatedly challenging Michael, you would be well served to "lurk" a bit. Read what Michael has to say, both in the message boards and in his articles. As you learn more about Michael's work and see his photographs, you may develop some respect. If not, well, there are plenty of other destinations on the web where you might find things more to your liking. As it is, your tone is disrespectful. While you had some interesting questions, it's Michael's option to debate these issues with you - or not.

You say the article "falls because it is neither; an expert assessment of hew the sensor device works or, an advice to photographers on how to get the results they desire." On your second point, I disagree.  Speaking for myself, I enjoyed the article in question, and feel I learned some things about histograms that hadn't occured to me before. I will adjust my shooting accordingly. I think my response to the article was just as Michael would have hoped for many of us.

As for your first point, if you're looking for expert assessments of how the CMOS works, why berate an article that is intended to do something different? Why not look elsewhere - or offer to do your own research and post an article with the sort of information you're interested in?

Sorry for the rant - I simply felt your last posting was unfair to Michael.

Best Regards,

Jim

[jwarthman]

Ray,
Yes, my understanding, based on Michael's article and the follow-up discussion here, is that image data to the right side of the histogram is of higher quality (less noise, defined by more bits) than image data to the left. As you say, we ought to get better tonality from a low-dynamic range image if we shift it in-camera to the right, then correct later in PS.

Enjoy!

-- Jim

[dlashier]

> In brief, it is due to the extra accuracy allowed by the extra four significant bits used in 16-bit editing.

Ok, I see this. It depends upon at what stage you do the EC. My assumption was that EC would be done on the linear data but I just did a quick experiment and that doesn't seem to be the case, with C1 anyway. -EC does not drop stuff off the left, but compresses it into the toe of the TRC.

But I would still make the point that gradient resolution usually only becomes an issue when you raise shadows and in a compressed image shot to more or less center the histo, there's nothing in the shadows to be raised. More than likely you're going to push stuff down by setting BP or increasing contrast, and as long as you do this in the raw converter (or convert to 16 bits and do it in PS) you're not going anywhere close to running short of values.

Do I really need floating point to balance my checkbook? Maybe BG, but not me  

- DL

[BJL]

Thanks Jim for all those references on gamma; this seems to be another subject that many of us need to understand better than we do in order to get the best looking images (sorry, I mean "to optimize our digital workflow").

Perhaps it would be nice for this site to have an article on Understanding colour spaces and gamma, but then again, perhaps one of the above references already does the job; I will start reading.

[AJSJones]

Michael,

As digital photography evolves, I hope we see more incremental development of tools to improve the acquisition process.  Thomas's suggestion to have camera manufacturers have blown pixels flash in the color of the channel that has clipped is an example of this and I hope he is heard by folks like Canon.  A couple of thoughts from my recent noodling  :

Since we are concerned about tonal gradations and smoothness (in addition to the relentless quest for resolution  :p  ) it seems that an alternative (i.e. selectable by a button when the histogram is displayed) histogram display which is linear, rather than logarithmic, might be useful. This would facilitate assessment of whether the areas of the image where that might be critical do indeed have enough "levels" to achieve the desired image.  In this space, one would be aiming for the centered histogram (again!).

A few posts have mentioned the possibility of a "live" histogram in the viewfinder (based on a small - few enough pixels to be processed on the fly)  image sensor in the pentaprism somewhere, as being something that would be helpful - another incremental aid.

Also, now the LCD can be zoomed somewhat, it would be nice to be able to see clipped pixels in the zoomed version.  I often know that to get the shot I want, I will accept "blown" highlights - they may actually be "blown" in reality - specular highlights for example.  So some clipping in the histogram will be expected, but it's hard to know that only those are blown, and not some others for which we want to retain some levels information.  Examination of a zoomed section for blown pixels would be another incremental help.  This would be evn more important if we start to adopt the suggestion that started this thread, of moving the histogram to the right.

None of these is real urgent, but they might be considered analogous to the increments in automotive technology such as ABS, airbags and satellite navigation systems - all incrementally beneficial, do-able as technology moves forward and , not least of all, sales tools as incentives for manufacturers and all that implies!

Just a little ramble!

Andy

[victoraberdeen]

Thank you Jim

So the what now that seems to be missing is the ability of the sensor to represent low light levels. Rather than over exposing, would it not be better to have a sensor that has a better range! If I may dream for 18 months, maybe we will get a full color sensor that has 512 or 1024bits and s/n problems fixed.

Why full color, go and understand how film works then you wont ask the question. There is a simple rule to remember capture the real color info from the lens, not what the software thinks it should be. If you think Bayer is OK, then good for you!

[victoraberdeen]

Do tell me what improvement you would prefer?

I was not looking for 500khz but covering a greater range than is required makes good quality sense. Most modern technology is built to the average need, this is not OK for serious photography. Some how there are boundaries to define what is acceptable level of quality in the representation of changes in tone (regardless of illumination). I hope we see sensor makers delivering in excess so we can produce pictures that represent the subjects we photograph. However the assessment of tone, like lens quality is perceptive making it difficult to decide how much is needed tonal definition is needed.  

On a sunny 5500 kelvin day the contrast can be in excess of 5000:1. Today the compromise gets me a max of 500:1 (if I'm lucky) there is no reason we cannot capture  closer to the real detail in future. Then we can photograph the muscle contours on a black horse against white sand, and see the grains! Today you have to choose which you want.

I'll keep dreaming  

[dlashier]

> Most readers of this board are sophisticated enough to know that prints are done in 8 bit mode. The point that you appear to overlook with the comment is with regard to the advantages of working in a larger bit space prior to printing. That's why an understanding of this issue is worthwhile. (The fact that 6 or 7 ink dithering simulates a wider space is another issue).

Michael, I agree that working in higher bit space is desirable prior to printing, but my point was that in most cases of compressed light where this technique is applicable that the first thing you're going to do (probably in the raw converter) is to -EC to get midtones back in line, then most likely set BP and WP to expand tonality. In the process you lose back all the gains in bit depth (and more) from over-exposing. If you don't lower the shadows back down then gains are preserved but in that case I would argue that the higher exposure is the 'proper' exposure to begin with, so you're really not applying this technique.

That's not to say that I don't see merit in the technique in special situations, but I'd say those situations are the exception rather than the rule for most photographers. And the gains are in reduced shadow noise rather than increased dynamic range.

- DL

[jwarthman]

Ken,
It's right here:  http://www.luminous-landscape.com/tutorial...ose-right.shtml.

Enjoy!

-- Jim

[BJL]

With subjects of modest contrast range, exposing to the right is probably not of much practical value:

on one hand,
it will give more levels at each subject brightness level, and these extra gradations will then be preserved when one does compensating EC downwards in 16-bit mode,

but on the other hand,
all the important brightness levels in a low contrast subject are high enough to get plenty of levels of gradation when one does a normal "18% gray" metering, so the extra effort proabbly does not pay off in any visible improvement in final image quality.


I would suggest as a rule of thumb that the extra effort of pushing the histogram to the right is worthwhile in roughly the same cases as when careful spot metering of highlights and shadows are worth it. Perhaps comparing these two approaches in the field is preferable than endless attempts to anwer the question theoretically.

[BJL]

"on one hand, it will give more levels at each subject brightness level, and these extra gradations will then be preserved when one does compensating EC downwards in 16-bit mode,"

Please explain how the extra gradations will be preserved when you do compensating EC downwards. There must be something I'm missing here.

OK Don, here is my reasoning, let me know what you think.

In brief, it is due to the extra accuracy allowed by the extra four significant bits used in 16-bit editing.

At great length,

a) Suppose that a one stop subject brightness range down in the shadows is placed by traditional exposure metering at -8 stops (8 below the maximum brightness handled by the A/D converter) and so 12-bit D/A conversion give you only eight levels over that one stop.

 Increasing the exposure by two stops puts it at -6 and gives you 32 levels over the same one stop range of subject brightness.

c) Converting to 16 bit linear increases the number of numerical levels 16-fold, so those 32 values are scattered over a range of 512 (15 as yet unused levels have been added in between each level output by the D/A converter.)

d) If you then correct downwards by the same two stops, dividing each of these value by four, the range is compressed to 128 levels; still a gap of four between each recorded level: no distinctions of level are lost. In other words, each full stop of downward EC discards one bit at the least significant end of the 16-bit number, but since originally the last four bits were meaningless zeros anyway, you can go down four stops before arithmetic round-off loses any read information from the A/D converter. EC by other than whole stop amounts will make the numbers more complicated, but the rounding error from the 16-bit arithmetic will not be as much as the A/D discretisation error until the drop is four stops (a factor of 16) or more.

e) Gamma compression increases the number of levels in each f-stop range in the shadows, so the situation is even better if EC is applied after gamma.


Another point: as Don has mentioned in a different way, if you simply EC back down and do a straight print, these deep shadows will be off the bottom of the scale of any print anyway; the only point to making this effort to accurately record these shadow levels is to keep them somewhat above their natural level, by contrast reduction or whatever. So if you are wanting to print those shadow details, they should not in the end be EC'd all the way back to where they came from.


(I sometimes teach numerical computing, so I hope I have got at least this stuff about rounding, bits and errors right, but I could more easily be wrong about the many other aspects of this situation, and in particular I make no claim to any expertise about the ultimate aesthetic value of the different methods.)

[Ian Lyons]

Michael's interesting article on dynamic range and exposure with dSLRs does not seem to me to go far enough. I have noticed that even though my exposure is accurate enough so that the histogram is similar to those pictured in the article, often one of the color channels is blown out.

I think with digital ACCURATE is the wrong term: try IDEAL!

Basically the ideal exposure is as Michael describes: get your histogram as close to the right side as possible but not so close as to cause the over exposure indicator to flash. The ideal exposure ensures that you have maximum number of levels describing your image without loosing important detail in the highlights. The closer you get to this ideal then the more of those levels are being used to describe your shadows. If you underexpose an image to the extent that the shadows block, which is often what folk do to protect their highlights; then you will need to open them again to ensure the final image is as you require. The problem with this approach is that we only have 128 levels available to the shadows. You start pulling curves, etc to open the shadows and you'll get posterisation, etc.

We need to get away from the concepts of exposure that have served us well with film. The CCD/CMOS isn't film and does not react like film in the highlight shadow regions. Exposure on film tends to roll-off smoothly in the shadows and highlights. With digital the capture is linear and there is no roll-off.  Unfortunately (as Thomas has indicated) the behavior of these sensors isn't perfect and we can (often do) get one or two channels going into saturation (blowing). This was a major problem with the Canon D30 and the current version of Adobe Camera Raw simply can't handle such images. Thomas knows my thoughts on this and I know his.

Its worth noting that I've not yet seen a "linear" raw image from a Canon EOS D30/60 or 10D that went full-scale, close but not all the way. This means that even though existing conversion apps might not meet your needs. So far as the raw conversion apps go I am confident that things will improve, but in the meantime I really do urge folk to look at this technique. Remember that you will likely still have the CRW file for a long time. It really is your equal of the negative; don't trash it. Even if the current crop of conversion apps can't handle the blown highlights future apps will. However, NOTHING will ever get you back the lost shadow detail.

You can see in the example shown on the following page how much info still exists in the highlights of an image that current raw convertors say is blown. This technique is virtually redundant now but should still give help you better understanding of just how mcuh info exists in yor highlights.

http://www.computer-darkroom.com/d30-profiling/d30_1.htm