Pages: 1 [2] 3 4 5   Go Down

Author Topic: Shooting to the right and raw conversion  (Read 45813 times)

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Shooting to the right and raw conversion
« Reply #20 on: June 26, 2007, 10:28:47 am »

Quote
No comments on the analysis?

Bill
[a href=\"index.php?act=findpost&pid=124962\"][{POST_SNAPBACK}][/a]

Sorry, been on the road (but that doesn't stop others from commenting).

Now that this trip is almost over (writing from lovely Ohare airport), I hope to spend some time looking into this a lot more, talking with the product manager for the new meter and doing many more tests. Got to get to the bottom of this.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

nma

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 312
Shooting to the right and raw conversion
« Reply #21 on: June 26, 2007, 02:31:38 pm »

This is  very interesting thread. There is some agreement that one should expose to the right, but not overdo it.  I think that is the problem.  The histogram is great tool. But as it is derived from the incamera jpeg, it is not very accurate. With the rgb histogram on the Canon 5D, I have a better idea of what is going on, but in some images there are some highlight values that have poor representation in the histogram. You can think of the histogram as the fraction of pixels with intensity between I and I+DI, where I is the intensity value (0-255). There can be a highlight "toe" that is not well defined in the histogram because the fraction of pixels with that value is low. It can be hard to see the toe under contrasty review conditions. It is only later, when we examine the raw image and set the white balance that we really understand if we got "the ideal" exposure. Those that espouse moderation know what they are talking about.  Thiis is an area where experience really counts.
Logged

hubell

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1135
Shooting to the right and raw conversion
« Reply #22 on: June 26, 2007, 02:55:15 pm »

Quote
This is  very interesting thread. There is some agreement that one should expose to the right, but not overdo it.  I think that is the problem.  The histogram is great tool. But as it is derived from the incamera jpeg, it is not very accurate. With the rgb histogram on the Canon 5D, I have a better idea of what is going on, but in some images there are some highlight values that have poor representation in the histogram. You can think of the histogram as the fraction of pixels with intensity between I and I+DI, where I is the intensity value (0-255). There can be a highlight "toe" that is not well defined in the histogram because the fraction of pixels with that value is low. It can be hard to see the toe under contrasty review conditions. It is only later, when we examine the raw image and set the white balance that we really understand if we got "the ideal" exposure. Those that espouse moderation know what they are talking about.  Thiis is an area where experience really counts.
[a href=\"index.php?act=findpost&pid=125008\"][{POST_SNAPBACK}][/a]

This is very consistent with my experience with a Hasselblad H3D-39. The histogram on the back does not show any evidence of highlight clipping in many images, but it is there when the file is opened in ACR or Flexcolor, and I really dislike the missing highlight detail. Much prefer black shadows to highlights with no detail. It seems that the highlight clipping display on the LCD is much more reliable in showing clipping at the far ends of the histogram and I am starting to rely on that as an important check on the accuracy of the histogram.

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #23 on: June 26, 2007, 03:13:30 pm »

Quote
You want to place as much data to the right so that you end up with the most data in the last stop of the tone curve (shadows). If you have a 12 bit file that can produce 6 stops, the first half of the data is contained in the first stop of exposure data (2048 levels). The last stop has only 64. See:

http://www.ppmag.com/reviews/200612_rodneycm.pdf
[a href=\"index.php?act=findpost&pid=124199\"][{POST_SNAPBACK}][/a]

This is an oft-quoted model of what is going on, but it is not the most accurate one.  The number of levels in a stop is not an issue with current cameras.  They have too much noise to be limited by posterization, except for a few cameras at their lowest ISO (Pentax K10D at ISO 100, for example), and then, just barely, and in the deepest shadows.

The signal-to-noise ratio is what matters, in the absence of any real posterization threat.  In the deepest shadows, dominated by read noises, the SNR doubles with each doubling of exposure (+1 EC).  In the midtone and highlight areas, SNR doubles with each quadrupling of exposure (+2 EC), and the in-between zones, in-between.  The benefit is strongest in the dark shadow areas.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #24 on: June 26, 2007, 03:21:52 pm »

Quote
Above unity gain of the camera (which varies from 800 -1600 with most DLSRs--see Roger Clark), if you are strapped for exposure by shutter speed or f/stop necessities, it does not help to raise the camera ISO any further than that of unity gain, and you can increase the exposure in the raw converter.
[a href=\"index.php?act=findpost&pid=124202\"][{POST_SNAPBACK}][/a]

The concept of unity gain, as proposed by Roger, is meaningless, IMO.  ADC units are arbitrary except in their ability to posterize.  He has no evidence for that conclusion; the camera he based it on has no real ISO 3200 with analog gain - it is just 1600 pushed, so of course there is no real noise benefit in using it.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #25 on: June 26, 2007, 03:31:46 pm »

Quote
This is  very interesting thread. There is some agreement that one should expose to the right, but not overdo it.  I think that is the problem.  The histogram is great tool. But as it is derived from the incamera jpeg, it is not very accurate. With the rgb histogram on the Canon 5D, I have a better idea of what is going on, but in some images there are some highlight values that have poor representation in the histogram.
[a href=\"index.php?act=findpost&pid=125008\"][{POST_SNAPBACK}][/a]

The camera histograms are derived from the JPEG preview, but you do have some control over their appearance via the camera settings. The contrast control, for example, applies an S curve to the data to lower the quarter tones and raise the three quarter tones. This should not affect the end-points of the histogram but does affect the shape towards the extremes. Therefore, many set the camera to low contrast to get a better view of the histograms.

If the camera permits the uploading of a custom tonal response curve, one can use this to calibrate the histogram so that clipping in the histogram correlates with clipping in the raw file. For example, if the histogram indicates clipping when there is none, one can upload a curve with roll off in the highlights. The RGB histograms reflect the status of the channels after white balance, but one can upload a custom WB to obtain an indication of the contents of the channels prior to WB.

Bill
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #26 on: June 26, 2007, 04:37:54 pm »

Quote
If you're "shooting to the right" in a natural lighting situation by using the camera histogram, you're probably (although not certainly) losing specular highlights -- the upper 0.05% of your image -- that help the scene come alive.  I prefer to underexpose significantly (a full stop) to capture these highlights (gaining shutter speed) and relying on post processing to correct exposure in a non-linear space.[a href=\"index.php?act=findpost&pid=124204\"][{POST_SNAPBACK}][/a]

Well, your standpoint does not contradict the shooting to the right idea, as long as you clarify what the right edge of your tonal levels is.  If you want specular highlights, then you can expose them as far to the right as possible.  "To the right" doesn't necessarily mean positive EC.  It just means pushing the tones you wish to preserve just short of clipping.  For black spraypaint on on a dark grey wall, ETTR might mean +3 EC.  For capturing detail in city lights at night, ETTR might mean -2 EC.  The underlying principle is to get the brightest tones you wish to record just short of clipping.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #27 on: June 26, 2007, 04:47:21 pm »

Quote
I think my question is misunderstood.  I personnaly always shoot to the right using RAW and do not underexpose.  However if I do happen to underexpose, it appears that I can recover the situation using the exposure compensation facilities available in the RAW converter.
If this is an acceptable practice then why promolgate the view that you should shoot to the right?[a href=\"index.php?act=findpost&pid=124223\"][{POST_SNAPBACK}][/a]

Acceptable and optimum are two different things.  If you expose one image a stop more to the right than another, the one exposed more to the right will have usable shadows a stop deeper in real world light, and the acceptable shadows will be good shadows.
« Last Edit: June 26, 2007, 06:28:40 pm by John Sheehy »
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Shooting to the right and raw conversion
« Reply #28 on: June 26, 2007, 04:48:42 pm »

Quote
The camera histograms are derived from the JPEG preview, but you do have some control over their appearance via the camera settings. The contrast control, for example, applies an S curve to the data to lower the quarter tones and raise the three quarter tones. This should not affect the end-points of the histogram but does affect the shape towards the extremes. Therefore, many set the camera to low contrast to get a better view of the histograms.

Good point! This deserves a lot more attention. It would be nice if the camera makers would just allow us to view a linear encoded Histogram (it will take some getting used to, it's all shoved to one side). Then you need to get the brightness on the LCD way down too.

Quote
Well, your standpoint does not contradict the shooting to the right idea, as long as you clarify what the right edge of your tonal levels is. If you want specular highlights, then you can expose them as far to the right as possible.

Exactly. We need to know how to nail the highlights.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #29 on: June 26, 2007, 05:01:44 pm »

Quote
If you shoot one stop down, you are losing one stop of dynamic range.[a href=\"index.php?act=findpost&pid=124245\"][{POST_SNAPBACK}][/a]

Actually, no DR is lost; it is simply shifted from the shadows to the highlights.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #30 on: June 26, 2007, 06:47:27 pm »

Quote
The concept of unity gain, as proposed by Roger, is meaningless, IMO.  ADC units are arbitrary except in their ability to posterize.  He has no evidence for that conclusion; the camera he based it on has no real ISO 3200 with analog gain - it is just 1600 pushed, so of course there is no real noise benefit in using it.
[a href=\"index.php?act=findpost&pid=125016\"][{POST_SNAPBACK}][/a]

Yes, John, we have been over this before. However, it does not make sense to quantify beyond 1 electron = 1 ADU. At that point you have completely quantified the number of electrons that have been captured--you have the actual count and that is all you need.

I have yet to hear your refutation of that point.

Bill
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #31 on: June 26, 2007, 07:08:10 pm »

Quote
Actually, no DR is lost; it is simply shifted from the shadows to the highlights.
[a href=\"index.php?act=findpost&pid=125035\"][{POST_SNAPBACK}][/a]

That is assuming that you still have data in all 12 bits of your ADC. If the upper bit is empty, you have lost potential DR. In that case, the maximal:minimal recorded signal drops from 4096:1 to 2048:1 with a 12 bit ADC. With decreased exposure, the noise floor for DR also increases, further limiting actual DR.

Bill
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #32 on: June 27, 2007, 08:54:41 am »

Quote
That is assuming that you still have data in all 12 bits of your ADC. If the upper bit is empty, you have lost potential DR. In that case, the maximal:minimal recorded signal drops from 4096:1 to 2048:1 with a 12 bit ADC. With decreased exposure, the noise floor for DR also increases, further limiting actual DR.
[a href=\"index.php?act=findpost&pid=125048\"][{POST_SNAPBACK}][/a]

You are talking about the DR of the capture; I tend to think in terms of the medium, as the capture is generally a wild card, as in the specular highlights that the other poster was concerned with.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #33 on: June 27, 2007, 10:26:08 am »

Quote
You are talking about the DR of the capture; I tend to think in terms of the medium, as the capture is generally a wild card, as in the specular highlights that the other poster was concerned with.
[a href=\"index.php?act=findpost&pid=125134\"][{POST_SNAPBACK}][/a]

Yes, in a thread about exposure to the right, are not we all talking about the capture and how to optimize the captured data with due consideration given to the limitations of the medium? Specular highlights and other features of the scene are not really wild cards, but are subject to the laws of physics and scientific analysis.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #34 on: June 27, 2007, 01:40:36 pm »

Quote
Yes, in a thread about exposure to the right, are not we all talking about the capture and how to optimize the captured data with due consideration given to the limitations of the medium?

Many times I have seen people imply that some choice that they make in exposure affects the DR of the camera, and that is what I meant to dispell.

Quote
Specular highlights and other features of the scene are not really wild cards, but are subject to the laws of physics and scientific analysis.
[a href=\"index.php?act=findpost&pid=125159\"][{POST_SNAPBACK}][/a]

They are wild because you can not accurately measure them in many situations;  you can only gamble.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #35 on: June 27, 2007, 03:23:44 pm »

Quote
Many times I have seen people imply that some choice that they make in exposure affects the DR of the camera, and that is what I meant to dispell.
They are wild because you can not accurately measure them in many situations;  you can only gamble.
[{POST_SNAPBACK}][/a]

In most cases it is not necessary to render specular highlights accurately, but merely place them above the level of diffuse white in the scene and then let them blow out at higher levels. In movies and slides, the specular highlights are often placed at 200% as discussed in this [a href=\"http://www.color.org/iccprofile.html]ICC Paper[/url]. In this case you would allow 1 stop of headroom as suggested early in this thread. For prints, probably a bit less headroom would be advisable.

Bill
« Last Edit: June 27, 2007, 09:43:44 pm by bjanes »
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #36 on: June 27, 2007, 07:56:35 pm »

Quote
In most cases it is not necessary to render specular highlights accurately, but merely place them above the level of diffuse white in the scene and then let them blow out at hither levels. In movies and slides, the specular highlights are often placed at 200% as discussed in this ICC Paper. In this case you would allow 1 stop of headroom as suggested early in this thread. For prints, probably a bit less headroom would be advisable.
[a href=\"index.php?act=findpost&pid=125221\"][{POST_SNAPBACK}][/a]

Nevertheless, it is still up to the individual how much detail they want from these small, brighter areas, and how much they are willing to gamble.  There are ways to compress DR locally in images if they want.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #37 on: June 27, 2007, 09:13:35 pm »

Quote
Yes, John, we have been over this before. However, it does not make sense to quantify beyond 1 electron = 1 ADU. At that point you have completely quantified the number of electrons that have been captured--you have the actual count and that is all you need.

I have yet to hear your refutation of that point.
[a href=\"index.php?act=findpost&pid=125046\"][{POST_SNAPBACK}][/a]

I'm pretty certain I have addressed this before.  Regardless, the fact is that cameras are *NOT* counting electrons, even if that is what we'd really want them to do.  The discreet electrons come packaged in a bundle of analog noise caused by reading, amplifying, transporting, (possibly amplifying again,) and digitizing the electron charge.  This extra read noise is *NOT* in units of electrons; it is analog until digitization.

With customized circuitry, Canon has been able to get the level of read noises at the highest ISO down to the equivalent of a few electrons (not a few discreet electrons!).  They have been able to do this at the *highest* amplification used in the cameras.  Nothing that Roger writes on his website addresses what may or may not happen with more and better amplification; he simply jumps to the conclusion that nothing is gained, and uses the fact that his 1Dmk2 has the same total read noise in electrons at ISO 3200 as it does at ISO 1600.  That is not any real support for his conclusion, because ISO 3200 *IS* ISO 1600 on that camera.  Had he used a Minolta K7, which uses real amplification at ISO 3200, he would have measured slightly less noise at ISO 3200, and if he had actually looked at the shadows, there would be slightly less line noise at 3200, and less chromatic noise in a RAW at 3200 than 1600 pushed to 3200.  I'd offer the 1Dmk3's ISO 3200 as additional support for my claim, but the fact that it is 14 bit may make you feel that the goal post for unity gain has moved (despite the fact that mk3 ISO 3200 quantized to 8 bits is still far less noisy than the mk2's ISO 3200).

I'm sure I have shown you this chart before, in previous refutations of the "unity gain" limit:



That is the total read noise, and the isolated horizontal and vertical line noises.
The total read noise, the yellow line, is scaled to 10% to fit in with the others, and the vertical axis is the read noise normalized to ISO 100 for all other ISOs, as standard deviation in ADUs (which can be considered arbitrary units of electrons).  The noises clearly show no sign of flatlining completely by 3200, as far as the trends up to 1600 are concerned, especially the line noises, which are far more visible than their statistical strength suggests.  Horizontal line noise *is*, without a doubt, the most troublesome aspect of high-ISO shadow areas in Canon cameras.

Roger has nothing to really support his unity gain hypothesis; he is simply applying the concept of one equals one, but these ones are really apples and oranges; one is discreet integer values, and the other is discreet multiples of a single value, with variance at a finer degree.  The ADC in these cameras can *NOT* count electrons.  They can only get so close to counting them, and by all appearances, with Canon's technology, the more you amplify the signal, the more you can reduce the inaccuracy, which is why it is illogical to declare that something as arbitrary as the ADU unit is a meaningful limit to practical amplification.  And the ADU truly *is* arbitrary when it is fine enough not to cause posterization of RAW data.  Only when it is coarse enough to cause posterization does the actual absolute meaning of the ADU have any value (the ability to posterize).
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #38 on: June 27, 2007, 09:17:00 pm »

Quote
We need to know how to nail the highlights.
[a href=\"index.php?act=findpost&pid=125034\"][{POST_SNAPBACK}][/a]

Even without direct support on the camera, it could conceivably be done now with a computer hooked up to the camera; a program could look for new RAW files on the card or in the computer, and display a RAW image and/or histogram on the screen.  Not for action shooting, of course!
« Last Edit: June 27, 2007, 09:20:31 pm by John Sheehy »
Logged

nma

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 312
Shooting to the right and raw conversion
« Reply #39 on: June 27, 2007, 11:00:23 pm »

Quote
The camera histograms are derived from the JPEG preview, but you do have some control over their appearance via the camera settings. The contrast control, for example, applies an S curve to the data to lower the quarter tones and raise the three quarter tones. This should not affect the end-points of the histogram but does affect the shape towards the extremes. Therefore, many set the camera to low contrast to get a better view of the histograms.

If the camera permits the uploading of a custom tonal response curve, one can use this to calibrate the histogram so that clipping in the histogram correlates with clipping in the raw file. For example, if the histogram indicates clipping when there is none, one can upload a curve with roll off in the highlights. The RGB histograms reflect the status of the channels after white balance, but one can upload a custom WB to obtain an indication of the contents of the channels prior to WB.

Bill
[a href=\"index.php?act=findpost&pid=125019\"][{POST_SNAPBACK}][/a]

Modifying the contrast seems like a good idea. But in the final analysis, the camera manufacturers have let us down. There are not enough bins in the histogram and it comes from the jpeg, not the raw. Because there are so few bins in the histogram, its shape is poorly defined. This matters most in the highlights.  In some case parts of the histrgram can be distorted becasue the bin-width is too large.  One experiment for someone with ambition and too much time is to measure the in-camera histogram and compare its shape  to the one in photoshop. It would be good to do this experiment for several lighting conditions
Logged
Pages: 1 [2] 3 4 5   Go Up