Pages: 1 ... 8 9 [10] 11   Go Down

Author Topic: “Expose to the Right” & relation to ISO Invariance  (Read 93951 times)

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #180 on: June 24, 2015, 10:55:57 am »

Here's a screen capture of the entire image also showing the somewhat radical settings used in Develop. Clipping indicators are on in Histogram, no clipping. RGB value over brightest area of dogs head reads 98%.
What's interesting (to me) is that while the 'exposure' was boosted 3 stops via ISO, note that the Exposure slider is set to -1.55 stops which as I pointed out earlier is just about the limit of the ETTR testing on this camera done in the past. I can't explain why there is this disconnect between the +3 ISO and -1.5 Exposure slider among the other sliders as set.



The lighting in this scene is highly directional, and I question of the incident light reading gave the best exposure for the highlights of the dog's head, which are at 98% in your adjusted rendering. One could get different readings according to the direction in which the integrating sphere of the incident light meter is pointed. A proper exposure at ISO 100 would have these highlights just short of clipping in the raw file. It would be of interest to look at the raw file in Rawdigger to determine if the exposure was really optimum. It would be helpful if you would post the raw files so we could draw our own conclusions. If the highlights in the ISO 100 exposure are just short of clipping, they would be severely clipped if one used the same exposure (f/stop, shutter speed) at ISO 800. Highlight recovery could reduce clipping in the rendered image, but 3 stops of recovery is doubtful.

Bill
Logged

jrsforums

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1288
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #181 on: June 24, 2015, 02:03:55 pm »

The lighting in this scene is highly directional, and I question of the incident light reading gave the best exposure for the highlights of the dog's head, which are at 98% in your adjusted rendering. One could get different readings according to the direction in which the integrating sphere of the incident light meter is pointed. A proper exposure at ISO 100 would have these highlights just short of clipping in the raw file. It would be of interest to look at the raw file in Rawdigger to determine if the exposure was really optimum. It would be helpful if you would post the raw files so we could draw our own conclusions. If the highlights in the ISO 100 exposure are just short of clipping, they would be severely clipped if one used the same exposure (f/stop, shutter speed) at ISO 800. Highlight recovery could reduce clipping in the rendered image, but 3 stops of recovery is doubtful.

Bill

I have found that the optimal exposure (no clipped highlights) can be obtained by spot metering the brightest significant highlight and adding 3 to 3 1/3 stops.

This will normally result in LR having the brightest highlight in the mid to low 80%.  Anything above that, in the current PV, will have clipped and recovered channel(s). (This is very well explain in the George Jardin article I posted earlier in this thread)

This was arrived at using RawDigger for tests.  As I remember, Eric Chan uses 3 1/3 for his 5D3.
Logged
John

Dave Ellis

  • Full Member
  • ***
  • Offline Offline
  • Posts: 103
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #182 on: June 24, 2015, 03:25:32 pm »

Jack and Bart, thanks for your explanations of clipping level. All is becoming clear !

Dave
Logged

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #183 on: June 24, 2015, 03:33:55 pm »

I have found that the optimal exposure (no clipped highlights) can be obtained by spot metering the brightest significant highlight and adding 3 to 3 1/3 stops.
...
This was arrived at using RawDigger for tests.  As I remember, Eric Chan uses 3 1/3 for his 5D3.

Keep in mind that this value is camera specific because Sensitivity (ISO) in its current incarnation is unrelated to raw data.

Jack
Logged

jrsforums

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1288
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #184 on: June 24, 2015, 04:11:56 pm »

Keep in mind that this value is camera specific because Sensitivity (ISO) in its current incarnation is unrelated to raw data.

Jack

I am aware and agree.  That's why I mentioned the 5d3.  Also, many cameras do not have spot meters, which means they would need a different approach to determine the exposure for the significant highlight.

EDIT: Did you mean camera model specific or individual camera specific?  also,  not sure what you mean by "unrelated to raw data"
« Last Edit: June 24, 2015, 04:20:01 pm by jrsforums »
Logged
John

jrsforums

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1288
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #185 on: June 24, 2015, 04:16:35 pm »

I don't really understand what you are trying to convey.  We already knew why Andrew's image's are the way they are.  I'm just saying that you are not correct to say one was a higher exposure than the other.  They were the same exposure and the noise (actually SNR) difference is the variable that has been isolated.

I agree that I did not make clear what I was trying to say.  We are generally in agreement.
Logged
John

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #186 on: June 24, 2015, 04:21:20 pm »

I have found that the optimal exposure (no clipped highlights) can be obtained by spot metering the brightest significant highlight and adding 3 to 3 1/3 stops.

This will normally result in LR having the brightest highlight in the mid to low 80%.  Anything above that, in the current PV, will have clipped and recovered channel(s). (This is very well explain in the George Jardin article I posted earlier in this thread)

This was arrived at using RawDigger for tests.  As I remember, Eric Chan uses 3 1/3 for his 5D3.

Agreed, I can confirm that I also measured 3 1/3rd for my 1Ds Mark III, unless there is a lot of noise (e.g. very high operating temp), then there may be a very tiny bit, a few pixels, of the shot-noise tail getting clipped. Also extremely saturated (flower) colors may clip one channel of the Raw data. So bracketing is still useful for that last bit of capture quality.

Cheers,
Bart
« Last Edit: June 24, 2015, 04:24:22 pm by BartvanderWolf »
Logged
== If you do what you did, you'll get what you got. ==

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #187 on: June 24, 2015, 05:22:54 pm »

EDIT: Did you mean camera model specific or individual camera specific?  also,  not sure what you mean by "unrelated to raw data"

Camera model, and in fact it seems that individual manufacturers are fairly consistent across their own camera classes of the same generation.

As to raw data, unfortunately the latest standards that regulate metering and ISO do not concern themselves with raw data.  Madness, I know.  This is the formula that ties them together (from wikipedia)



For a given spot luminance (L) from the scene the (reflected light) meter in the camera will serve up values for f-number (N) and exposure time (t) that satisfy the equation.  K is a manufacturer specific constant (apparently 12.5 for CaNikon) and S is, you guessed it, ISO.  But as we know the new ISO non-standard (ISO 12232:2006) allows camera manufacturers to report ISOs that they individually believe will produce pleasing brightness in the OOC jpeg.  And they nowadays mostly take advantage of this possibility, reported as REI in the exif - check it out in one of your files. After a whole bunch of subjective, highly non linear transformations. And they call it a 'standard'.  Ugh.  That's why DxO is forced to determine their own 'Measured ISOs' (Ssat), which instead are related to the raw data linearly: Hsat = 78 / Ssat.

In other words, the spot meter may provide perfectly calibrated Exposure values for 'middle gray' at ISO 100 - but we have no idea where the manufacturer will place it in the raw data.  We only know that that manufacturer will be pleased by its camera's OOC image.  Enter RawDigger :(

Jack
« Last Edit: June 24, 2015, 05:45:14 pm by Jack Hogan »
Logged

jrsforums

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1288
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #188 on: June 24, 2015, 05:55:44 pm »

Camera model, and in fact it seems that individual manufacturers are fairly consistent across their own camera classes of the same generation.

As to raw data, unfortunately the latest standards that regulate metering and ISO do not concern themselves with raw data.  Madness, I know.  This is the formula that ties them together (from wikipedia)



For a given spot luminance (L) from the scene the (reflected light) spot meter in the camera will serve up values for f-number (N) and exposure time (t) that satisfy the equation.  K is a manufacturer specific constant (apparently 12.5 for CaNikon) and S is, you guessed it, ISO.  But as we know the new ISO non-standard (ISO 12232:2006) allows camera manufacturers to report ISOs that they individually believe will produce pleasing brightness in the OOC jpeg.  And they nowadays mostly take advantage of this possibility, reported as REI in the exif - check it out in one of your files. After a whole bunch of subjective, highly non linear transformations. And they call it a 'standard'.  Ugh.  That's why DxO is forced to determine their own 'Measured ISOs' (Ssat), which instead are[/b ]related to the raw data linearly: Hsat = 78 / Ssat.

In other words, the spot meter may provide perfectly calibrated Exposure values for 'middle gray' at ISO 100 - but we have no idea where the manufacturer will place it in the raw data.  We only know that that manufacturer will be pleased by its camera's OOC image :(

Jack

When I talk about a spot meter, I am talking about one in the camera, not a handheld one.

I then "calibrate" the camera,  using RAwDigger to determine how much +EC to get the significant highlight "middle grey" to be just below clipping.

If you were talking about a handheld meter, I agree....you have a problem.  The meter calibration will not necessarily be calibrated to the metering of the camera.  Chuck Gardner, in. One of his tutorials, covered how to get the meter and camera in sync.  http://super.nova.org/DPR/SpotmeterCompensation/
Logged
John

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #189 on: June 24, 2015, 06:58:54 pm »

Camera model, and in fact it seems that individual manufacturers are fairly consistent across their own camera classes of the same generation.

As to raw data, unfortunately the latest standards that regulate metering and ISO do not concern themselves with raw data.  Madness, I know.  This is the formula that ties them together (from wikipedia)



For a given spot luminance (L) from the scene the (reflected light) meter in the camera will serve up values for f-number (N) and exposure time (t) that satisfy the equation.  K is a manufacturer specific constant (apparently 12.5 for CaNikon) and S is, you guessed it, ISO.  But as we know the new ISO non-standard (ISO 12232:2006) allows camera manufacturers to report ISOs that they individually believe will produce pleasing brightness in the OOC jpeg.  And they nowadays mostly take advantage of this possibility, reported as REI in the exif - check it out in one of your files. After a whole bunch of subjective, highly non linear transformations. And they call it a 'standard'.  Ugh.  That's why DxO is forced to determine their own 'Measured ISOs' (Ssat), which instead are related to the raw data linearly: Hsat = 78 / Ssat.

In other words, the spot meter may provide perfectly calibrated Exposure values for 'middle gray' at ISO 100 - but we have no idea where the manufacturer will place it in the raw data.  We only know that that manufacturer will be pleased by its camera's OOC image.  Enter RawDigger :(

Jack

It is not as complicated as it would first appear. Japanese camera manufactures are required to use the SOS or REI calibration for the sensor. Since REI (recommended exposure index) allows an arbitrary value, the camera makers can use the old saturation standard which allows 0.5 EV of highlight headroom and places middle gray at 12.7% saturation. Things have not changed that much since Thom Hogan discussed the subject of gray cards in 2003. Doug Kerr discusses exposure meter calibration and points out that Canon uses ISO2721 with a k value of 12.5 so that the camera meter reading will agree with a hand held meter using a k of 12.5 as employed by Sekonic as well as Nikon. Minolta, Pentax, and Kenko (Wikipedia) use a k of 14 (different by about 1/6 EV). According to Doug's Canon source, Canon rates its sensors such that an exposure according to the meter reading will result in a saturation of 17.3%. They do not allow the 0.5 EV cushion for the highlights.

My Nikon D800e places the exposure of a uniformly illuminated surface (white or gray card) exposed according to the built in meter at 14% saturation as measured by Rawdigger. This is close to the saturation based value of 12.7% and is 2.84 stops below clipping.

Bill
« Last Edit: June 24, 2015, 07:04:45 pm by bjanes »
Logged

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #190 on: June 25, 2015, 09:05:30 am »

When I talk about a spot meter, I am talking about one in the camera, not a handheld one.

So am I :)

It is not as complicated as it would first appear. Japanese camera manufactures are required to use the SOS or REI calibration for the sensor. Since REI (recommended exposure index) allows an arbitrary value, the camera makers can use the old saturation standard which allows 0.5 EV of highlight headroom and places middle gray at 12.7% saturation. Things have not changed that much since Thom Hogan discussed the subject of gray cards in 2003. Doug Kerr discusses exposure meter calibration and points out that Canon uses ISO2721 with a k value of 12.5 so that the camera meter reading will agree with a hand held meter using a k of 12.5 as employed by Sekonic as well as Nikon. Minolta, Pentax, and Kenko (Wikipedia) use a k of 14 (different by about 1/6 EV). According to Doug's Canon source, Canon rates its sensors such that an exposure according to the meter reading will result in a saturation of 17.3%. They do not allow the 0.5 EV cushion for the highlights.

My Nikon D800e places the exposure of a uniformly illuminated surface (white or gray card) exposed according to the built in meter at 14% saturation as measured by Rawdigger. This is close to the saturation based value of 12.7% and is 2.84 stops below clipping.

Bill

Hi Bill,

If it only were so simple ;-)  Unfortunately the 2006 incarnation of the standard for ISO made things worse than they were originally.

REI is arbitrary and SOS is based on the rendered OOC sRGB jpeg, not raw data.  Thom's article is woefully inaccurate and Kerr's out of date.

If you have any other Nikon camera from an earlier or later generation, test them out.  Both my D90 and D610 put spot metered middle gray below 10% of saturation in the raw data and, with 14 stops of eDR in the D610, I am glad about that.

Jack

Logged

spidermike

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 535
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #191 on: June 25, 2015, 09:44:47 am »

How much effect does that have on comparing cameras? For example, if camera A is touted as great for low light because it is usable at ISO 12,800 could this be because it is actually equivalent to camera B at 6,400?
Logged

jrsforums

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1288
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #192 on: June 25, 2015, 10:03:55 am »

So am I :)

OK.....so then I am confused.  How does this affect "calibrating" the camera...using RawDigger....to know the EC to add to spot metered highlight to get optimized exposure....I.e. ETTR?
Logged
John

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #193 on: June 25, 2015, 10:45:23 am »

OK.....so then I am confused.  How does this affect "calibrating" the camera...using RawDigger....to know the EC to add to spot metered highlight to get optimized exposure....I.e. ETTR?

This excellent article by the Rawdigger staff explains the process in some detail. The important steps are to determine the clipping point of the green channels and the sensor saturation according to the camera light meter reading. To complete the process, note the relationship of raw clipping to the camera histogram and blinkies. Once understood, the process is not that complicated.

One fine point that Iliah points out is that clipping is somewhat of a gradual process. It begins when the right tail of the bell shaped histogram (shot noise distribution) begins to clip and is complete when the left tail reaches clipping. To include 95% of the data, one can add two standard deviiations to the mean.

Bill
« Last Edit: June 25, 2015, 10:54:46 am by bjanes »
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #194 on: June 25, 2015, 11:09:16 am »



One fine point that Iliah points out is that clipping is somewhat of a gradual process. It begins when the right tail of the bell shaped histogram (shot noise distribution) begins to clip and is complete when the left tail reaches clipping. To include 95% of the data, one can add two standard deviiations to the mean.


That would be true if the shot noise were Gaussian. As per my previous discussion with Erik on this thread, I usually elide the fact that it's actually Poisson. I haven't run the numbers, and I'm sure the difference is small near fullscale where you're doing this test, but, since I brought it up before, I thought I'd point it out as possibly being of academic interest.

The difference is not small near zero, and I would like to publicly thank Jack Hogan for pointing that out and for suggesting fixes to some of my Matlab code that fixed some earlier errors.

Jim
« Last Edit: June 25, 2015, 11:53:04 am by Jim Kasson »
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #195 on: June 25, 2015, 11:41:50 am »

How much effect does that have on comparing cameras? For example, if camera A is touted as great for low light because it is usable at ISO 12,800 could this be because it is actually equivalent to camera B at 6,400?

Hi,

It's anybody's guess what that means, until one calibrates for actual Raw file data levels.

Really, it's very simple, it can also be done with Fast RAW Viewer (FRV) automatically if one doesn't have RawDigger or similar. With FRV, just set the preferences to "ETTR style Autoexposure (shift histogram to the right)", and set the "Saturate up to" option to 0.0% pixels. Now shoot a uniform surface (well lit gray card, or as I do shoot through a piece of opal glass or white translucent perspex flush with the filter threads of the lens) with the camera on Aperture priority metering, and set that aperture to something like f/8 (or narrower).

FRV will tell you after Auto Exposure (Shift A), how many EVs the Raw exposure levels are removed from clipping the Raw data. To reduce the Photon shot-noise tail from clipping, you can also use an exposure correction (e.g. +2EV) on the Aperture priority, and add that to the remaining over exposure latitude.

I have a series of Flat-frames for several of my lenses on my EOS 1Ds Mark III, and I've exposed them at +2 1/3rd EV through a sheet of opal glass. At ISO 100 FRV gives me the following remaining latitudes; +0.82 EV, +1.08 EV, +1.09 EV, +1.02 EV, +1.02 EV, +0.98 EV, etc.

Depending on the camera used, at higher ISOs you may get other values, just try it for your camera.

The variation is due to random noise, slight variations in exposure time due to daylight changing after metering and before exposing, and irregularities in aperture blades and shutter curtains closing to the same opening. But on average it shows a 1 stop latitude in addition to the 2 1/3rd EV I pushed the exposure over the measured exposure level. So on average a total of 3 1/3rd EV above medium gray measurement. That was already what Rawdigger had told me, but now it's calculated for me.

Cheers,
Bart
« Last Edit: June 25, 2015, 12:58:28 pm by BartvanderWolf »
Logged
== If you do what you did, you'll get what you got. ==

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #196 on: June 25, 2015, 12:06:13 pm »

That would be true if the shot noise were Gaussian. As per my previous discussion with Erik on this thread, I usually elide the fact that it's actually Poisson. I haven't run the numbers, and I'm sure the difference is small near fullscale where you're doing this test, but, since I brought it up before, I thought I'd point it out as possibly being of academic interest.

The difference is not small near zero, and I would like to publicly thank Jack Hogan for pointing that out and for suggesting fixes to some of my Matlab code that fixed some earlier errors.

Jim,

That is true, but with a 14 bit file at highlight clipping the λ is well over 1000, and the normal distribution is a good approximation.

Wikipedia:

"For sufficiently large values of λ, (say λ>1000), the normal distribution with mean λ and variance λ (standard deviation sqrt[λ]) is an excellent approximation to the Poisson distribution. If λ is greater than about 10, then the normal distribution is a good approximation if an appropriate continuity correction is performed, i.e., P(X ≤ x), where x is a non-negative integer, is replaced by P(X ≤ x + 0.5).
FPoisson(x;λ) ~Fnormal(x;λ,σ2=λ).

Another fine point is that the normal distribution is continuous whereas the ADUs of the raw file are discrete integers. For low values of λ, one can apply the continuity correction as above. For read noise with current Sony sensors, λ is less than 10, but I don't know how much differences this would make for practical work.

Bill
« Last Edit: June 25, 2015, 12:24:30 pm by bjanes »
Logged

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #197 on: June 25, 2015, 12:53:04 pm »

How much effect does that have on comparing cameras? For example, if camera A is touted as great for low light because it is usable at ISO 12,800 could this be because it is actually equivalent to camera B at 6,400?

Indeed, hence the value of DxO's Measured ISOs and related jabs at, say, Olympus :-)
Logged

Jack Hogan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 798
    • Hikes -more than strolls- with my dog
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #198 on: June 25, 2015, 01:06:58 pm »

Jim,

That is true, but with a 14 bit file at highlight clipping the λ is well over 1000, and the normal distribution is a good approximation.

Wikipedia:

"For sufficiently large values of λ, (say λ>1000), the normal distribution with mean λ and variance λ (standard deviation sqrt[λ]) is an excellent approximation to the Poisson distribution. If λ is greater than about 10, then the normal distribution is a good approximation if an appropriate continuity correction is performed, i.e., P(X ≤ x), where x is a non-negative integer, is replaced by P(X ≤ x + 0.5).
FPoisson(x;λ) ~Fnormal(x;λ,σ2=λ).

Another fine point is that the normal distribution is continuous whereas the ADUs of the raw file are discrete integers. For low values of λ, one can apply the continuity correction as above. For read noise with current Sony sensors, λ is less than 10, but I don't know how much differences this would make for practical work.

Bill

Well put, Bill.  Imo it makes a difference when wishing to be precise in the deep shadows with today's sub 2e- read noise cameras at base ISO (e.g. the new crop of Exmors coming on line now).  Aside from a different looking histogram it makes a big difference with small signals because poisson has no negative values, while gauss does.  It's amazing how precisely physics fits our models when we use the right formulas.

Jack
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: “Expose to the Right” & relation to ISO Invariance
« Reply #199 on: June 25, 2015, 01:44:48 pm »

Well put, Bill.  Imo it makes a difference when wishing to be precise in the deep shadows with today's sub 2e- read noise cameras at base ISO (e.g. the new crop of Exmors coming on line now).  Aside from a different looking histogram it makes a big difference with small signals because poisson has no negative values, while gauss does.  It's amazing how precisely physics fits our models when we use the right formulas.

Jack

Jack and Jim K,

How would these considerations affect my modeling of the Canon sensors using the model proposed by Roger Clark? I don't have matab and am not sufficiently versed in the fine points of probability distributions to carry out this analysis, but perhaps you guys can help. These considerations could well affect the determination of engineering DR, but would be less critical when one is dealing with practical photographic DR.

Bill
Logged
Pages: 1 ... 8 9 [10] 11   Go Up