Pages: 1 ... 10 11 [12] 13   Go Down

Author Topic: The Optimum Digital Exposure  (Read 63553 times)

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: The Optimum Digital Exposure
« Reply #220 on: November 06, 2014, 11:49:00 am »

If the histogram comes directly from the raw data and not from the JPEG preview data, the color space set on the camera for rendering of the JPG does not matter.

and firmware can show both or one of these as you decide... and also technically firmware can imitate FRV's exposure correction emulation along with OE indication during post shot review, so that you can see where clipping is going to happen if you adjust your exposure to get more light.
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: The Optimum Digital Exposure
« Reply #221 on: November 06, 2014, 12:00:03 pm »

If the histogram comes directly from the raw data and not from the JPEG preview data, the color space set on the camera for rendering of the JPG does not matter. Let us know what you decide on further reflection.

Bill, I'm talking about the making the in-camera histogram that's derived from a JPEG preview serve as a raw clipping indicator, so in that case the in-camera histogram does not come directly from the raw data, although the world would be a better place if that were an option.

I've about come down on the side of saying that in theory (as opposed to close enough in practice, which we know is true already) you can make the in-camera histogram that's derived from a JPEG preview serve as a raw clipping indicator with the proper UniWB settings. However, I'm having to make some assumptions about the raw-to-colorimetric conversion process that aren't always met in practice.

Jim

barryfitzgerald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 688
Re: The Optimum Digital Exposure
« Reply #222 on: November 06, 2014, 12:39:49 pm »

My take fairly simple

ETTR useful for low light/high ISO (to keep noise down) not overly relevant to low ISO shooting though modern sensors are good and can pull quite heavily into the shadows at base ISO levels
Logged

Eyeball

  • Full Member
  • ***
  • Offline Offline
  • Posts: 150
Re: The Optimum Digital Exposure
« Reply #223 on: November 06, 2014, 12:51:32 pm »

Bill, I'm talking about the making the in-camera histogram that's derived from a JPEG preview serve as a raw clipping indicator, so in that case the in-camera histogram does not come directly from the raw data, although the world would be a better place if that were an option.

I've about come down on the side of saying that in theory (as opposed to close enough in practice, which we know is true already) you can make the in-camera histogram that's derived from a JPEG preview serve as a raw clipping indicator with the proper UniWB settings. However, I'm having to make some assumptions about the raw-to-colorimetric conversion process that aren't always met in practice.

Jim

Jim, I have used UniWB on a few occasions and while I don't care much for the green previews, I found it insightful.  The doubt that I have had though is this:

Can the UniWB histogram be considered at all accurate beyond what it shows in terms of clipping on the right side of the histogram?

I can't express it very well but it always gives me the feeling that it is using a "linear" calculation to simulate what may not be linear (the white balance process) - kind of a "brute force" method, if you will.

I guess I could do some tests and compare the LCD histogram with what I see in RawDigger.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: The Optimum Digital Exposure
« Reply #224 on: November 06, 2014, 01:09:08 pm »

At least for Nikon add the selected Picture Control to the list of WB and ColorSpace.  Nikon introduced that Flat picture control for those post processing.  Of course, it is only applied to JPGs AND the historgram, not the RAW file.

I'll check into that picture control, but could you let us know how one accesses that flat picture control?

Thanks,

Bill

PS

On checking the Nikon web site online, I see that the linear option is available only on the D810 and cameras released subsequent to the release date of the D810. How could one achieve the same effect for the D800?
« Last Edit: November 06, 2014, 01:13:33 pm by bjanes »
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: The Optimum Digital Exposure
« Reply #225 on: November 06, 2014, 01:10:54 pm »

My take fairly simple

ETTR useful for low light/high ISO (to keep noise down) not overly relevant to low ISO shooting though modern sensors are good and can pull quite heavily into the shadows at base ISO levels

That's interesting, exactly opposite to my perspective. You want to minimize read noise. I want to minimize photon/shot noise, and I'm not interested in SNRs below about 10.

Jim

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: The Optimum Digital Exposure
« Reply #226 on: November 06, 2014, 01:13:22 pm »

That's interesting, exactly opposite to my perspective. You want to minimize read noise. I want to minimize photon/shot noise, and I'm not interested in SNRs below about 10.

sure, but some times you have to - when your exposure is limited by the situation for a given camera/lens that you have w/ you... and then by optimizing read noise you can still move something above your S/N = 10.
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2004
    • http://www.guillermoluijk.com
Re: The Optimum Digital Exposure
« Reply #227 on: November 06, 2014, 01:47:30 pm »

It seems to me that RawDigger would be evidence refuting both of your points above.

One of the reasons we want to see a raw histogram is to verify to what extent the tonal values were truly clipped or not.
Seems to me that camera manufacturers could do just a good a job as the RawDigger folks, if not better, since they know all the characteristics of their sensors and the processes being used for capture.

I'm plotting my own RAW histograms since 2008 from DCRAW's RAW data extractions:

Canon 40D genuine RAW histogram where black (1023) and sat (13824) points are clearly seen:




Once the linear data is corrected by those black and sat points:



It is easy to plot log histograms as RAWdigger does, very useful to photographers:






Gabor Schorr's RAWnalyze also plotted RAW histograms time ago (although they were not as nice as mines ;D):




An interesting exercise I did with two Canon 350D RAW files 4 stops apart (one was the the ETTR; to be fair I clipped some few G pixels), was to merge them into an output 16-bit DNG RAW file with more DR than today's best Sony sensor's single shots. Looking at the RAW histograms is claryfying of the merging process (the merge took the best parts of each histogram, basically the most exposed non-clipped pixels):






An histogram is an statistical tool, it doesn't need an image to exist. RAW histograms are very useful, and introducing them in any camera would be very simple. Camera makers are not interested.


Regards

« Last Edit: November 06, 2014, 01:51:39 pm by Guillermo Luijk »
Logged

Eyeball

  • Full Member
  • ***
  • Offline Offline
  • Posts: 150
Re: The Optimum Digital Exposure
« Reply #228 on: November 06, 2014, 01:52:22 pm »

I'm plotting my own RAW histograms since 2008 from DCRAW's RAW data extractions:

Gabor Schor's RAWnalyze also plotted RAW histograms time ago (although they were not as nice as mines ;D):


An histogram is an statistical tool, it doesn't need an image to exist. RAW histograms are very useful, and introducing them in any camera would be very simple. Camera makers are not interested.


Yep, I've been following your work for a long time, Guillermo, and I was a big fan of RAWnalyze, too.  I still have a copy that works with my Canon cameras.  Gabor did some fine work before he passed away.
Logged

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: The Optimum Digital Exposure
« Reply #229 on: November 06, 2014, 01:53:52 pm »

Gabor Schorr

may he rest in peace, he departed untimely.
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: The Optimum Digital Exposure
« Reply #230 on: November 06, 2014, 04:04:49 pm »

Can the UniWB histogram be considered at all accurate beyond what it shows in terms of clipping on the right side of the histogram?

I can't express it very well but it always gives me the feeling that it is using a "linear" calculation to simulate what may not be linear (the white balance process) - kind of a "brute force" method, if you will.

I thought through the preview color space issue, and my gut feeling appears to have been wrong. I'll post something about that soon, but first let me deal with your issue. I will try to restate it: are the intermediate points on a real raw histogram accurate? You'll notice that I changed your UniWB histogram to the real raw histogram. I did that because I didn't want to get into trying to figure out exactly how a proper UniWB histogram differs from a real raw histogram; answering that question for an arbitrary camera is beyond my pay grade.

Before I do that, let's talk about the x-axis on a raw histogram, and the size of the histogram buckets. One way to present a raw histogram is for the x axis to be linear, and all the buckets the same size. This has a certain mathematical purity, since the raw data is linear, but is not very useful to photographers. For one thing, it means that the entire right half of the histogram is the top stop of the dynamic range of the image, and the other 13 or so stops are all crammed into the left half. It also means that half the buckets are devoted to that top f-stop.

One way to deal with this situation to provide something more photographically useful has been to present the logarithm (base 2) of the input values as the x-axis. That's appealing, because one-stop intervals are all the some distance on the plot. In one of those happy accidents associated with the way beam current translated to brightness on a now-ancient cathode ray tube, you can get a rough cut at the logarithmic axis by scaling the raw values into the range [0,1] and raising each one to the power 0.45. In the arcane terminology of digital imaging, that's referred to as "gamma 2.2".

This may seem strange, but it's the same way that Lightroom, which uses a linear color space as its working space, presents its histograms. sRGB JPEG uses a gamma of 2.2 as well. So does Adobe 1998 RGB.

And now, we get to a key word.

"Mr. Clinton, are the histogram values thus presented accurate?"
"It depends on what the meaning of the word 'accurate' is."

Yes, those intermediate values are accurate, in that they're derived from the raw image in a repeatable, mathematically-well-specified manner. But are they useful? IMHO, not very. But then again, again, IMO, the intermediate values in and sRGB histogram don't tell me much. Sure, I can see that there are or aren't a lot of midtone values, but does that help me select the exposure? If we were working with 8-bit images, I could see histogram depopulation from too-aggressive tone curve moves, but those days are well behind us.

One thing you can do with the intermediate values is, once you decide to let something clip (say there's a sunlit window behind your subject), you can look for a little bump that indicates the brightest part of the scene that's not the window.

Does any of that help?

Jim





« Last Edit: November 06, 2014, 04:20:00 pm by Jim Kasson »
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
UniWB histograms, camera color spaces, and clipping estimates
« Reply #231 on: November 06, 2014, 04:37:16 pm »

The question that I've been wrestling with today:

Given a UniWB in-camera histogram in an arbitrary colorimetric color space, does clipping on any channel indicate clipping on the raw image?

My initial guess was "Yes", but I don't believe that any more. Let me walk you through my reasoning, and please tell me if I've made an error.

Warning: mathematics ahead
.

I will use this notation: matrices, M, in bold uppercase, column vectors, v,  in bold lowercase.

OK, let's get started.

Let's say that a pixel in a raw file can be represented as the column vector r. If r, g, and b are raw values in each plane (yeah, yeah, I know there are two g values; cut me a little slack here), r = [r g b]' where ' is the symbol for transposition.

Now let's say that the camera manufacturer has provided me with a compromise matrix, C, for converting raw values to some colorimetric color space. If c is a column vector representing a color in that colorimetric space,

c = C * r

Now let's say that we've come up with a diagonal matrix, WB, that when combined with the other variables in the following way:

c = C * WB * r

makes the white point of the raw space the same as the white point of the colorimetric one.

We can create a new matrix, M, that does both the color space conversion and the white point mapping:

M = C * WB

Lets say that M = [a11 a12 a13; a21 a22 a23; a31 a32 a33]

Having the white point of the raw color space and the colorimetric one be the same means that:

[1 1 1]' = M * [1 1 1]'

And that means

a11 + a12 + a13 = 1
a21 + a22 + a23 = 1
a31 + a32 + a33 = 1

Having each raw space clipping point [1 x x]' [x 1 x]' [x x 1]' where x could be anything, map to the clipping points of the colorimetric color space means that


[1 x x]' = M * [1 x x]'
[x 1 x]' = M * [x 1 x]'
[x x 1]' = M * [x x 1]'

and that means

a11 = 1 and a12 = a13 = 0
a22 = 1 and a21 = a23 = 0
a33 = 1 and a31 = a32 = 0

which means that M is the identity matrix, which means that the colorimetric color space has to be the same as the raw space.

So the UniWB trick will work best where all the raw channels approach clipping at the same time; ie, for magenta-ish highlights (since raw images are usually overly green-sensitive).

This is not a wonderful thought. Please, somebody show me where I erred!

Jim

« Last Edit: November 07, 2014, 07:29:51 am by Jim Kasson »
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: The Optimum Digital Exposure
« Reply #232 on: November 06, 2014, 06:46:16 pm »

sure, but some times you have to - when your exposure is limited by the situation for a given camera/lens that you have w/ you... and then by optimizing read noise you can still move something above your S/N = 10.

I hear you, but it's just not a problem in almost all my photographic situations. Let's work through an example with the D810. FWC is about 75,000 electrons. For a shot noise sigma of 10 electrons, I need a mu of 100 electrons.

75,000/100 = 750.

log2(750) = 9.5 stops, at base ISO.

At base ISO, the D810 read noise is about 5 electrons.

http://blog.kasson.com/?p=7656

With my 100 electron signal, the noise is

sqrt(shotNoise ^2 + readNoise^2) = sqrt(5^2 + 10^2) = sqrt(125) = 11 electrons.

So the RN has increased the total noise by one measly electron.

Well, I hear you thinking, read noise isn't white. There are low frequency components that make it uglier than photon noise, so it should get more weight.

There is merit to that way of thinking. I've done an analysis of the D810 read noise vs low-pass filter cutoff frequency, and, with the exception of the really low ISOs (100; I didn't test 64, but it's probably worse), you could argue that you ought to weight the read noise by at most a factor of two in one direction and a factor of four in the other. But I started with base ISO, so let's weight it by 8. so, at Low frewuency:

sqrt(shotNoise ^2 + readNoise^2) = sqrt(40^2 + 10^2) = sqrt(1700) = 41 electrons.

This sound awful, but because it occurs at such low frequency, and the photon noise SNR is 10 or better, you just don't see it.

Now, let's consider a higher ISO. 3200 is a nice number, since the number of electrons per count on the D810 is 1/10 at that point. Now, full scale is 1638 electrons. Our 100 electron signal is log2(1638/100) = 4 stops down from full scale. The read noise of the D810 at ISO 3200 is about 2 electrons. Multiplying it by 4 to get a worst-case number gives 8.

sqrt(shotNoise ^2 + readNoise^2) = sqrt(8^2 + 10^2) = sqrt(164) = 13 electrons.

It's all very well to talk about numbers, but what do the images look like? I went looking for the D810's read noise in this series of tests:

http://blog.kasson.com/?p=7886

I found that I had to get the average exposure of the bright parts of the test image down to low single-digit electron counts before the low frequency component of the read noise was readily apparent.

YMMV, of course.

Jim

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re:
« Reply #233 on: November 07, 2014, 09:20:13 am »

Unless the article in the link I posted flawed, the ratio of bits representing photon fluctuations to the bits representing the total signal amplitude is constant for all exposures, i.e. for all signal-to-noise ratios. Reducing the S/N does not change the bit ratio.

Is the article flawed?

Is it true the ratio of bits representing photon fluctuations to the bits used for the total signal remain constant?

Sorry it's taken me so long to respond. I find no error in the article once you accept the assumptions, some of which aren't stated. I'm not sure what the utility of the observation is in photography.

Jim

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: The Optimum Digital Exposure
« Reply #234 on: November 07, 2014, 10:20:55 am »

Let's work through an example with the D810.
no, no - I 'd take a Canon camera or Sony A7s - but not like something like D810e even it is not the purest example if ISO-less sensor...
Logged

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: UniWB histograms, camera color spaces, and clipping estimates
« Reply #235 on: November 07, 2014, 10:30:48 am »

Please, somebody show me where I erred!

nobody was talking about mathematically exact precision - getting clipping indication with your subjects under daylight to tungsten light spectrum within 1/6 EV is good enough and it does (as you test with rawdigger or FRV when you tune your OOC JPG settings)... do you need to be precise to 0.00000000000001 EV ? nope...

Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: The Optimum Digital Exposure
« Reply #236 on: November 07, 2014, 11:13:12 am »

no, no - I 'd take a Canon camera or Sony A7s - but not like something like D810e even it is not the purest example if ISO-less sensor...

I don't have any Canons around. I could work through it all with the a7S, if that would be useful. Increasing the ISO on the a7S from 1600 to 2000 triggers the change in conversion gain, which lowers read noise dramatically. Is that what you're looking for?

BTW, I'm still struggling with getting a good low-frequency read noise metric from my data.

Jim

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: UniWB histograms, camera color spaces, and clipping estimates
« Reply #237 on: November 07, 2014, 11:17:57 am »

nobody was talking about mathematically exact precision - getting clipping indication with your subjects under daylight to tungsten light spectrum within 1/6 EV is good enough and it does (as you test with rawdigger or FRV when you tune your OOC JPG settings)... do you need to be precise to 0.00000000000001 EV ? nope...

That's good, because it's going to depend on the compromise matrix from raw to JPEG color space, and all bets are off if LUT-based color space conversion is used in your raw processor. That said, I've found UniWB to work pretty well in practice with a JPEG space of Adobe 1998 RGB. It's just going to be hard to figure out in general if Bill's PPRGB JPEG color space is better or worse. I might be able to do it for a particular camera.

Jim

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: The Optimum Digital Exposure
« Reply #238 on: November 07, 2014, 11:26:33 am »

I don't have any Canons around. I could work through it all with the a7S, if that would be useful. Increasing the ISO on the a7S from 1600 to 2000 triggers the change in conversion gain, which lowers read noise dramatically. Is that what you're looking for?
my logic, naturally, is that the further away you are from ISO-less camera the more you are going to use that play with gain (ISO), right ? and the closer you are to the ISO-less camera (may be Nikon D7000 is the ideal example of that in the nature - except cameras where ISO is only by tag in raw file) the less you might be inclined to play with gain (ISO) - unless you want for OOC JPG or postshot-review-in-camera purposes... and naturally if you are shooting some action with D4/D4s (or even A7s) - those are further away from ISO less than D810 - the more chances it seems you have that some important part of the image will be the area below your S/N = 10 and you might get that above 10 by pushing gain/ISO... no ? you have D4 as I recall too.
Logged

deejjjaaaa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1170
Re: UniWB histograms, camera color spaces, and clipping estimates
« Reply #239 on: November 07, 2014, 11:32:36 am »

It's just going to be hard to figure out in general if Bill's PPRGB JPEG color space is better or worse. I might be able to do it for a particular camera.
I think if you are creating a model you need to introduce some tolerances there too... I don't think it makes sense to talk about anything related to camera hoping for any kind of precision precision to be better than 1/6 EV, no ? is there a real need to be more precise than 1/6 EV with clipping ? are you really going to tune your exposure that close ? if I could I 'd then rather dial 1/3-1/2 ev back right away (one Iliah Borg always mention that some cameras at least are non linear near clipping)... I talking not about any clipping of course, but clipping in valuable part of the image... as sometimes you want to clip (sacrifice) some specular reflections to achieve the proper exposure for the important parts of the image.
« Last Edit: November 07, 2014, 11:34:35 am by deejjjaaaa »
Logged
Pages: 1 ... 10 11 [12] 13   Go Up