Pages: 1 2 3 [4] 5   Go Down

Author Topic: Shooting to the right and raw conversion  (Read 45812 times)

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #60 on: July 03, 2007, 07:13:04 am »

Quote
Yes, this is what you can read almost everywhere about ACR, but I never saw an example where it was clear that ACR recreated a blown channel (mean blown even in the RAW data) with the information of the other two. "This feat was demonstrated in the analysis I did on Digidog's overexposed file"-> where exactly can I find this analysis? I would be interested in looking at it.


[a href=\"index.php?act=findpost&pid=125575\"][{POST_SNAPBACK}][/a]

My analysis of Andrew's ETTR files was posted earlier in this thread. One can use the same images to demonstrate highlight recovery in ACR.

Here is a histogram of the green channel of Andrew's image taken at 1/50 sec @ f/16 from the white portion of the target. The image was rendered with DCRaw and the histogram is from ImageJ, which supports 16 bit histograms. The highlights are totally blown.

[attachment=2730:attachment]

Here is the histogram after Andrew's highlight recovery. It is apparent that the green channel has been reconstructed in the highlight area.

[attachment=2731:attachment]

Bill
Logged

charleski

  • Newbie
  • *
  • Offline Offline
  • Posts: 34
Shooting to the right and raw conversion
« Reply #61 on: July 04, 2007, 06:14:49 pm »

Quote
The quality of my arguments is my qualification.  There is nothing mysterious, either in what I say.  You can test most of it for yourself.
[{POST_SNAPBACK}][/a]
I agree that a reliance on popularity can be misleading. But the fact is that arguments are not enough. Science is based on data. I've read through [a href=\"http://luminous-landscape.com/forum/index.php?showtopic=17706&st=37#]your data post[/url] a few times and must admit I have a hard time deciphering it to reconcile with your conclusions (despite a medical degree and a neurobiology PhD that was founded on digital data acquisition).

Maybe you could expand on the data you collected and explain (or at least link to) how you reached the arguments that you now propound.
« Last Edit: July 04, 2007, 06:16:02 pm by charleski »
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #62 on: July 04, 2007, 09:41:45 pm »

Quote
I agree that a reliance on popularity can be misleading. But the fact is that arguments are not enough. Science is based on data.

Without *any* data from myself, I can still challenge Roger Clarks' conclusions, based on his data.  His data does not support his conclusion, IMO, except as weak circumstantial evidence.  He sees a tapering off in improvement in read noises as measured in electrons, going from ISO 800 to 1600, and then no improvement at all for ISO 3200 (no surprise there, as the camera doesn't really have an ISO 3200 amplification).  Since 1 electron is approximately 1 ADU somewhere just below ISO 1600, he concludes that it is because of "unity gain", a point past which there is no further gain from more amplification.  He supports this with "logic" that says that it makes no sense to count beyond unity gain, because electrons are discreet units and you only need one unit to count one.  That logic would be fine, if cameras were actually counting photons-turned-electrons, and I wish that were true, but THEY ARE NOT COUNTING ELECTRONS.  They are digitizing an analog amplification of the discreet electron counts, blanketed in camera-generated noises with an infinite number of in-between analog values from at least a couple of sources before the signal becomes binary data.

The variance due to read noises in apparent electron counts is tremendous at the ISOs near "unity gain".  The situation is not even remotely close to "counting electrons" with any degree of accuracy, at the pixel level.  Here is a histogram of a blackframe from the 20D at ISO 1600:



The variance here is due to read noises picked up throughout the signal chain, including noise introduced by the ADC unit.  The noise introduced by the ADC unit, alone, would be cut in half, relative to signal, if there were a true gain-based ISO 3200.

Quote
I've read through your data post a few times and must admit I have a hard time deciphering it to reconcile with your conclusions (despite a medical degree and a neurobiology PhD that was founded on digital data acquisition).

I can only guess what you're refering to, without your being specific.  I have said many things.

The point of my chart, if that is what you are refering to, is that the values at ISO 3200 are *NOT* a natural progression of the curves from ISO 100 through ISO 1600.  The curve for "total noise" from 100 to 1600 is the closest one to actually suggesting the ISO 3200 value, but the 3200 value is still an abrupt flattening of the curve by a small degree.  Everyone who has ever tried to use the shadows of Canon DSLRs at high ISOs knows that one of the most visible aspects of noise is the banding or line noise.  Its statistical strength is very small, compared to its visible strength.  Subtracting the banding component from a blackframe decreases the standard deviation by almost nothing.  With my 20D, it lowered the standard deviation at ISO 1600 from about 4.7 to about 4.6, yet the visible improvement is much more than the numbers would suggest.

Quote
Maybe you could expand on the data you collected and explain (or at least link to) how you reached the arguments that you now propound.
[a href=\"index.php?act=findpost&pid=126507\"][{POST_SNAPBACK}][/a]

It's more a matter of *not* jumping to the conclusions that Roger Clark does, IMO.  My interpretation of his data and mine combined suggest to me that it may be possible to get less read noise in electrons with greater and higher quality amplification.  None of the data contradicts that possibility.  The "unity gain" theory totally ignores the fact that the real limitation to low signal quality is camera-generated read noises, which have been demonstrated to decrease, relative to absolute signal, at higher amplifications and with improved technologies.  It is currently of no practical value, whatsoever.  If cameras *were* actually counting electrons, then of course, there would be no point in amplifying to an extent, or digitizing to such a depth, where each electron was represented by more than 1 ADU, but this is the real world, and read noises are a real obstacle to counting electrons.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #63 on: July 04, 2007, 11:48:30 pm »

Quote
The variance due to read noises in apparent electron counts is tremendous at the ISOs near "unity gain".  The situation is not even remotely close to "counting electrons" with any degree of accuracy, at the pixel level.  Here is a histogram of a blackframe from the 20D at ISO 1600:



The variance here is due to read noises picked up throughout the signal chain, including noise introduced by the ADC unit.  The noise introduced by the ADC unit, alone, would be cut in half, relative to signal, if there were a true gain-based ISO 3200.

[{POST_SNAPBACK}][/a]

Your histogram of the dark frame (bias frame) is similar to that determined by others, for example [a href=\"http://www.astrosurf.org/buil/20d/20dvs10d.htm]Christian Buhl[/url]. However, the bias frame does not represent read noise, since it also includes fixed pattern noise derived from non-uniformity of the light field (e.g. due to vignetting in the lens), dust specs on the sensor, pixel to pixel sensitivity variations and other effects. To get the true read noise, you must subtract the fixed pattern noise as Christian points out in the above link. The horizontal and vertical banding noise on which you dwell is one type of fixed pattern noise. Since you do not describe your methods, I have no idea of what you are doing and whether or not any of your conclusions are valid.

The primary component of read noise is is from the on chip amplifier (see here, read noise). If you increase amplification, this noise would most likely increase also and you might not gain much. The noise introduced by the A/D converter is relatively minor and can be reduced by increasing the bit depth of the A/D converter (as Roger points out for the new EOS 1D M3, which uses a 14 bit device).

The relative contributions of photon noise to read noise at unity gain vary with signal strength. With higher signals, the noise source is mainly photon noise, while read noise predominates at lower signal strength. You did not state what signal strength was involved in your statement about read noise at unity gain. Of course, with a dark frame with a short integration time, the noise is almost entirely read noise, since there would be no photon noise and thermal noise would be minimal.

To use Roger's data for the 1DM2, full well at ISO 50 is 80,000 electrons. At ISO 1600  (unity gain),  full scale on the A/D converter would represent 2500 electrons. The variance of photon noise would be 2500 and the variance due to read noise would be 15.2. If you look at noise in the shadows, for example 6 EV down, one would collect about 40 electrons. The variances of the photon noise and read noise would be 40 and 15.2 respectively. Photon noise still predominates.

Furthermore, when one determines read noise, one is measuring total noise and the various contributing components can not be separated out, and your statement "The noise introduced by the ADC unit, alone, would be cut in half, relative to signal, if there were a true gain-based ISO 3200" does not make much sense since you can't separate out amplifier noise from ADC noise.

Finally, the proof of the pudding is in the eating, and you should measure noise at various ISOs rather than theorizing at what they might be. Astronomers who have done their homework apparently do not bother with amplification much above unity gain, since it has no advantage with respect to noise and merely reduces dynamic range. The unity gain of the Canon cameras is relatively high, and the differences might be demonstrated more easily with a camera with a lower unity gain (such as the Nikon D200 with a unity gain of 800).
« Last Edit: July 05, 2007, 09:44:20 am by bjanes »
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Shooting to the right and raw conversion
« Reply #64 on: July 08, 2007, 05:57:31 am »

Quote
My analysis of Andrew's ETTR files was posted earlier in this thread. One can use the same images to demonstrate highlight recovery in ACR.

Here is a histogram of the green channel of Andrew's image taken at 1/50 sec @ f/16 from the white portion of the target. The image was rendered with DCRaw and the histogram is from ImageJ, which supports 16 bit histograms. The highlights are totally blown.

[attachment=2730:attachment]

Here is the histogram after Andrew's highlight recovery. It is apparent that the green channel has been reconstructed in the highlight area.

[attachment=2731:attachment]

Bill
[a href=\"index.php?act=findpost&pid=126206\"][{POST_SNAPBACK}][/a]

mmmm I see, you are totally right. ACR recovered both the red and green channels from the blue one. I thought highlight recovery in ACR was a fake from Adobe, glad to see I was wrong.
Thanks for the info Bill.

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #65 on: July 08, 2007, 07:39:50 pm »

Quote
Your histogram of the dark frame (bias frame) is similar to that determined by others, for example Christian Buhl. However, the bias frame does not represent read noise, since it also includes fixed pattern noise derived from non-uniformity of the light field

"Light field"?  Hello?

Quote
(e.g. due to vignetting in the lens),

Vignetting in the lens?  Hello?

Quote
dust specs on the sensor,

How does the sensor know that the specs are there?  Are they telepathic?

Quote
pixel to pixel sensitivity variations and other effects

Sensitivity to what?  Do you know what a "blackframe" is?  It is the noise of having to read a sensor, with no light having exposed it.

Quote
To get the true read noise, you must subtract the fixed pattern noise as Christian points out in the above link.

No, the read noise of a black frame is the noise of the blackframe.  The *RANDOM* component of a blackframe, is the blackframe minus any fixed pattern noise.  The Canon 20D has no significant fixed pattern noise in 1/8000 second black "exposures".  Stacking 16 of them yields 1/4 of the statistical noise, and no non-random patterns appear.  I've taken 16 exposures interleaved with 16 blackframes, stacked all the exposures together, stacked all the blacks together, and subtracted the black result from the exposure result and all I got was a 40% increase in random noise.  It takes a couple of seconds at ISO 1600 for fixed-pattern noise to become measurably higher.

The problem with your understanding is that you are not familiar enough with these issues to have a clear picture in your head of what the terms are referring to.

You see me write the word "blackframe" and the only thing you can think of is astrophotography, with its long exposures, and reply as if I were talking about stacking many long exposures, subtracting fixed-pattern noises, and flat fields.  None of that has anything directly to do with how much amplification is worth using for readout.  In fact, if other noises were present which I needed to subtract out, then the curves in my 20D noise profile graph would be *STEEPER*, not flatter, as the curves approached ISO 1600.  Do you understand that?  Noises in the sensor itself have the same intensity in "electrons" at all ISOs; unsubtracted fixed pattern noises would compress the differences in measured noise from single frames at different ISOs, as measured in electrons!

Roger often mistakes line noise for fixed-pattern noise; the line noises I speak of *DO NOT* repeat from frame to frame; they are in a different place with a different pattern, in *EVERY* frame.  No matter how many times I have told this to him, he goes on again to call the line noises "fixed pattern" noises.  They are random 1-dimensional noises.  In some cases, there is a periodic component, but the phases are totally random.

Quote
The horizontal and vertical banding noise on which you dwell is one type of fixed pattern noise. Since you do not describe your methods,

I have described many times to you what I do.  I take blackframes (NOTHING TO DO WITH ASTRONOMY!) at a short exposure at all ISOs, load each RAW file into IRIS, window the display to 0 to 255 ADU, save out into a BMP, load into photoshop.  I get the standard deviation of the blackframe from the histogram tool.  That is the "total read noise".  Then I make a copy of the blackframe, and resize it with bicubic so that it is one pixel wide.  Then, I resize it again to the original width (not really necessary, but I may want to subtract it from the original), and get the standard deviation.  This is the standard deviation of the horizontal line noise.  Do the same vertically for the vertical line noise.  Very simple.  I would have included noise data for the original with the line noises subtracted, but they are almost exactly the same as the original, and would not be distinct on the graph.

Quote
I have no idea of what you are doing and whether or not any of your conclusions are valid.

The primary component of read noise is is from the on chip amplifier (see here, read noise).
"The major component of readout noise arises from the on-chip preamplifier."

That's accurate with older CCD cameras.  The cameras Roger draws his conclusions from are not older, or CCD.

Quote
If you increase amplification, this noise would most likely increase also and you might not gain much.

We are discussing at what point it becomes worthless.  Saying that it will become worthless at some point is stating the obvious.  I am saying that there is no evidence that read noise can't be reduced further than it is now by higher and/or better amplification.

Quote
The noise introduced by the A/D converter is relatively minor and can be reduced by increasing the bit depth of the A/D converter (as Roger points out for the new EOS 1D M3, which uses a 14 bit device).

There is no evidence of this happening in the 1Dmk3.  The Ratio of max RAW signal to read noise is almost exactly the same on the mk2 and the mk3 at ISO 100.  Low ISOs are the ones with the largest percentage of their noise contributed by the ADC, and ISO 100 has no improvement.  That means that it is likely that there is no real improvement in ADC-contributed noise in the mk3.  ISO 3200 on the mk3 has a read noise of 6.0 ADU compared to 9.4 ADU on the mk2, but the benefits are fully there even if the RAW data is quantized to 12 or even 10 bits.
« Last Edit: July 20, 2007, 11:35:47 am by John Sheehy »
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #66 on: July 09, 2007, 01:17:24 pm »

Quote
"Light field"?  Hello?
Vignetting in the lens?  Hello?
How does the sensor know that the specs are there?  Are they telepathic?
Sensitivity to what?  Do you know what a "blackframe" is?  It is the noise of having to read a sensor, with no light having exposed it.
No, the read noise of a black frame is the noise of the blackframe.  The *RANDOM* component of a blackframe, is the blackframe minus any fixed pattern noise. 

The problem with your understanding is that you are not familiar enough with these issues to have a clear picture in your head of what the terms are referring to.
[a href=\"index.php?act=findpost&pid=127184\"][{POST_SNAPBACK}][/a]

Your post is very difficult to read, since you did not get the quote/unquotes right.

Anyway, I will respond to a few of your points. If you are determining the total noise in an image, you need to take the shot noise, read noise, and dark current (thermal noise) into account. With short exposures (up to several seconds), the dark current is negligible and may be ignored. You dwell on read noise, but photon noise is dominant in most situations. Photon noise may not be a wild card, but it needs to be taken into account. If if overshadows read noise by a large margin in a particular situation, read noise can be neglected for practical purposes, just as we are ignoring thermal noise.

To analyze the photon noise, you need some light falling on the sensor. If you merely take a picture of a supposedly uniform field, the light falling on the sensor may vary due to the factors I mentioned (taken from Roger's web site). These factors are fixed, and can be eliminated by subtracting images taken under identical conditions as Roger explains. For read noise, lens vignetting, dust on the sensor, etc do not apply since the image is taken in the dark with the lens cap on (and I should not have mentioned them in the context of read noise).

If you are analyzing a black frame for read noise (yes, I know they are different from a dark frame to determine the dark current in the long exposures used in astronomy), you must subtract fixed pattern noise. Your understanding of read noise is different from what Roger uses or from Christian Buhl's concept: "Subtraction of an offset image and of the fixed pattern structure for the EOS 10D. It is the true readout noise "image" of the camera." You allege that I don't have a clear concept of the terms involved, but you make up definitions on your own.

Quote
I have described many times to you what I do.  I take blackframes (NOTHING TO DO WITH ASTRONOMY!) at a short exposure at all ISOs, load each RAW file into IRIS, window the display to 0 to 255 ADU, save out into a BMP, load into photoshop.  I get the standard deviation of the blackframe from the histogram tool.  That is the "total read noise".  Then I make a copy of the blackframe, and resize it with bicubic so that it is one pixel wide.  Then, I resize it again to the original width (not really necessary, but I may want to subtract it from the original), and get the standard deviation.  This is the standard deviation of the horizontal line noise.  Do the same vertically for the vertical line noise.  Very simple.  I would have included noise data for the original with the line noises subtracted, but they are almost exactly the same as the original, and would not be distinct on the graph.
[a href=\"index.php?act=findpost&pid=127184\"][{POST_SNAPBACK}][/a]

This is the first time I have seen an explanation of your methods in such detail. I don't know why you save in 8 bit format, since you lose precision. I multiply the results by 16 to convert 12 bit data to 16 bit format. The standard deviation of the black frame is not the read noise, since it includes fixed pattern noise. To get read noise, you must subtract a bias frame as Christian explains. That is also what Roger does.

Quote
You've just described my blackframes!  Why all the confusion at the beginning of your reply?
[a href=\"index.php?act=findpost&pid=127184\"][{POST_SNAPBACK}][/a]

You jumped to the wrong conclusion.

Quote
A chain is no stronger than its weakest link, and the red channel is 1 stop less sensitive in daylight WB, so Sat-6EV is only 1.5 stops below "middle red".  Change the WB to tungsten, and the blue channel is only 0.5 stops below "middle blue", with Sat-6EV!
[a href=\"index.php?act=findpost&pid=127184\"][{POST_SNAPBACK}][/a]

Yes, I was using only the green channel to simplify things. A full analysis would require all channels to be evaluated.

Quote
The bottom line, which you seem to ignore, is that the steps of the ADU are arbitrary relative to the analog equivalent of an electron.
[a href=\"index.php?act=findpost&pid=127184\"][{POST_SNAPBACK}][/a]

Not really. The ADU is related to the number of electrons as described by the camera gain, which is in units of electrons/ADU.
« Last Edit: July 09, 2007, 01:22:47 pm by bjanes »
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Shooting to the right and raw conversion
« Reply #67 on: July 19, 2007, 10:19:26 am »

What do you guys think of this?

[attachment=2848:attachment]

Top, minus 1 stop over flash meter recommendation (ISO 400)

Middle Normal

Bottom Plus 1.5 stops, normalized with exposure slider in Lightroom (-1.46).

Oh, I should add, small section of a big image with many different targets. New images, shoot with strobe (very even lighting according to flash meter). Did ISO 100/400 and 800 (this is but one small example). Bracketed with strobe back NOT aperture (that affected the results too much).

On my 5D, 2 stops over was blown out, couldn't recover less than 100% on whites with exposure slider in LR. 1.5, over, no problem.
« Last Edit: July 19, 2007, 10:23:23 am by digitaldog »
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Shooting to the right and raw conversion
« Reply #68 on: July 20, 2007, 12:21:49 am »

Quote
What do you guys think of this?

[attachment=2848:attachment]

Top, minus 1 stop over flash meter recommendation (ISO 400)

Middle Normal

Bottom Plus 1.5 stops, normalized with exposure slider in Lightroom (-1.46).
[a href=\"index.php?act=findpost&pid=129007\"][{POST_SNAPBACK}][/a]

Andrew,
I've always found ACR can do a spectacular job recovering greys. A grey sky or a grey concrete wall with fine texture seems to be able to accept more exposure before clipping than a scene with color. A particularly common indication of clipping is a deep blue sky that shows an obvious shift towards cyan.

It was mentioned earlier in the thread that ACR always applies a white balance to the conversion and that this apparently is responsible for a degree of clipping. There's no option, as far as I see, for 'no color balance' in ACR.

I'm not sure if the following is relevant, but recently whilst experimenting with a calibration of my scanner with an old IT8 target using Vuescan software, I noticed a dramatic change in the histogram when selecting 'no color balance'.

The following 2 previews of a slightly overexposed Ektachrome compare 'no color balance' with 'white balance'. Whatever characteristic of WB I select, neutral or landscape etc, the histogram is always pushed to the extreme right, with this particular overexposed slide, but with no WB the histogram is very far from the right.

[attachment=2852:attachment]  [attachment=2853:attachment]
« Last Edit: July 20, 2007, 01:02:21 am by Ray »
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #69 on: July 20, 2007, 11:38:44 am »

This is the second half of a reply I made last week; for some reason, the Quote feature won't work with the entire post, but does broken in into two halves.  Apparently there is a limit on the number of Quote delimiters in the software.

Quote
The relative contributions of photon noise to read noise at unity gain vary with signal strength. With higher signals, the noise source is mainly photon noise, while read noise predominates at lower signal strength. You did not state what signal strength was involved in your statement about read noise at unity gain.

Perhaps that's because it is irrelevant.  Shot noise is not a wild card; shot noise is dependent only on the geometry of photon collection and quantum efficiency, including the effects of filters.  Read noise is the thing that varies greatly from one camera to another, and its relationship to ISO varies greatly on any given camera.  None of the things that you do to improve read noise have any effect on shot noise.  The less read noise you have, the more usable your shadows become.

Quote
Of course, with a dark frame with a short integration time, the noise is almost entirely read noise, since there would be no photon noise and thermal noise would be minimal.

You've just described my blackframes!  Why all the confusion at the beginning of your reply?

Quote
To use Roger's data for the 1DM2, full well at ISO 50 is 80,000 electrons. At ISO 1600  (unity gain),  full scale on the A/D converter would represent 2500 electrons.

No; that's not Roger's data.  Roger is well aware (pun intended) that RAW saturation at ISO 100 is not half as many photons as well saturation at ISO 50.  ISO 50 is really only capable of about ISO 70 to 75 on the 1Dmk2, but is exposed like it is actually ISO 50, leaving less highlight headroom than the other ISOs, relative to metering (a major engineering blunder, IMO).  RAW saturation at ISO 1600 on the 1Dmk2 is approximately 52300 (by his data) divided by 16 = 3269 electrons.  I don't know if his values are correct, though, because he may assume that the 1Dmk2 has 4095 RAW levels to use, but signal is only represented by ADU 128 through about 3700+.  He seems to derive either ADU:electron "gain" or max photon count from the other, using the uncertain figure.

Quote
The variance of photon noise would be 2500 and the variance due to read noise would be 15.2. If you look at noise in the shadows, for example 6 EV down, one would collect about 40 electrons. The variances of the photon noise and read noise would be 40 and 15.2 respectively. Photon noise still predominates.

No.  Six stops down from saturation would be 3269/(2^6) = 51.1 electrons, which would have a shot noise of 7.15 electrons.  Total read noise is about 4.19 electrons.  Read plus shot would be (51.1 + (4.19)^2)^0.5 = 8.29 electrons, or about 0.21 stops more noise.  How far down is Sat-6EV, though?  There is about 3.5 stops of headroom above metered grey in most Canon DSLRs (4.0 for the XTi).  That puts Sat-6EV about 2.5 stops below middle grey; not very far into the shadows at all, and that's just the green channel.  A chain is no stronger than its weakest link, and the red channel is 1 stop less sensitive in daylight WB, so Sat-6EV is only 1.5 stops below "middle red".  Change the WB to tungsten, and the blue channel is only 0.5 stops below "middle blue", with Sat-6EV!

Now, let's talk about some *real* (but still not extreme) shadows:  4 stops below middle blue in tungsten lighting - that's Sat-9.5EV.  3269/2^9.5 is 4.51 electrons.  That's a shot noise of 2.2, against a read noise of 4.19.  Total noise is (4.51+(4.19)^2) = 4.7, or 1.09 stops more total noise.

Not to mention the fact that the read noise has highly-perceptible line noises not present in shot noise of equal intensity.

Quote
Furthermore, when one determines read noise, one is measuring total noise and the various contributing components can not be separated out, and your statement "The noise introduced by the ADC unit, alone, would be cut in half, relative to signal, if there were a true gain-based ISO 3200" does not make much sense since you can't separate out amplifier noise from ADC noise.

It is not clear how much of the noise is due to the ADC, but ADC noise is real, and whatever part it is, would be reduced 50% relative to signal, a small, but real improvement.  Total noise at ISO 3200 would be (non-ADC-1600-noise^2+(ADC-1600-noise/2)^2)^0.5.  You don't even need any more at-sensor gain to get this small benefit, so the issue of the point of diminishing returns of sensor-read amplification is another issue.  Some manufacturers use amplification of some kind for ISO 3200; the Minolta K7 a case in point; it's ISO 3200 has less noise than 1600 under-exposed by a stop.

Quote
Finally, the proof of the pudding is in the eating, and you should measure noise at various ISOs rather than theorizing at what they might be.

I did; what are you talking about?

Quote
Astronomers who have done their homework apparently do not bother with amplification much above unity gain, since it has no advantage with respect to noise and merely reduces dynamic range.

That's a very vague statement, and it may apply to old technology, and/or equipment that doesn't have any gain above unity gain, because the manufacturers believed the same myth, or technology wasn't ready.

The bottom line, which you seem to ignore, is that the steps of the ADU are arbitrary relative to the analog equivalent of an electron.

Quote
The unity gain of the Canon cameras is relatively high, and the differences might be demonstrated more easily with a camera with a lower unity gain (such as the Nikon D200 with a unity gain of 800).
[a href=\"index.php?act=findpost&pid=126528\"][{POST_SNAPBACK}][/a]


Go ahead, demonstrate it, and I will offer alternate explanations for your results.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #70 on: July 20, 2007, 12:12:43 pm »

Quote
Your post is very difficult to read, since you did not get the quote/unquotes right.

It is fixed now.  I had to break the post in half for the Quote feature to work properly.  I did have a pair of quote delimiters reversed originally, but even after making sure they were all correct, it still wouldn't post correctly.

Quote
Anyway, I will respond to a few of your points. If you are determining the total noise in an image, you need to take the shot noise, read noise, and dark current (thermal noise) into account.

I was determining the total noise of a blackframe at a short exposure.  There is no shot noise, and there is infinitessimal dark current noise in a short exposure.

Quote
  With short exposures (up to several seconds), the dark current is negligible and may be ignored. You dwell on read noise, but photon noise is dominant in most situations.

Photon noise is dominant in the highlights, and read noise is dominant in the shadow areas.  Read noise is dominant in determining ultimate DR.

Quote
Photon noise may not be a wild card, but it needs to be taken into account. If if overshadows read noise by a large margin in a particular situation, read noise can be neglected for practical purposes, just as we are ignoring thermal noise.

Dark current noise is infinitessimal in a short exposure, compared to read noise.

Quote
To analyze the photon noise, you need some light falling on the sensor.

To measure read noise in a blackframe?  Blackframe read noise is a noise that has no relationship to signal, whatsoever, other than the fact that clipping the signal will also clip away the read noise (and any other noise).

Quote
If you merely take a picture of a supposedly uniform field, the light falling on the sensor may vary due to the factors I mentioned (taken from Roger's web site). These factors are fixed, and can be eliminated by subtracting images taken under identical conditions as Roger explains. For read noise, lens vignetting, dust on the sensor, etc do not apply since the image is taken in the dark with the lens cap on (and I should not have mentioned them in the context of read noise).

If you are analyzing a black frame for read noise (yes, I know they are different from a dark frame to determine the dark current in the long exposures used in astronomy), you must subtract fixed pattern noise. Your understanding of read noise is different from what Roger uses or from Christian Buhl's concept: "Subtraction of an offset image and of the fixed pattern structure for the EOS 10D. It is the true readout noise "image" of the camera." You allege that I don't have a clear concept of the terms involved, but you make up definitions on your own.
This is the first time I have seen an explanation of your methods in such detail. I don't know why you save in 8 bit format, since you lose precision..

Only the 8 LSBs are used in a 20D blackframe.  You could probably count on one hand the number of pixels above 255 ADU in a blackframe from a 12-bit RAW.

BMP is the only lossless format that IRIS outputs without loss in a file format understandable and unmangled by photoshop.

Quote
I multiply the results by 16 to convert 12 bit data to 16 bit format.

The standard deviation of the black frame is not the read noise, since it includes fixed pattern noise.

In the Canon 20D, fixed pattern noise is infinitessimal in short exposures.  That is one reason why Canons are so relatively noise-free in the shadows at high ISOs; the readout circutry is aware of any *consistent* significant offsets from black.

Quote
To get read noise, you must subtract a bias frame as Christian explains. That is also what Roger does.

Does for what?

I've been playing with Canon RAW data long enough to know that there is no fixed pattern noise in any short exposure that has any visible effect, even in very under-exposed images pushed.

Quote
You jumped to the wrong conclusion.
Yes, I was using only the green channel to simplify things. A full analysis would require all channels to be evaluated.

Not really. The ADU is related to the number of electrons as described by the camera gain, which is in units of electrons/ADU.
[a href=\"index.php?act=findpost&pid=127288\"][{POST_SNAPBACK}][/a]

You can measure any two quantities and calculate their ratio, and give that ratio a name.  That doesn't mean, however, that there is some special significance to "Unity" .

Here are some parallels:

"It's not worth driving at a speed of less than a mile a minute; you won't get anywhere that way."

"All resistances in the circuit less than 1 ohm should be removed, as they offer no resistance".

I could go on and on, but the point is that there is no empirical relationship between ADU steps and amplified electron counts clouded by analog read noise.  All you would need to do is change the bit depth of the camera's RAW output to determine the maximum usable amplification at the initial readout.  The 1Dmk3 does just that; it gives a 4x boost to the "ADU:electron gain".  If "unity gain" was at ISO 1200 on the 1Dmk2, then it is now at ISO 300 on the 1Dmk3!  If you think about nothing else at all, think about that!  By the same token, just dropping bits off of any ADC would make it worthwhile to use greater amplification at the inital read!

There is nothing about the amplification levels at the photosite (CMOS) that has anything directly to do with electrons per se, nor with the gain of the ADU.  Somehow you think that the camera knows about electrons, and their signal magnitude.  The ADC is only taking an analog signal which has no spiking in its virtual analog histogram, and digitizing it.  In order for the original signal to be preserved as best as possible, the noise needs to be so small that individual electron counts would have spikes; then we could talk about actually counting electrons.  Since we can't count electrons, what we need is amplification that gives the least noise, no matter how many ADU that turns out to be.

Unity gain is an arbitrary ratio with current levels of analog noise.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #71 on: July 20, 2007, 09:34:20 pm »

Quote
I could go on and on, but the point is that there is no empirical relationship between ADU steps and amplified electron counts clouded by analog read noise.

Here are some parallels:

"It's not worth driving at a speed of less than a mile a minute; you won't get anywhere that way."

"All resistances in the circuit less than 1 ohm should be removed, as they offer no resistance".
[a href=\"index.php?act=findpost&pid=129173\"][{POST_SNAPBACK}][/a]

Please don't, since your parallels are not particularly well chosen  

Quote
All you would need to do is change the bit depth of the camera's RAW output to determine the maximum usable amplification at the initial readout.  The 1Dmk3 does just that; it gives a 4x boost to the "ADU:electron gain".  If "unity gain" was at ISO 1200 on the 1Dmk2, then it is now at ISO 300 on the 1Dmk3!  If you think about nothing else at all, think about that!  By the same token, just dropping bits off of any ADC would make it worthwhile to use greater amplification at the inital read!
[a href=\"index.php?act=findpost&pid=129173\"][{POST_SNAPBACK}][/a]

You do not seem to understand the concept of unity gain. As Roger explained, if you use unity gain to compare the sensitivity of sensors, you must express it in the same bit depth.

Yes, with a unity gain at ISO 300 with the 1D M3 you could leave the ISO at 300 and shoot in dim light without any need to adjust the ISO to a higher value. If you increased the bit depth to 17, you wouldn't even need to adjust the ISO at all, since you would have captured all the information at base ISO. This assumes that the read noise at base ISO is kept under control. With the 1DM2, Roger's data show that read noise increases considerably at at base ISO as compared to ISO 1600. With the Nikon D200, the increase is less marked (up to 10 electrons from 7.4).
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Shooting to the right and raw conversion
« Reply #72 on: July 21, 2007, 05:21:59 pm »

First off, I'd like to point out that John Sheehy is absolutely correct in that the camera sensors mentioned by Mr. Clark do not count individual electrons.

Counting single electrons is so hard that it was first done just over two years ago by Swedish physicists.

If John Sheehy doesn't understand the concept of unity gain, it may be because it's practically impossible to have a setting in the camera where it can count single electrons.

If bjanes wants to prove Roger Clark right (or he wants to do so himself), then my guess is that there's a Nobel Prize in physics waiting; doing this at room temperature would be a significant leap forward.
Logged
Jan

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #73 on: July 23, 2007, 11:02:25 am »

Quote
If bjanes wants to prove Roger Clark right (or he wants to do so himself), then my guess is that there's a Nobel Prize in physics waiting; doing this at room temperature would be a significant leap forward.
[a href=\"index.php?act=findpost&pid=129357\"][{POST_SNAPBACK}][/a]

I expect lots of density in discussion of things like politics or religion, but stuff like this is puzzling.  How can anyone, aware that the noise in the cleanest of systems varies in a smooth, analog manner over a couple dozen electrons, think that counting of electrons is occuring.

On top of the lack of any logic in the concept of unity gain as a limit in a noisy analog system, the 1Dmk3 totally contradicts the concept of unity gain.  Even if all ADUs must be converted to 12-bit to qualify, the 1Dmk3 at ISO 3200 has an ADU_12:electron ratio ("gain") of 2.18; far above "unity gain", yet the amplification is immensely useful.
« Last Edit: July 23, 2007, 11:03:56 am by John Sheehy »
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #74 on: July 23, 2007, 11:30:36 am »

Quote
Please don't, since your parallels are not particularly well chosen 

All they are chosen for is their demonstration of how arbitrary unity can be.  They are good examples.

Quote
You do not seem to understand the concept of unity gain.

I do understand it.  I understand it to be false and simplistic.

Quote
As Roger explained, if you use unity gain to compare the sensitivity of sensors, you must express it in the same bit depth.

So, what's the magic bit depth?  12, because it's one of God's favorite numbers?

Maybe we can get back to ancient Greek cosmologies while we're at it; Pythagorean harmony between bits and electrons.

Quote
Yes, with a unity gain at ISO 300 with the 1D M3 you could leave the ISO at 300 and shoot in dim light without any need to adjust the ISO to a higher value. If you increased the bit depth to 17, you wouldn't even need to adjust the ISO at all, since you would have captured all the information at base ISO. This assumes that the read noise at base ISO is kept under control. With the 1DM2, Roger's data show that read noise increases considerably at at base ISO as compared to ISO 1600.

The point that you and Roger seem to ignore is that it is the increased amplification at the photosites that reduces the noise in electron units.

Obviously, just counting electrons would be the ideal; ISO would only be a way to tell the camera how to expose, and how bright to make the review image or the default JPEG.

Quote
With the Nikon D200, the increase is less marked (up to 10 electrons from 7.4).
[a href=\"index.php?act=findpost&pid=129247\"][{POST_SNAPBACK}][/a]

You could double-check yourself.  Just shoot short blackframes and load them into IRIS, and issue the "stat" command.  Look at the Sigma value.  That is the standard deviation of clipped black.  To get the true noise, multiply by 1.725, which will result in what a symmetrical, unclipped standard deviation would be.

Take each of these values, and multiply them by the number of maximum electrons at RAW saturation, and divide by 4095 (yes, the D200 uses all 4096 values).  I don't know the exact saturation of the D200, but I would guess about 30000 electrons at base ISO.  The correct number is not required to compare ISOs, though.
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Shooting to the right and raw conversion
« Reply #75 on: July 23, 2007, 12:27:41 pm »

Quote
To get the true noise, multiply by 1.725, which will result in what a symmetrical, unclipped standard deviation would be.
What is the origin of this number?
Logged
Jan

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #76 on: July 23, 2007, 01:05:43 pm »

Quote
What is the origin of this number?
[a href=\"index.php?act=findpost&pid=129536\"][{POST_SNAPBACK}][/a]

I took a bunch of symmetrical gaussian noise samples (and fabrications in PS), measures their standard deviations, clipped them in the center, and measured again.  1.725 was typical of the ratios in the useable range.
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #77 on: July 23, 2007, 09:30:44 pm »

Quote
With the Nikon D200, the increase is less marked (up to 10 electrons from 7.4).
[a href=\"index.php?act=findpost&pid=129247\"][{POST_SNAPBACK}][/a]

I've just been looking at some D200 RAW files, ISO 100 and 1600, and it doesn't seem like those values are likely.  It would be interesting to see what values you get for blackframes from the D200.

Or anyone with a D200.
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Shooting to the right and raw conversion
« Reply #78 on: July 24, 2007, 04:09:04 pm »

Quote
First off, I'd like to point out that John Sheehy is absolutely correct in that the camera sensors mentioned by Mr. Clark do not count individual electrons.

Counting single electrons is so hard that it was first done just over two years ago by Swedish physicists.

If John Sheehy doesn't understand the concept of unity gain, it may be because it's practically impossible to have a setting in the camera where it can count single electrons.

If bjanes wants to prove Roger Clark right (or he wants to do so himself), then my guess is that there's a Nobel Prize in physics waiting; doing this at room temperature would be a significant leap forward.
[a href=\"index.php?act=findpost&pid=129357\"][{POST_SNAPBACK}][/a]

No one has claimed that the camera is counting electrons individually. However, at high ISO we are working with relatively few electrons in the shadows. As an example, the table below is for the Nikon D200 and is derived from Roger Clark's sensor analysis of that camera.The table shows data for ISO 1600 from highlights to shadows in decrements of 1 f/stop. The number of electrons collected, the shot noise, the read noise, and total noise are shown along with the signal to noise ratio. All noise values are in electrons.

The full well capacity of the camera is about 33,000 electrons. However, at ISO 1600 the full well is not utilized and only about 2043 electrons are collected at full scale on the ADC. (output value = 4095 or thereabout). Seven stops down in the shadow area, only 16 electrons are collected. The total noise is 8.4 electrons and this represents the noise fog to which John Sheehy refers. Since the gain is 0.5 electrons per 12 bit ADU, the ADU output would be 16, corresponding to an 8 bit gamma 2.2 value of 28. This is Zone 8, and near where the blackpoint would be set in a normal photograph. We are not counting individual electrons, but are coming close.  Since unity gain for the D200 is at ISO 800, we could have left the camera at ISO 800 and increased exposure in the raw converter. That wouldn't win a Nobel prize, but it might us to make better use of the camera.

Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Shooting to the right and raw conversion
« Reply #79 on: July 24, 2007, 07:33:46 pm »

Quote
No one has claimed that the camera is counting electrons individually.

That would be necessary, though, for Unity Gain to be fully meaningful.  With a great deal of uncertainty in the counting of electrons, the only thing that should limit the usefulness of amplification is no further decrease in analog noise, as measured in units of electrons.

Quote
However, at high ISO we are working with relatively few electrons in the shadows. As an example, the table below is for the Nikon D200 and is derived from Roger Clark's sensor analysis of that camera.The table shows data for ISO 1600 from highlights to shadows in decrements of 1 f/stop. The number of electrons collected, the shot noise, the read noise, and total noise are shown along with the signal to noise ratio. All noise values are in electrons.

It would be nice if someone measured D200 blackframe noise at all ISOs in IRIS, which loads literal RAW data.  I am skeptical about the software and flow that Roger uses for measuring RAW statistics.

10.0 to 7.4 electrons seems pretty low; that would make the D200 as clean as the 1Dmk3 at black at ISO 100, and I just don't see that in the RAW images.  As I zoom into the D200 shadows, I see the noise become visible much quicker, and in out-of-focus deep shadow areas, the read noise seems to be about 3 ADU, which is 33000*3/4095 = about 24.2 electrons, a more typical value for a DSLR at ISO 100.

Quote
The full well capacity of the camera is about 33,000 electrons. However, at ISO 1600 the full well is not utilized and only about 2043 electrons are collected at full scale on the ADC. (output value = 4095 or thereabout). Seven stops down in the shadow area, only 16 electrons are collected. The total noise is 8.4 electrons and this represents the noise fog to which John Sheehy refers. Since the gain is 0.5 electrons per 12 bit ADU, the ADU output would be 16, corresponding to an 8 bit gamma 2.2 value of 28. This is Zone 8, and near where the blackpoint would be set in a normal photograph. We are not counting individual electrons, but are coming close.

I don't see how you come to that conclusion.  Do you know what a sigma of 7 or 8 electrons means?  It doesn't mean that most counts are right, and occasionally one will be 3.5 - 4 electrons off.  If the average signal is 15 electrons, you will have many pixels clipping at zero, and going as far off as 40 or 50 electrons worth of analog signal.  It's a big mess.

And as I've said earlier, these RAW levels are not what you might think they are.  The top stop isn't even used in the green channel with matte subjects in even lighting, at the rated ISO, and even more isn't used in the blue and red in daylight, and three stops get clipped away normally in the blue channel in incandescent.  That puts middle blue at 6.5 stops below saturation in incandescent light.  Sure, you can expose a stop to the right, but you are no longer actually at the stated ISO.  Every camera has a highest ISO, and to use Av and Tv values that are needed may mean shooting way down in the RAW signal, in at least one color channel.

Quote
Since unity gain for the D200 is at ISO 800, we could have left the camera at ISO 800 and increased exposure in the raw converter. That wouldn't win a Nobel prize, but it might us to make better use of the camera.
[a href=\"index.php?act=findpost&pid=129758\"][{POST_SNAPBACK}][/a]

You could probably push ISO 400 to 1600 as well, with little loss, but the converter must not treat the midtones as shadows, as ACR does, for example, if you leave the Shadows slider at 5.

I don't see "unity gain" explaining anything here.  If the results are the same (except for mild quantization with the pushes), then the real gain at the first stage may very well be the same for all ISOs, and the D200 doesn't do unity gain, but only 1:8 gain.
Logged
Pages: 1 2 3 [4] 5   Go Up