Pages: 1 [2] 3   Go Down

Author Topic: ISO/Noise testing  (Read 25856 times)

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #20 on: March 04, 2011, 02:00:53 am »

Honestly, while I admire you guys with the talent for scientific analysis, I do not have neither the patience nor sufficient knowledge. So I did this extremely crude experiment, and I admit I did not give it a lot of thought, so please tell me what I did wrong.

I put a cap on the lens, used a manual 1/125s and f/5.6 and shot like that at every ISO. Then, without even loading it into the computer, I checked the file size on the camera's (Canon 40D) LCD and plotted them on a graph. My assumption is that the variations in file sizes have to do with noise only. And, not surprisingly, I got the same results as everybody else: !60 has the least noise, and the best ISOs are 160, 320, 640 and 1250.

Is it safe to conclude that for the best performance (noise and everything else) under sufficient light I should use ISO 160 (and not 100), and then if I need more speed to jump to 320-640-1250?

What you are testing is the compressibility of the raw data.  This correlates to the std dev (the spread) of raw levels in the images you took; the more uniform the neighboring values, the more the data can be compressed -- for instance, if all values are the same, we could encode the difference of one pixel from the next, which is zero, and file full of zeros compresses very easily.  With the lens cap on, you are looking at the std dev of read noise in raw levels.  For noise in an image, the figure of merit is S/N ratio.  The signal is photons, so what is more important than noise in raw levels is noise in photon units.  For that one needs to know how many extra photons an increase in one raw level amounts to.  This is inversely proportional to ISO, since ISO is an amplification; a raw level at ISO 100 is more photons than the same raw level at ISO 160.  Therefore, under the stated conditions -- sufficient light so that you can center the histogram no matter what ISO you are shooting at -- the S/N ratio will be higher at ISO 100 than at 160; while the read noise is lower in raw levels at 160, the signal at ISO 100 for a given raw level represents more photons, and the S/N ratio is actually higher.  More photons trumps less read noise.


Logged
emil

joofa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 544
Re: ISO/Noise testing
« Reply #21 on: March 04, 2011, 02:23:59 am »



I know which one I would prefer having to deal with in an image.

Hi Emil,

We don't do flat patches, and would welcome your graduation out of "baby signal processing", into the world of real images ;D.

But, more seriously, I have interpretted Erik's remarks (ref to "salt and pepper") to mean John Sheehy's usual statements on Dpreview regarding the "texture" of shot noise having anything to do with the signal. So, the basic premise was in showing that Shot noise has no correlation to signal. It has nothing to do with your personal ease to deal with it.

Ref.: http://forums.dpreview.com/forums/read.asp?forum=1032&message=37062241

Sincerely,

Joofa

« Last Edit: March 04, 2011, 02:49:58 am by joofa »
Logged
Joofa
http://www.djjoofa.com
Download Photoshop and After Effects plugins

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #22 on: March 04, 2011, 08:37:23 am »

Hi Emil,

We don't do flat patches, and would welcome your graduation out of "baby signal processing", into the world of real images ;D.

But, more seriously, I have interpretted Erik's remarks (ref to "salt and pepper") to mean John Sheehy's usual statements on Dpreview regarding the "texture" of shot noise having anything to do with the signal. So, the basic premise was in showing that Shot noise has no correlation to signal. It has nothing to do with your personal ease to deal with it.

Ref.: http://forums.dpreview.com/forums/read.asp?forum=1032&message=37062241

Sincerely,

Joofa


Flat patches indicate the degree of corruption of the image by noise, without it being masked by signal variation. 

My interpretation of Erik's remarks were that shot noise is more uniform in character spatially, as opposed to the patterning and impulsiveness that characterize read noise on many DSLR's.

I can do without the gratuitous insults, BTW.
Logged
emil

joofa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 544
Re: ISO/Noise testing
« Reply #23 on: March 04, 2011, 09:09:49 am »


I can do without the gratuitous insults, BTW.

I'm sorry if you felt so, and apologize. However, I also find it ironic that you would consider the long list of adjectives that you have assigned to me on various forums over the years as otherwise. But, sorry, still.

Sincerely,

Joofa
« Last Edit: March 04, 2011, 09:16:12 am by joofa »
Logged
Joofa
http://www.djjoofa.com
Download Photoshop and After Effects plugins

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #24 on: March 04, 2011, 10:36:13 am »

But, more seriously, I have interpretted Erik's remarks (ref to "salt and pepper") to mean John Sheehy's usual statements on Dpreview regarding the "texture" of shot noise having anything to do with the signal. So, the basic premise was in showing that Shot noise has no correlation to signal.

Ref.: http://forums.dpreview.com/forums/read.asp?forum=1032&message=37062241


I suspect what John is talking about is that shot noise variance is correlated to signal, as your example shows.  You don't need multiple samples to see that; for instance,


Logged
emil

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #25 on: March 04, 2011, 10:57:34 am »

I took the time today to test the noise from L thru H2 with the 5DM2 and was stunned to see the surprises that were the result. ie less noise @ iso 640 than iso 125. ; less @iso160 than 100.  Does anyonyone know if this is across the board , or camera sensor to camera sensor.

I used the manual, cap on method for test shots, opened raw files @ 100% and did auto process  to compare blind...the results matched the file size in metadata... I have not figured out how to do the uncapped test, but so far based on this it appears I need to rethink my iso settings...


As I mentioned in my reply to Slobodan, the real figure of merit is how much noise there is relative to signal; what is colloquially called noise in images is really the noise relative to the level of illumination, and since one is typically using lower ISO when there is a higher ambient illumination, noise/signal will be lower at lower ISO -- it typically is increasing faster than the electronic read noise (which is what your lens cap tests measured) is increasing.  And no, it is not the same for all cameras; for instance, the DxO plot of dynamic range as a function of ISO is correlated to the amount of read noise -- if the plot is linear (as it tends to be with DSLRs using the Sony Exmor sensors), read noise in units of photons is relatively constant; if it flattens at low ISO, that means that read noise in photons equivalent goes up at low ISO.  Your 5D2 is one of the worst offenders (along with other Canon DSLRs); read noise is quite high at ISO 100, more than four times as much as ISO 800 in photon equivalents.  It's just that at ISO 800, you typically have 8x less light than a shooting situation that calls for ISO 100, so if you can adjust the exposure to make use of the lower ISO, you win.  Note also that this discussion pertains to read noise; photon shot noise (random fluctuations in signal intensity due to the fact that light is composed of discrete photons) varies as the square root of signal, and so noise/signal at ISO 100 due to this source is about sqrt[8]~2.8 times less at ISO 100 than ISO 800 and so you win even more than with read noise on your 5D2.  And since read noise is only dominant in deeper shadows at low ISO, it's the shot noise scaling that is usually more relevant.
« Last Edit: March 04, 2011, 11:05:21 am by ejmartin »
Logged
emil

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: ISO/Noise testing
« Reply #26 on: March 04, 2011, 11:24:43 am »

I know which one I would prefer having to deal with in an image.

Hi Emil,

Indeed. However, the "read noise" example consists of more than just read noise, specifically pattern noise and PRNU. The result of subtracting two black frames (divided by Sqrt(2), and adding an offset) will show a better approximation of real read noise. AFAIK, its distribution looks a lot like Poisson/Gaussian noise. but you are correct that in every day use, lots of photons will produce a nicer noise pattern than some sensor arrays do with lower signal levels (unless we do postprocessing of multiple frames at the Raw level before demosaicing).

I agree with your suggestion that ISO 100 can produce a smoother image than ISO 160, even if "read noise" seems lower at ISO 160, purely due to the more random nature and higher S/N of shot noise. You also are right in mentioning that the Photon statistics are altered a bit by the quantization differences at different ISOs (up to unity gain).

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #27 on: March 04, 2011, 11:39:30 am »

Hi Emil,

Indeed. However, the "read noise" example consists of more than just read noise, specifically pattern noise and PRNU. The result of subtracting two black frames (divided by Sqrt(2), and adding an offset) will show a better approximation of real read noise. AFAIK, its distribution looks a lot like Poisson/Gaussian noise. but you are correct that in every day use, lots of photons will produce a nicer noise pattern than some sensor arrays do with lower signal levels (unless we do postprocessing of multiple frames at the Raw level before demosaicing).

I agree with your suggestion that ISO 100 can produce a smoother image than ISO 160, even if "read noise" seems lower at ISO 160, purely due to the more random nature and higher S/N of shot noise. You also are right in mentioning that the Photon statistics are altered a bit by the quantization differences at different ISOs (up to unity gain).

Cheers,
Bart

Hi Bart,

What I meant by 'read noise' is all the non-signal added by the camera electronics.  In many cameras it has a substantial patterned component which is hard to deal with using standard NR filters since it looks like a bunch of edges, which the filters are designed to try to preserve.  And the totality of all this non-signal variation is also rather impulsive too, and tends to be preserved as well by the filters.

I have never found 'unity gain' to be a useful concept; current sensors have too much analog noise relative to the quantization step.
Logged
emil

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: ISO/Noise testing
« Reply #28 on: March 04, 2011, 12:01:57 pm »

Hi Bart,

What I meant by 'read noise' is all the non-signal added by the camera electronics.  In many cameras it has a substantial patterned component which is hard to deal with using standard NR filters since it looks like a bunch of edges, which the filters are designed to try to preserve.  And the totality of all this non-signal variation is also rather impulsive too, and tends to be preserved as well by the filters.

Topaz Labs' Denoise 5 does a decent job in reducing the "banding" type of noise, athough it's obviously preferrable not having to deal with it.

Quote
I have never found 'unity gain' to be a useful concept; current sensors have too much analog noise relative to the quantization step.

Well, if anything, we know that below the Unity Gain level it takes more than a single photon to change the ADU or DN. Shot noise is a function of Photons, gain below Unity Gain changes it a bit. Whether we want to attach more meaning to it is up to the individual ;-)

Cheers,
Bart
« Last Edit: March 05, 2011, 07:31:53 pm by BartvanderWolf »
Logged
== If you do what you did, you'll get what you got. ==

joofa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 544
Re: ISO/Noise testing
« Reply #29 on: March 04, 2011, 12:33:20 pm »

I suspect what John is talking about is that shot noise variance is correlated to signal, as your example shows.  You don't need multiple samples to see that; for instance,


I hope that you realize that this is not the same as saying that noise is correlated with the signal. Multiple images were only for people to see what is going on here.

Sincerely,

Joofa
Logged
Joofa
http://www.djjoofa.com
Download Photoshop and After Effects plugins

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #30 on: March 04, 2011, 02:47:16 pm »

I hope that you realize that this is not the same as saying that noise is correlated with the signal. Multiple images were only for people to see what is going on here.

I was giving a generous interpretation of what could be meant by the admittedly non-scientific term 'texture' so that John's statement is valid, rather than an ungenerous interpretation which makes John's statement patently false.
Logged
emil

RFPhotography

  • Guest
Re: ISO/Noise testing
« Reply #31 on: March 05, 2011, 06:43:25 pm »

The only real ISOs in the 5d2 are 100,200,400,800 & 1600. The others are just software.


Actually, the only 'real' ISO in any digital camera is the native ISO of the sensor.  Any other ISO setting is fudged. 

There's an article in the most recent Digital Photo Pro on this very topic and it references a video on Vimeo where someone did a test with a 7D. 

Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: ISO/Noise testing
« Reply #32 on: March 05, 2011, 07:37:28 pm »

Hi,

Please see the graphic below where the image on the top left is treated as representing the "true" signal. On the top right is a sample shot noise simulation derived from the statistics of this image. Seems pretty random to me and devoid of any "natural texture" as claimed.

Hi Joofa,

I'm not entirely sure what you did, but if you can reconstruct the original image from the noise, then the noise isn't random.

Cheers,
Bart

P.S. If your parents gave you a name, what's wrong with sharing it with us?
Logged
== If you do what you did, you'll get what you got. ==

joofa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 544
Re: ISO/Noise testing
« Reply #33 on: March 05, 2011, 09:57:08 pm »

Hi Joofa,

I'm not entirely sure what you did, but if you can reconstruct the original image from the noise, then the noise isn't random.

Hi Bart,

The bottom two images are precisely the visualization of the well-known relation that shot noise standard deviation is sqrt (mean signal). So, noise and signal are related, in some sense, but that does not make shot noise non-random. Because, the noise values are themselves random, but the standard deviationof these values is not random.

If your parents gave you a name, what's wrong with sharing it with us?

If you are in doubt then Google is your buddy  ;D

Sincerely,

Joofa
Logged
Joofa
http://www.djjoofa.com
Download Photoshop and After Effects plugins

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: ISO/Noise testing
« Reply #34 on: March 06, 2011, 09:49:35 am »

The bottom two images are precisely the visualization of the well-known relation that shot noise standard deviation is sqrt (mean signal). So, noise and signal are related, in some sense, but that does not make shot noise non-random. Because, the noise values are themselves random, but the standard deviationof these values is not random.

Hi DJ (?),

Your example in post no. 15 (top right quadrant, labeled "sample shot noise") is a misleading presentation of shot-noise. What you are actually displaying is a large number of pixels/samples, each and every one from Poisson distributed random noise. Each pixel is the product of a different level of signal, hence it has different mean levels with an accompanying probability distribution at each spatial sampling position. You are not showing "sample shot noise", but you are showing multiple samples (one sample at each pixel position). A better label would have been "many shot noise samples", but it would still not show the nature of Poisson (shot) noise, which was the topic.

You also do not mention the fact that the human visual system will notice noise more when the signal levels are spatially more uniform. In parts of the image where the signal spatially fluctuates rapidly, IOW lots of detail, the character of the noise is much harder to appreciate and often less of an issue.

Quote
If you are in doubt then Google is your buddy  ;D

I know of many people on the internet who use someone else's name as a moniker. Even if my real name was Albert Einstein I would still be someone else than the person who most people think of (the real proof being that I am alive). Google links are no proof.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #35 on: March 06, 2011, 11:17:30 am »

Hi Bart,

I believe the upper right is noise sampled from a Poisson distribution for each pixel determined by the upper left image, assumed to be ground truth.  It is thus a sample of shot noise.  The lower images are the variance of several such samples drawn from the distribution, and thus will converge to the ground truth image in the limit of a large number of samples.

As far as names go, here and elsewhere he has always presented himself as DJ Joofa (or rather djjoofa).  If you're going to complain about that, you might as well complain about my using 'ejmartin' (not that it will have any effect  ;D ).
Logged
emil

joofa

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 544
Re: ISO/Noise testing
« Reply #36 on: March 06, 2011, 11:29:43 am »

Hi DJ (?),

Your example in post no. 15 (top right quadrant, labeled "sample shot noise") is a misleading presentation of shot-noise. What you are actually displaying is a large number of pixels/samples, each and every one from Poisson distributed random noise. Each pixel is the product of a different level of signal, hence it has different mean levels with an accompanying probability distribution at each spatial sampling position. You are not showing "sample shot noise", but you are showing multiple samples (one sample at each pixel position). A better label would have been "many shot noise samples", but it would still not show the nature of Poisson (shot) noise, which was the topic.

Hi Bart,

I respectfully beg to disagree. The issue is that a real image is an example of non-stationary Poisson noise process - i.e., in theory, the noise statistics are different at each pixel location, which you yourself noted, hence, the top right image does indeed show a single sample sequence of such non-stationary noise, and as I mentioned in the post you referenced, derived using the top left image as the "true" signal.

You also do not mention the fact that the human visual system will notice noise more when the signal levels are spatially more uniform. In parts of the image where the signal spatially fluctuates rapidly, IOW lots of detail, the character of the noise is much harder to appreciate and often less of an issue.

The way I see is that how to develop a simple model of noise vis-a-vis signal even in such signal fluctuation cases, which is btw what happens in a real image, so that we can move forward to questions such as snr at image level, the effect of resampling on that snr, best possible window sizes in spatial averaging, frequency analysis of non-stationary noise, and all sort of other interesting questions.

Sincerely,

Joofa
« Last Edit: March 06, 2011, 11:37:16 am by joofa »
Logged
Joofa
http://www.djjoofa.com
Download Photoshop and After Effects plugins

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: ISO/Noise testing
« Reply #37 on: March 06, 2011, 11:49:10 am »

Hi Joofa,

As always you contribute a lot of theory that at least I have difficulty to follow. Does the stuff you discuss have relevance for the tools we use in everyday photography or tools that may be around the corner?

If the discussion is purely theoretical it may be better to start a new topic, at least in my humble opinion.

Best regards
Erik




Hi Bart,

I respectfully beg to disagree. The issue is that a real image is an example of non-stationary Poisson noise process - i.e., in theory, the noise statistics are different at each pixel location, which you yourself noted, hence, the top right image does indeed show a single sample sequence of such non-stationary noise, and as I mentioned in the post you referenced, derived using the top left image as the "true" signal.

The way I see is that how to develop a simple model of noise vis-a-vis signal even in such signal fluctuation cases, which is btw what happens in a real image, so that we can move forward to questions such as snr at image level, the effect of resampling on that snr, best possible window sizes in spatial averaging, frequency analysis of non-stationary noise, and all sort of other interesting questions.

Sincerely,

Joofa
Logged
Erik Kaffehr
 

Slobodan Blagojevic

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 18090
  • When everyone thinks the same, nobody thinks
    • My website
Re: ISO/Noise testing
« Reply #38 on: March 06, 2011, 12:27:19 pm »

... With the lens cap on, you are looking at the std dev of read noise in raw levels...

To continue Erik's point (about real-world relevance), here is my next question for the squints (aka scientists :)): ok.. so it is a read noise, but even in that case, why is there less noise at 160 than at 100 (and so on) and what (if any) real-world relevance does it have?

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
Re: ISO/Noise testing
« Reply #39 on: March 06, 2011, 01:42:48 pm »

To continue Erik's point (about real-world relevance), here is my next question ... so it is a read noise, but even in that case, why is there less noise at 160 than at 100 (and so on) and what (if any) real-world relevance does it have?

There is less read noise in RAW levels at 160 than there is at 100 for eg the 5D2.  This is because the amplifier/ADC noise is about the same at ISO 200 as it is at ISO 100, and ISO 160 is obtained from ISO 200 by multiplying all the output values by 0.8 so the noise in raw levels will be about 20% less at 160 relative to 100.  However, the photon count that saturates the raw data is 60% less at 160 than at 100.  So yes, 20% less read noise at the cost of 60% less photon capacity.  If you have 60% less photons, then it makes sense to use 160 instead of 100.  If your shooting conditions allow the higher exposure, better to use 100 -- the benefit of more photons outweighs the disadvantage of higher read noise in raw levels.  It's clearer if one translates the read noise into photon equivalents, then one can see what is going on since the signal and noise are denominated in the same units.  Just because the read noise is lower in raw levels doesn't mean much, since the meaning of a raw level in terms of absolute exposure changes in proportion to ISO.  Again, the goal is the highest S/N ratio in actual photographs, not the lowest noise with no signal (unless your interest is photographing the inside of your lens cap as an end in itself).
Logged
emil
Pages: 1 [2] 3   Go Up