Pages: 1 ... 4 5 [6] 7 8 9   Go Down

Author Topic: Sony A900 noise, dynamic range and noise reduction  (Read 85070 times)

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #100 on: March 17, 2009, 12:49:55 pm »

Quote from: Iliah
> "it" can be influenced by NR selection

Please demonstrate it.

> The green channels would be equally affected.

Please demonstrate this too
Sony A900 - the effect of noise reduction on the raw data is doing just that.

Quote
what I read in your post is a lot of statements without any proof
Interesting; I see every statement pretty well proven.

Quote
all subjective interpretation
Really? What about some specifics?

My qualification of the noise reduction as crude is certainly subjective. Isn't it interesting that you used the same term?

Quote
send your study to Sony and see what they will answer you
Why would I? I am not working for them, nor do I own the camera.
Logged
Gabor

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
Sony A900 noise, dynamic range and noise reduction
« Reply #101 on: March 17, 2009, 01:13:05 pm »

I appreciate all the hard work, Gabor, and I'm really interested in what's going on with the A900 at low ISO, but since Iliah seems to be implying that what we're seeing is caused by hardware, and, in the past you've said, "I have zero competence on the hardware area," I'm having a hard time with your assertions, and consider them dubious without further proof.  If you are correct, then Sony should take care of this ASAP.
« Last Edit: March 17, 2009, 01:14:14 pm by douglasf13 »
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #102 on: March 17, 2009, 01:19:17 pm »

http://www.dyxum.com/dforum/forum_posts.as...=43626&PN=6

Originally posted by qbic

Is it possible that "blobs" are only effects of data quantization at low bit resolutions? When you amplify low resolution signal you will get "blocks"

Your reply:

What you are saying is, that the sensor yields data enough only for let's say ten bits, and that gets "blown up". It's a bit thin for a top of the line camera.

Anyway, if that were so, then

a. the same phenomenon would be observable in the brighter areas as well,

b. the green channel would exhibit the same phenomenon,

c. the effect would not depend on the NR selection.

Now, from the current thread:
Your statement:
The green channels would be equally affected.
Me:
Please demonstrate this too
You:
Sony A900 - the effect of noise reduction on the raw data is doing just that.

Something does not compute here.

And of course I see you are not looking analytically on your own histogram. The histograms, by the way, are typical for level drops and posterization.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #103 on: March 17, 2009, 01:22:31 pm »

We sent to Sony engineers a brief summary of findings from practical shooting situations about a month ago, and detailed analysis based on shooting targets under strictly controlled conditions yesterday. I suggest once again Gabor to contact Sony directly.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Sony A900 noise, dynamic range and noise reduction
« Reply #104 on: March 17, 2009, 01:49:02 pm »

Hi,

Here is a link to the "Lighthouse image" in RAW

https://rcpt.yousendit.com/664943793/8e93f5...e5b8adac573e337



Quote from: Panopeeper
Erik, what about uploading this raw file?
Logged
Erik Kaffehr
 

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #105 on: March 17, 2009, 02:11:39 pm »

Quote from: ErikKaffehr
Hi,

Here is a link to the "Lighthouse image" in RAW

https://rcpt.yousendit.com/664943793/8e93f5...e5b8adac573e337

Please set you camera to ISO 320 instead of 200.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Sony A900 noise, dynamic range and noise reduction
« Reply #106 on: March 17, 2009, 03:08:20 pm »

Iliah,

I have read your recommendation but I don't really understand it. I'm prepared to listen. Can you explain in what way 320 ISO would benefit? I also don't really see the benefit of intermediary ISO. A pointer to some other discussion would be just fine!

Best regards
Erik

Quote from: Iliah
Please set you camera to ISO 320 instead of 200.
Logged
Erik Kaffehr
 

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #107 on: March 17, 2009, 03:43:59 pm »

Erik,

regarding the lighthouse shot:

1. if you want to see or demonstrate the camera's DR and noise capability, you should *always* set the Black to 0 in ACR; otherwise you are simply cutting off the most important parts. Pure black is always noiseless.

2. with blacks = 0, pls load the image in PS and take a look at the really dark areas, like the bottom right corner. Switch to channel mode and look at only the red and the green channel, alternating. You may want to boost the brightness before. You will notice, how much more detail is in the green channel.

I think Sony chose this way of noise reduction (in this case the implicite NR) with good reason:

a. the green channel, which is higher than the others in the majority of cases and thus less noisy, remains unchanged; it carries the details,

b. the other channels, which are more often noisy, get "smoothened", leading to posterization/lack of details, but the green will make good for much of it.

The downside is blotchiness, but only in the very shadows, and visible only on smooth areas.
Logged
Gabor

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #108 on: March 17, 2009, 03:53:07 pm »

Quote from: Iliah
Now, from the current thread:
Your statement:
The green channels would be equally affected.
Me:
Please demonstrate this too
You:
Sony A900 - the effect of noise reduction on the raw data is doing just that.

Something does not compute here
Well, I have no reason to convince you to actually *read* that paper instead of only talking about it.

Quote
And of course I see you are not looking analytically on your own histogram. The histograms, by the way, are typical for level drops and posterization.
It is simply amazing (and amusing), that you want to know what I see in the histograms, even without having read what I had to say about them.
Logged
Gabor

Plekto

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 551
Sony A900 noise, dynamic range and noise reduction
« Reply #109 on: March 17, 2009, 03:54:38 pm »

Quote
a. the same phenomenon would be observable in the brighter areas as well,
What we are seeing appears to be the reverse of highlight clipping.  ie - A typical sensor blows out the highlights and puts 255 for brightness.  Game over.  This appears to be that when there's too little data in dark areas, it goes off the deep end and adds in noise by mistake.  Over-aggressive NR trying to reduce stuff that's really not there, maybe?

Quote
b. the green channel would exhibit the same phenomenon,
There are twice as many green sensors in a Bayer pattern, so the effect would be about 1/4 as pronounced.  Which is what we see.  There's enough data to keep the green channel "clean" at low levels.  This points to the sensor either not being sensitive enough in low light or a problem with the internal software/converters.

Quote
c. the effect would not depend on the NR selection.
NR appears to be non-functional and a gimmick for anything under ISO800, at least according to the photos and examples that I have seen.  Usually NR makes an enourmous and immediate impact when pixel-peeping.  Yet we see virtually no change at all at ISO100, 200, or 400.  What gives?

Perhaps the issue is simpler, then?  Maybe Sony shipped the first batch of cameras with software that automatically has the NR set to maximum no matter what you're actually telling it to do?  At least I hope it's as simple as this.


I'd like to also add that the clipped histograms are immediately obvious.  The D200 and the A900 both have this chunky clipped pattern that jumps out at you.  (as said, it's not only Sony that appears to have bad in-camera processing from time to time)
« Last Edit: March 17, 2009, 03:57:51 pm by Plekto »
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #110 on: March 17, 2009, 04:05:08 pm »

Quote from: douglasf13
I appreciate all the hard work, Gabor, and I'm really interested in what's going on with the A900 at low ISO, but since Iliah seems to be implying that what we're seeing is caused by hardware, and, in the past you've said, "I have zero competence on the hardware area," I'm having a hard time with your assertions, and consider them dubious without further proof
That's ok with me; never mind, that the validity of none of my assertions in that paper depends on the question if the actions are performed in firmware or in hardware. As the matter of fact, I have not mentioned this aspect at all.
Logged
Gabor

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Sony A900 noise, dynamic range and noise reduction
« Reply #111 on: March 17, 2009, 04:08:10 pm »

Hi Gabor,

The idea with the picture was not to demonstrate anything but to show a simple example of a very high contrast target with normal processing. Now that I have published the raw image anyone can interprete it to her/his own liking.

Best regards
Erik

Quote from: Panopeeper
Erik,

regarding the lighthouse shot:

1. if you want to see or demonstrate the camera's DR and noise capability, you should *always* set the Black to 0 in ACR; otherwise you are simply cutting off the most important parts. Pure black is always noiseless.

2. with blacks = 0, pls load the image in PS and take a look at the really dark areas, like the bottom right corner. Switch to channel mode and look at only the red and the green channel, alternating. You may want to boost the brightness before. You will notice, how much more detail is in the green channel.

I think Sony chose this way of noise reduction (in this case the implicite NR) with good reason:

a. the green channel, which is higher than the others in the majority of cases and thus less noisy, remains unchanged; it carries the details,

b. the other channels, which are more often noisy, get "smoothened", leading to posterization/lack of details, but the green will make good for much of it.

The downside is blotchiness, but only in the very shadows, and visible only on smooth areas.
Logged
Erik Kaffehr
 

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #112 on: March 17, 2009, 04:23:05 pm »

Quote from: ErikKaffehr
Iliah,

I have read your recommendation but I don't really understand it. I'm prepared to listen. Can you explain in what way 320 ISO would benefit? I also don't really see the benefit of intermediary ISO. A pointer to some other discussion would be just fine!

Best regards
Erik

We use to say "try and see"  The explanation as it seems is that if the ADC is fed with pre-amplified signal it copes better. From our experiments on 3 cameras we see IS) 320 to be consistently the optimum. We push further in raw converter as needed.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #113 on: March 17, 2009, 04:24:32 pm »

Quote from: Panopeeper
Well, I have no reason to convince you to actually *read* that paper instead of only talking about it.


It is simply amazing (and amusing), that you want to know what I see in the histograms, even without having read what I had to say about them.

We read your paper carefully, several times, and spent 2 hours discussing it.
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #114 on: March 17, 2009, 04:30:51 pm »

Quote from: Plekto
This appears to be that when there's too little data in dark areas, it goes off the deep end and adds in noise by mistake
No, this is not the case. As I posted already, it is not black clipping.

Quote
There are twice as many green sensors in a Bayer pattern, so the effect would be about 1/4 as pronounced.  Which is what we see
I don't see that, and you don't either. Btw, the proportion of the green pixels has nothing to with that aspect. (Though it does have to do with my suspicion posted above to Erik.)

Quote
NR appears to be non-functional and a gimmick for anything under ISO800, at least according to the photos and examples that I have seen.  Usually NR makes an enourmous and immediate impact when pixel-peeping.  Yet we see virtually no change at all at ISO100, 200, or 400.  What gives?
A change between what? Between the omnipresent NR and the omnipresent NR?

Anyway, you see the same pictures again differently than I see them, for example in the samples rendered by ACR

Look at the fourth (the blue) and the last two strips.

Quote
Perhaps the issue is simpler, then?  Maybe Sony shipped the first batch of cameras with software that automatically has the NR set to maximum no matter what you're actually telling it to do?  At least I hope it's as simple as this
This is not impossible., but the difference should be obvious in the firmware release. Is there any new firmware out there? I have raw files from five cameras, all are version 1.

Quote
The D200 and the A900 both have this chunky clipped pattern that jumps out at you
I really don't know what you are referring to. I have not observed anything like that on D200 raw data, but my collection is quite limited.
Logged
Gabor

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Sony A900 noise, dynamic range and noise reduction
« Reply #115 on: March 17, 2009, 04:58:41 pm »

OK, I'll check, I changed two om my presets to ISO 320. That said I personally never had any problem with noise at low ISO on the A900.

BTW, in some of your writings I have seen that you were of the opinion that the biggest difference between the Nikon D3x and the A900 that the Nikon captures more photons. If we assume that the sensels are similar I would presume that the difference would depend on the color filter array (better transmission on the Nikon) or the microlenses which my differ. What's your taken on this?

By the way I downloaded RPP and play around with it, I think I also made a donation but I'm not really sure. For the time being I'll stay with LR as main application in many ways it's the application I would have written if my time and knowledge has allowed me.

Best regards
Erik

Quote from: Iliah
We use to say "try and see"  The explanation as it seems is that if the ADC is fed with pre-amplified signal it copes better. From our experiments on 3 cameras we see IS) 320 to be consistently the optimum. We push further in raw converter as needed.
Logged
Erik Kaffehr
 

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
Sony A900 noise, dynamic range and noise reduction
« Reply #116 on: March 17, 2009, 05:05:50 pm »

I felt the same way about lightroom, Erik, but I've finally given it up completely. LR/ACR mishandles A900 like no other, and I figure there's no reason to be interested in the nuances of all these FF cameras if I'm using a poor conversion.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Sony A900 noise, dynamic range and noise reduction
« Reply #117 on: March 17, 2009, 05:32:09 pm »

So, what are you using? Aperture?

Best regards
Erik

Quote from: douglasf13
I felt the same way about lightroom, Erik, but I've finally given it up completely. LR/ACR mishandles A900 like no other, and I figure there's no reason to be interested in the nuances of all these FF cameras if I'm using a poor conversion.
Logged
Erik Kaffehr
 

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
Sony A900 noise, dynamic range and noise reduction
« Reply #118 on: March 17, 2009, 05:41:29 pm »

Quote from: ErikKaffehr
So, what are you using? Aperture?

Best regards
Erik

  I'm on PC, so I'm using RAW Therapee and C1.  The bulk of my work is with C1, because, while it may give up a little to RAW Therapee, it is still much better than LR, and it's workflow is not too bad.  Anyways, sorry to get off topic...
« Last Edit: March 17, 2009, 05:42:09 pm by douglasf13 »
Logged

douglasboyd

  • Newbie
  • *
  • Offline Offline
  • Posts: 32
    • http://dboyd.com
Sony A900 noise, dynamic range and noise reduction
« Reply #119 on: March 17, 2009, 05:45:54 pm »

Dear Panelpeeper,

I have just read this thread and its attachments with interest, since I have an A900 and I have a possible explanation based on my experience in medical imaging.

In medical image processing we sometimes use a technique called "adaptive smoothing".  This is a non-linear process that clips off outlying values from a stream of digital data.  It is very helpful to fix occasional hardware blips where the ADC sends out a wrong value.  It can also be used to reduce noise.  In this process the value of the current reading is compared to the value of the previous and following pixel (or pixels) and the 2nd derivative computed.  If the value exceeds the expected trend by a particular amount (say 1 or 2 std. dev.), then it is clipped to the expected value, say the mean slope adjusted value.  In a uniform region such as with your color rectangles, the clipping would be to the mean of the adjacent values.

The shape of your histograms in the red and blue channels is highly suggestive of this kind of processing, as you can see the histogram width is reduced by clipping in from the tails.  In medical imaging this works fairly well because the image is monochromatic, and the process selectively works on the noisier parts of the image (dark areas) where statistical noise is greatest.  But in digital photography this would be a terrible method of noise reduction, since the intensity of the blue, red, and green channels, and hence noise, depends on color as well as intensity.  Thus if imaging a red rectangle, the blue and green channels would be noisy even if the intensity is not low.  This will cause blotchiness in the image as you have described, not only in the shadows, but also in regions with near saturation of red, blue, or green.

A second problem of this kind of noise reduction should occur at edges.  An "outlyer reduction" algorithm, whether it tests 2nd derivative or something else can easily be fooled at a sharp edge and cause artifacts.  This would be something to look for in images, but as far as I know, this has not been seen yet.  Probably the best way to look for it would be with ISO1600 and NR set to high and an image of a wire against the sky.

In any case I love the image quality I am getting from my A900, and the noise cleans up very well in Neat Image.  By the way, according to my understanding Neat Image works by filtering the image with the inverse of the FFT pattern that Gluijk has shown elswhere in this thread.  But of course Neat can not remove the blotchiness and posterization caused by A900 noise reduction.  Thankfully this is usually a very tiny effect, but I agree it would be good to reduce it even further.

==Doug




Quote from: Panopeeper
Here are the findings in condensed form:

1. Noise reduction Off, Low, Normal and High are identical from ISO 100 to 800, ON THE RAW DATA. They correspond something like NR Low @ ISO 1600.

In other words: all reviews used pre-NRed samples.

However, NR Off means almost Off with ISO 1600 (this explains the "sudden" increase of noisiness at ISO 1600).

2. The noise reduction affects mainly the red and the blue channels, much less the green. This is a very primitive noise reduction; it simply eliminates some pixel levels in the affected areas.

3. Due to the nature of this noise reduction, the very dark, noise reduced areas become darker. The consequence is, that not only the visual appearance but the noise measurement too indicates not only lower noise that it would be without NR, but that in deeper shadow, suggesting a greater dynamic range.

The magnitude of this shift is about 0.6-0.7 EV between the red/blue and the green channel. As the green channel itself is not virgin either, the shift must be even greater.

I estimate that in end effect the noise gets shifted and the DR "enhanced" by at least one full stop.

A side effect of the above is the non-linearity of the pixel values, and a WB shift in the low ranges: if the WB is correct in the "normal" ranges, it is certainly incorrect in the very dark range.

The nature of this noise reduction explaines the blotchiness as well. This will be appearant from the coming demonstrations.
Logged
Pages: 1 ... 4 5 [6] 7 8 9   Go Up