Pages: 1 ... 6 7 [8] 9   Go Down

Author Topic: Sony A900 noise, dynamic range and noise reduction  (Read 85019 times)

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #140 on: March 19, 2009, 03:02:24 pm »

Quote from: douglasf13
Are you finding the same issues with non-Adobe RAW converters?  Could we see some examples with RAW Therapee, RPP, etc.
I base my findings alone on the raw data before raw conversion. The ACR samples demonstrate only, that these effects do appear in the converted image as well. I don't work with any other raw converter, but I think Erik would not mind that I pass some of his raw files, if someone wants to experiment on them.

The above posted crops showing the loss of detail in the red stoff are from Imaging Resources, one can download the raw files and interpret them at will. I am not authorized to use their images in publications; I hope they pardon the trespassing with the crops (anyway I did not put them in my HTML pages).

The final effect can be even worse with certain other converters without noise reduction, for ACR always carries out some NR even if you select 0.
Logged
Gabor

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #141 on: March 19, 2009, 04:15:32 pm »

Quote from: Panopeeper
As I posted, some reviewer found out the heavy NR on the raw data already last year. One does not need to go deep in the raw data; I presented samples from ACR prooving the NR effect, the details are only for the geeks. Thus the reviewer's oversight was not a calculable, except for DPReview, who don't understand anything of the issues past JPEG.

Btw, the apparent increasing of the DR through the darkening of the deepest shadows is not done in order to fool the measurements, it is inherent to the applied method. However, that should have been noted by everyone, who was *measuring* DR and noise instead of *prattling* about it.

How it is proved that the artefacts are result of noise reduction, and not of something else, like multiplication of data coming from red and blue channels accomplishe in digital domain? )

ACR is not a tool to measure anything, its processing chain is not intended for data analysis.

Darkening of shadows decreases measurable dynamic range and can't fool the measurements.
Logged

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
Sony A900 noise, dynamic range and noise reduction
« Reply #142 on: March 19, 2009, 05:19:19 pm »

Quote from: Panopeeper
I base my findings alone on the raw data before raw conversion. The ACR samples demonstrate only, that these effects do appear in the converted image as well. I don't work with any other raw converter, but I think Erik would not mind that I pass some of his raw files, if someone wants to experiment on them.

The above posted crops showing the loss of detail in the red stoff are from Imaging Resources, one can download the raw files and interpret them at will. I am not authorized to use their images in publications; I hope they pardon the trespassing with the crops (anyway I did not put them in my HTML pages).

The final effect can be even worse with certain other converters without noise reduction, for ACR always carries out some NR even if you select 0.


  Gotcha.  I realize that your findings are based on raw data before conversion, but I don't believe what you're showing in ACR backs that up, because ACR is awful for the A900.  Interestingly, I just saw this little website today, and, although I realize that DxO is far from the purest of raw converters, I thought that is was appropriate to post here, because the author says in reference to the A900 and ACR, "Dark areas often take on a blotchy appearance and it is difficult to increase shadow detail without turning the blacks to gray and ruining the contrast."  Regardless, take this site for what it's worth:   a900 DxO vs. ACR
« Last Edit: March 19, 2009, 06:58:16 pm by douglasf13 »
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #143 on: March 19, 2009, 09:37:46 pm »

Quote from: douglasf13
I don't believe what you're showing in ACR backs that up, because ACR is awful for the A900
You are misleading yourself if you believe that this is an issue with ACR. The details are destroyed by the noise reduction, no software will recover that.

Again: the green channel is supposed to carry the details in this schema. This does work as long as the green is dominant. I attach crops from the same images I posted above, but this time from Rawnalyze, no demosaicing, no conversion, this is the raw data. First NR Off, then NR Normal (NR OFF appears in the file name).

1. The green channel does not contribute to the details on the red cloth (the green is more than a stop darker than the red), but it contributes a bit to the muddiness.

2. The blue does not contribute to the details (it is 1.5 EV darker than the red), but it does contribute a lot to the muddiness.

3. The red carries all details - but the red is seriously damaged by NR Normal.

The last pair is the composite color image.

Now, do show please how much details DXO can extract from the NR Normal version. The raw files are at Imaging Resources, look for AA900hSLI1600.ARW and AA900hSLI1600_NR_OFF.ARW.

Quote
Dark areas often take on a blotchy appearance and it is difficult to increase shadow detail without turning the blacks to gray and ruining the contrast
That's it! Though I have not seen the claim, that DxO solves this problem.
Logged
Gabor

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
Sony A900 noise, dynamic range and noise reduction
« Reply #144 on: March 19, 2009, 11:45:13 pm »

Quote from: Panopeeper
You are misleading yourself if you believe that this is an issue with ACR. The details are destroyed by the noise reduction, no software will recover that.

Again: the green channel is supposed to carry the details in this schema. This does work as long as the green is dominant. I attach crops from the same images I posted above, but this time from Rawnalyze, no demosaicing, no conversion, this is the raw data. First NR Off, then NR Normal (NR OFF appears in the file name).

1. The green channel does not contribute to the details on the red cloth (the green is more than a stop darker than the red), but it contributes a bit to the muddiness.

2. The blue does not contribute to the details (it is 1.5 EV darker than the red), but it does contribute a lot to the muddiness.

3. The red carries all details - but the red is seriously damaged by NR Normal.

The last pair is the composite color image.

Now, do show please how much details DXO can extract from the NR Normal version. The raw files are at Imaging Resources, look for AA900hSLI1600.ARW and AA900hSLI1600_NR_OFF.ARW.


That's it! Though I have not seen the claim, that DxO solves this problem.

  I was referring to the low ISO shots, not high.  My mistake.  That being said, ACR slaughters A900 low ISO shots as well, while other converters don't.


  I realize that NR on ISO 1600+ should be left off.  I guess my confusion is still in how you haven't really proven that the problem of A900 isn't hardware based like Iliah mentioned.

Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #145 on: March 20, 2009, 01:12:56 am »

Quote from: douglasf13
I was referring to the low ISO shots, not high
I have given plenty of proof, that this is happening with ISO 100 to 800 as well, but to a lesser degree than the above demonstration, which was NR Normal. As this can not be turned off, a direct comparison like above is not possible.

Quote
I realize that NR on ISO 1600+ should be left off
It IS off, if you turn it off (well, almost). The issue is not ISO 1600.

Quote
I guess my confusion is still in how you haven't really proven that the problem of A900 isn't hardware based like Iliah mentioned.
You are forcing me to repeat myself.

1. NONE of my assertions depends on this aspect; as the matter of fact, I don't care. The question is relevant only from the perspective if a change can be expected or not.

2. I don't see any need to prove that such a function is not realized in the A/D converter. As this would be extreme (a super-duper A/D converter but what for?), one needs to prove, that Sony wasted the resources for this nonsense (for it can be done much cheaper outside of the sensor).
Logged
Gabor

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Sony A900 noise, dynamic range and noise reduction
« Reply #146 on: March 20, 2009, 08:04:50 am »

Well, you have absolutely no proof that low ISO artefacts are due to noise reduction. You do not realize that ADC used in A900 is unstable at low values, introducing arbitrary error in deep shadows, one of the reason of this error being heat build-up. One absolutely needs to understand hardware issues to attribute artefacts properly.
Logged

Plekto

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 551
Sony A900 noise, dynamic range and noise reduction
« Reply #147 on: March 21, 2009, 01:03:20 pm »

I did mention in the past that it looked like over-amplification to me.  Similar to what you'd see if you over-amplified a TV signal(at least the older analog ones).  Like it's just creating random garbage and ghosting in the signal path and the NR is trying to deal with it.  To me, the blotches and artifacts look too uniform, in a way, to be only something cause by the apparently crummy software.(which we can all agree has serious issues with the NR routines)  WHERE in the signal path it's getting messed up and corrupted, I don't know.  It could honestly be anything in the chain between the sensor and the software.

Every shot, they're there, and almost like a piece of wallpaper or an image etched on glass that's been overlaid on top of the sensor.  The NR seems to alter it, but it's always present.  This suggests another problem is also happening as well.
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #148 on: March 21, 2009, 07:24:19 pm »

Quote from: Plekto
Like it's just creating random garbage and ghosting in the signal path and the NR is trying to deal with it
And this is happening at ISO 100 and 200, but not at ISO 1600? A really strange circuitry.

Quote
To me, the blotches and artifacts look too uniform, in a way, to be only something cause by the apparently crummy software
This is the first time I see a claim, that a mess is "too big to be created by software".

Quote
The NR seems to alter it, but it's always present.  This suggests another problem is also happening as well.
Not at ISO 1600, NR Off. Although there are clear signs of adjustment even @ 1600 with NR Off, but the blobs disappear, taking the blotches with them from the converted image.
Logged
Gabor

aaykay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 359
Sony A900 noise, dynamic range and noise reduction
« Reply #149 on: March 21, 2009, 09:03:15 pm »

Interesting discussion thus far.  

However, sometimes I get the feeling that in any such situation, one can get better results (assuming the intention is constructive) if one does not go ahead and create admittedly provocative threads in every popular photography related forum around, and instead, directly contact Sony, and ask for their explanation on the how/why something was done the way it was.   There are extremely smart people in the team of over 200+ Sony (previously Konica-Minolta) engineers involved in the development of the A900, and I would not assume that I or anyone else I know, is automatically smarter than their collective knowledge.

Many of these things have input from the hardware side and also from the firmware side and without understanding the entire cradle-to-grave scenario and the inter-relationships between all these factors, one could be jumping to conclusions and providing fodder to the less knowledgeable, to diss a product unfairly.  This is the internet after-all.  
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Sony A900 noise, dynamic range and noise reduction
« Reply #150 on: March 22, 2009, 05:41:25 am »

Quote from: aaykay
Interesting discussion thus far.  

However, sometimes I get the feeling that in any such situation, one can get better results (assuming the intention is constructive) if one does not go ahead and create admittedly provocative threads in every popular photography related forum around, and instead, directly contact Sony, and ask for their explanation on the how/why something was done the way it was.   There are extremely smart people in the team of over 200+ Sony (previously Konica-Minolta) engineers involved in the development of the A900, and I would not assume that I or anyone else I know, is automatically smarter than their collective knowledge.

Many of these things have input from the hardware side and also from the firmware side and without understanding the entire cradle-to-grave scenario and the inter-relationships between all these factors, one could be jumping to conclusions and providing fodder to the less knowledgeable, to diss a product unfairly.  This is the internet after-all.  

I think you are being naive to think that Sony engineers are going to be allowed to express their personal opinion on these issues raised by Gabor, or that Sony will admit details of their processes into the public domain, other than details already released for purposes of 3rd party design of RAW converters. At best, there will be an official Sony response dominated by marketing spin.
Logged

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #151 on: March 22, 2009, 11:02:08 am »

Quote from: Ray
I think you are being naive to think ... or that Sony will admit details of their processes into the public domain... At best, there will be an official Sony response dominated by marketing spin.
It took a while, but finally Sony confirmed the NR in the A700 raw data A700 and issued a firmware update.

Quote from: aaykay
one could be jumping to conclusions and providing fodder to the less knowledgeable, to diss a product unfairly
I see the expectation is an official statement from Sony, if the blotches are blotches, if the loss of details is real and the color shift is not due to the inferiority of the monitor. Perhaps the answer is "those are perceptions only or artifacts caused by your monitor. If you were using a Sony Trinitron monitor, you would not see those phenomena".
Logged
Gabor

Plekto

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 551
Sony A900 noise, dynamic range and noise reduction
« Reply #152 on: March 22, 2009, 10:55:55 pm »

Quote from: Panopeeper
And this is happening at ISO 100 and 200, but not at ISO 1600? A really strange circuitry.

I have an odd theooy/question to propose.  What if the "ISO1600" is the actual native resolution/sensitivity of the sensor and they are artificially tweaking and boosting the gain to simulate lower ISO?  So NR off at 1600 would look clean because it doesn't need NR, but turning up the gain as you get to lower ISOs would require more aggressive NR at each level to combat the increased background noise?
« Last Edit: March 22, 2009, 10:56:53 pm by Plekto »
Logged

aaykay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 359
Sony A900 noise, dynamic range and noise reduction
« Reply #153 on: March 23, 2009, 12:23:45 am »

Quote from: Panopeeper
It took a while, but finally Sony confirmed the NR in the A700 raw data A700 and issued a firmware update.

They did not have to "finally confirm" about the NR in the A700.   Sony had freely published that they were doing NR on RAW, when they released the A700.  After being universally panned for that strategy, they came out with a firmware fix, that completely de-activated RAW NR in the A700 (via the "NR off" setting), along with a bunch of other improvements.

One curious factor in the Canon 5DII, is that Canon swiftly came out with a firmware fix, that  masked the "black dot" phenomenon (not present at all in the 21MP FF sensor in the 1DSMKIII).  Is that not a clear indication that the RAW in the Canon is very well cooked and that the "black dot" masking, was again a further cooking of the RAW ?  Any thoughts on that ?
Logged

Josh-H

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2079
    • Wild Nature Photo Travel
Sony A900 noise, dynamic range and noise reduction
« Reply #154 on: March 23, 2009, 12:48:53 am »

Quote
(not present at all in the 21MP FF sensor in the 1DSMKIII)

The 5D MKII and the Canon 1DS MK3 do not use the same sensor - the 5D MKII is a re-design - hence the reason it was and isnt an issue on the 1DSMK3 . ChuckWestfall wrote about this in one of his tech tips.

Looking at the file differences between a 5D MKII and a 1DS MK3 it does appear that the 5D MK2 is cooking more into the RAW. I wouldnt say one is any better than the other - they are just different.
« Last Edit: March 23, 2009, 12:50:44 am by Josh-H »
Logged
Wild Nature Photo Travel

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #155 on: March 23, 2009, 01:25:35 pm »

Quote from: Plekto
I have an odd theooy/question to propose.  What if the "ISO1600" is the actual native resolution/sensitivity of the sensor and they are artificially tweaking and boosting the gain to simulate lower ISO?
Your theories are getting more and more exotic. If the base ISO were 1600, then ISO 100 would cause burning out the top FOUR stops.

Think what is happening at ISOs lower than the base ISO: plain overexposure; for example the A900 overexposes by 2/3 EV at ISO 100, indicating that the native ISO is around 160.

Quote from: aaykay
They did not have to "finally confirm" about the NR in the A700.   Sony had freely published that they were doing NR on RAW, when they released the A700
That's right. I looked it up; the issue was, that there was no "off" setting.

Quote from: aaykay
One curious factor in the Canon 5DII, is that Canon swiftly came out with a firmware fix, that  masked the "black dot" phenomenon (not present at all in the 21MP FF sensor in the 1DSMKIII).  Is that not a clear indication that the RAW in the Canon is very well cooked and that the "black dot" masking, was again a further cooking of the RAW ?  Any thoughts on that ?
The questions are justified, even though this has nothing to do with the actions of the A900.

1. The black dots have been caused by "stray" pixels. This needs an explanation.

Canon's CR2 (and at least one Olympus) creates a raw file containing the pixel values with an offset. This means, that a "black" pixel has the value around 128, 256, 512 or 1024, depending on the model. One theory is, that the A/D increases the voltage by a certain fixed amount, in order to catch irate values (negative charges) as well. Probably the other CMOS sensors are doing the same, but are not passing on the data. (The Nikon D300 is a notable exception.)

In this setting there can be and are pixels far under the "black level"; they will be changed to zero by the raw processing. This is happening all the time; there are a few hundred such pixels in the very dark areas, usually with high ISO.

What was happening with the 5D2? Someone made a test shot directly before and after loading the new firmware version; these shots demonstrate the source of the black dots and the solution.  Anyway, the cause of the black dots were very stray pixels: even with value 0 at the average black level of 1024, i.e. those pixel values represented -1024; they occured always directly at the right end of saturated pixels. These stray pixel values were overall in the negative range but concentrated at -1024, which indicates that they would be even lower but could not be represented.

This looked pretty much like overflow caused by sensor bleeding. I guess this is a limitation of the hardware; other sensors (like the D3/D700) create different phenomena caused by bleeding.

Now, back to the issue: the new firmware "puts these pixels into their context", probably by looking further to the right and using those pixel values.  This is clearly "cooking" of the raw data; the same can be achieved in raw processing (and is done in DPP).  I do not see any likeness to the noise reduction on raw data; I don't have any problem with *this* cooking (the problem is somewhere else: where the stray pixels arize). However, there *is* some other cooking in the 5D2 files, which I can not explain.

Quote from: Josh-H
Looking at the file differences between a 5D MKII and a 1DS MK3 it does appear that the 5D MK2 is cooking more into the RAW
I am interested to see, how you came to this result.

Logged
Gabor

Quentin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1222
    • Quentin on Facebook
Sony A900 noise, dynamic range and noise reduction
« Reply #156 on: March 23, 2009, 02:20:41 pm »

Quote from: Panopeeper
While I find these shots very good, I think it is important to note, that your (or mine or anyone else's) judgement on the dynamic range of a scenery is of secondary relevance compared to the actual measurement.

I can't agree with that.  Actual measurement is subject to a raft of false science and bogus measurements, in my opinion.  Its the reason serious hi fi is not judged by measurement but by ear.

My judgment of the A900 is that it has very wide dynamic range.  Not quite as wide as medium format, but wider than most dslr's.  Having determined that for myself through my own observation and use, all else is irrelevant.

Quentin
Logged
Quentin Bargate, ARPS, Author, Arbitrato

ziocan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 426
Sony A900 noise, dynamic range and noise reduction
« Reply #157 on: March 23, 2009, 10:58:56 pm »

Quote from: Quentin
I can't agree with that.  Actual measurement is subject to a raft of false science and bogus measurements, in my opinion.  Its the reason serious hi fi is not judged by measurement but by ear.

My judgment of the A900 is that it has very wide dynamic range.  Not quite as wide as medium format, but wider than most dslr's.  Having determined that for myself through my own observation and use, all else is irrelevant.

Quentin
I agree. Eye never lies.
What the eye can see, it is all that matter.
All the rest is good stuff for spending time in front of the PC.

Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Sony A900 noise, dynamic range and noise reduction
« Reply #158 on: March 24, 2009, 09:41:48 pm »

Quote from: Quentin
My judgment of the A900 is that it has very wide dynamic range.  Not quite as wide as medium format, but wider than most dslr's.  Having determined that for myself through my own observation and use, all else is irrelevant.
Dynamic range through NR reduction (i.e. through detail loss) is not real DR. If you think it is, then I will give you a secret recipe to maximize your DR: apply to all your pictures a gaussian blur filter with 20 pixels radius. All noise will disappear at any ISO so your DR will be stunning. But please don't complain about loss of detail because that kind of DR is what you wanted.

BR

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
Sony A900 noise, dynamic range and noise reduction
« Reply #159 on: March 24, 2009, 11:21:33 pm »

Quote from: GLuijk
Dynamic range through NR reduction (i.e. through detail loss) is not real DR
There is more to it. This particular method of noise reduction reduces the average pixel values close to black, and that causes the appearance of higher dynamic range: not only that the noise is lower, but that noise appears in deeper shadows, where higher noise is expected.

Even the semi-professional measurements, like DxO, DPR, etc. have been fooled by this effect. Imatest is helpless in this situation.

Btw, I saw a link to a new review, at Digital Camera Info; see the Noise section, particularly following graphs:



(I have no idea, what the percentages mean.)

Notice, that "Low" over ISO 800 is the plain continuation of the line from 200 to 800, not "Off"? These graphs confirm just what I have proven, that the implicite noise reduction under ISO 1600 is comparable to "NR Low" at 1600.

The guy, who created this review does not understand his own graphs:

As shown above, noise reduction processing didn't kick in until ISO 800, at which point it provided a substantial improvement over results with noise reduction turned off
Logged
Gabor
Pages: 1 ... 6 7 [8] 9   Go Up