Post by: Guillermo Luijk on May 20, 2016, 09:04:25 pm
Scene with neutral RAW development (plain DCRAW):
(http://www.guillermoluijk.com/misc/mz_general.jpg)
I shot it 3 times. Find here 100% crops with S curve to clearly show the deep shadows (the images need to be displayed on a PC monitor; on mobile devices the posterization cannot be seen).
ISO100, compressed 12-bit, 5 stops underexposure, +5EV in pp: -> CLEAR POSTERIZATION
(http://www.guillermoluijk.com/misc/mz_iso10012b.jpg)
ISO100, uncompressed 14-bit, 5 stops underexposure, +5EV in pp:
(http://www.guillermoluijk.com/misc/mz_iso10014b.jpg)
ISO3200, uncompressed 14-bit, correct exposure:
(http://www.guillermoluijk.com/misc/mz_iso320014b.jpg)
- ISO3200 14-bits has the lowest noise (expected since the A7 II is not 100% ISO-invariant from ISO100 to ISO3200)
- ISO100 12-bits shows clear posterization
- ISO100 14-bits is a bit noisier than ISO3200 and is as noisy as ISO100 12-bits, but has no posterization
RAW histograms:
- ISO100 compressed 12-bits has barely 32 filled levels
- ISO100 uncompressed 14-bits has 128 filled levels. Not many but enough to avoid posterization
(http://www.guillermoluijk.com/misc/hista7ii.png)
Conclusion: 14-bits uncompressed is recommended if strong shadow processing is going to be applied. It seems the posterization issue is not caused by the compressed vs uncompressed variable, but simply by the available RAW leveles (12 vs 14 bits).
Probably at higher ISOs the 14 vs 12-bit advantage vanishes as the noise will prevail over the lack of levels.
Regards
Post by: Pete Berry on May 21, 2016, 02:05:30 pm
Just wanted to share a quick test to make clear the advantage of the 14-bit uncompressed RAW option (available on the Mark II A7's, not on the first models) when strongly lifting the shadows:
Scene with neutral RAW development (plain DCRAW):
(http://www.guillermoluijk.com/misc/mz_general.jpg)
I shot it 3 times. Find here 100% crops with S curve to clearly show the deep shadows (the images need to be displayed on a PC monitor; on mobile devices the posterization cannot be seen).
ISO100, compressed 12-bit, 5 stops underexposure, +5EV in pp: -> CLEAR POSTERIZATION
(http://www.guillermoluijk.com/misc/mz_iso10012b.jpg)
ISO100, uncompressed 14-bit, 5 stops underexposure, +5EV in pp:
(http://www.guillermoluijk.com/misc/mz_iso10014b.jpg)
ISO3200, uncompressed 14-bit, correct exposure:
(http://www.guillermoluijk.com/misc/mz_iso320014b.jpg)
- ISO3200 14-bits has the lowest noise (expected since the A7 II is not 100% ISO-invariant from ISO100 to ISO3200)
- ISO100 12-bits shows clear posterization
- ISO100 14-bits is a bit noisier than ISO3200 and is as noisy as ISO100 12-bits, but has no posterization
RAW histograms:
- ISO100 compressed 12-bits has barely 32 filled levels
- ISO100 uncompressed 14-bits has 128 filled levels. Not many but enough to avoid posterization
(http://www.guillermoluijk.com/misc/hista7ii.png)
Conclusion: 14-bits uncompressed is recommended if strong shadow processing is going to be applied. It seems the posterization issue is not caused by the compressed vs uncompressed variable, but simply by the available RAW leveles (12 vs 14 bits).
Probably at higher ISOs the 14 vs 12-bit advantage vanishes as the noise will prevail over the lack of levels.
Regards
I'm at a loss trying to see the "posterization" in the shadows you are seeing. What I see is the big difference in chroma noise between the 12-bit 5-stop pushed 100 ISO and the other two - not the compressed gradation that produces posterization. At 12 bits with it's 4096 levels you have 16X JPG 8-bit gradation levels.
Taking your JPG images into PS, the histogram for all show a full 256 levels. And after applying chroma NR to all, still no posterization to my eyes. My 10" 1080HD tablet screen shows identical images to my 24" sRGB 8-bit 1080HD desktop, which will definitely show posteriztion on the screen in some difficult colors that 16-bit PPRGB prints eliminate.
Pete
Post by: Guillermo Luijk on May 21, 2016, 03:54:12 pm
I'm at a loss trying to see the "posterization" in the shadows you are seeing. (...) At 12 bits with it's 4096 levels you have 16X JPG 8-bit gradation levels.
Maybe a 400% zoom helps, but at 100% posterization is clearly seen as well.
ISO100 12 bits + 5EV: this is posterization (insufficient levels to create a gradient, including contiguous pixels clipped to black and other contiguous pixels sharing the same values):
(http://www.guillermoluijk.com/misc/posterization.png)
ISO100 14 bits + 5EV: this is not posterization (few levels but still enough to make noise dither any visible posterization):
(http://www.guillermoluijk.com/misc/noposterization.png)
A 12-bit RAW file has 4096 levels only when they are full of information. This is not the case, 5 stops underexposure provoques that at least the first 5 stops are empty. In this particular case, the ISO100+5EV only has 32 levels containing information for the entire scene as the RAW histogram shows, that's far from 4096.
Regards
Post by: digitaldog on May 21, 2016, 03:56:14 pm
Post by: Zorki5 on May 21, 2016, 09:23:06 pm
One more proof that 14 bits are indeed 14 bits of information, not same 12 bits saved in 14-bit-format file.
Other than that, can't draw any conclusions; do not see compression artifacts; and yes, 12-bit image does look more noisy (for whatever reason!)
Post by: AlterEgo on May 21, 2016, 10:31:27 pm
One more proof that 14 bits are indeed 14 bits of information, not same 12 bits saved in 14-bit-format file.and now switch Sony to continuous shooting mode ;D
Post by: Zorki5 on May 21, 2016, 10:51:02 pm
and now switch Sony to continuous shooting mode ;D
Oh, I don't even have to do that to enjoy... err, compactness of the 12 bits -- I shoot a6000. :)
Post by: Pete Berry on May 21, 2016, 10:56:01 pm
I did a quick similar test with my GH4 between 10-bit elect. shutter RAW's and the standard shutter 12-bit depth. There's a huge difference seen in the -5/+5 EV pushed files comparison - the 12-bit with mild banding and moderate noise; the10-bit with much more chroma and lum. noise obscuring detail, but no banding or micro posterization I could see.
At uncomp'd 3200, though, much less difference, with the 10-bit showing some more difficult chroma noise to remove and slightly diminished detail at 200%, but no shadow banding.
So it seems pretty clear to me that it's the compression in the A7 II 12-bit RAWs rather than the diminished bit depth causing the posterization.
Pete
Post by: Guillermo Luijk on May 22, 2016, 04:04:48 am
1. NUMBER OF FILLED LEVELS IN THE 12-BIT RAW FILES
RAW histogram (portion of the whole range):
(http://www.guillermoluijk.com/misc/sonya7histogram.gif)
Just counting used levels in the whole 12-bit range, we find 4 zones in the decoded values:
- From 128 to 801 all levels used -> 674 levels (128 is the black level for this camera in 12-bit mode; it becomes 512 in 14-bit mode)
- From 802 to 1424 half the levels used -> (1424-802)/2=311 levels
- From 1427 to 2023 one out of each 4 used -> (2023-1427)/4=149 levels
- From 2029 to 4101 one out of each 8 used -> (4101-2029)/8=259 levels
In terms of encoding efficiency, the A7II in 12-bit compressed mode is a log2(1393)=10,44 bits camera.
In the shadows there seems to be no loss because of the compression, but we're still restricted to 12-bit linear density. This answers a question I made in the forum some months ago (http://forum.luminous-landscape.com/index.php?topic=87260.msg821430#msg821430):
"What I do not fully understand is that if one now develops this RAW file with DCRAW (gamma 1.0 output), levels spread in the final image in the same way as in the decoded RAW file, without any compression curve applied. This shocks me because given the high DR of this sensor, if the decoded values are already linear in a 12-bit range (from which only 10,44 bits are actually used), how can it avoid shadow posterization when lifting the shadows? and if they are not linear, how can the output from DCRAW not linearize them?."
The answer is clear to me now that I have the camera: indeed there is posterization (call it visible lack of levels if you wish), something I never expected the clever engineers from Sony would let happen.
Sony's 12-bit compression algorithm could be much more efficient (at CPU cost), becoming non-linear to put extra levels in the deep shadows substracting them from the highlights where there is still room for a reduction with no IQ penalty. The very efficient 8-bit compression algorithm in the Leica M8 does this kind of non-linear clever compression, managing to obtain all the information from its sensor with 8-bit non-linear RAW files (just 256 different levels).
2. 12-BIT RAW DEVELOPMENT FROM 14-BIT UNCOMPRESSED RAW FILES
To explain the fallacy of 14-bit RAW files in some cameras (Canon 40D), and the real need of them in some others (Pentax K5), time ago I pushed 5-stops a RAW file from the Pentax K5 (Sony Sensor) decimating its 14-bits to 12-bit prior to demosaicing. This is the same as having a 12-bit uncompressed RAW file. The result was exactly the same kind of posterization:
14-bits regular RAW development:
(http://www.guillermoluijk.com/article/rawbits/k5_1.jpg)
12-bit RAW development:
(http://www.guillermoluijk.com/article/rawbits/k5_2.jpg)
The article is here: DO RAW BITS MATTER? (http://www.guillermoluijk.com/article/rawbits/index.htm).
---
So my conclusion is that Sony's compression doesn't affect image quality, it is simply that 12 linear bits are not enough for the high dynamic range of these sensors since their debut. The compression just allows to make the files smaller, so maybe a compressed 14-bit format would be the best trade off:
- ISO100 12-bits compressed: 24MB
- ISO100 14-bit uncompressed: 48MB
- ISO3200 14-bit uncompressed: 48MB
If someone wants to play with the RAW files:
http://www.guillermoluijk.com/davidcoffin/
Regards
Post by: Guillermo Luijk on May 22, 2016, 04:39:54 am
Other than that, can't draw any conclusions; do not see compression artifacts; and yes, 12-bit image does look more noisy (for whatever reason!)
Both the ISO100 compressed 12-bit and the ISO100 uncompressed 14-bit had the same noise (SNR) at capture time because they shared exposure and ISO gain. But after the RAW encoding, the ISO100 compressed 12-bit becomes noisier because of the quantization roundings. Posterization is just the result of quantization errors, and quantization is a form of noise.
Regards
Post by: Jack Hogan on May 22, 2016, 08:30:06 am
Maybe a 400% zoom helps, but at 100% posterization is clearly seen as well.
ISO100 12 bits + 5EV: this is posterization (insufficient levels to create a gradient, including contiguous pixels clipped to black and other contiguous pixels sharing the same values):
(http://www.guillermoluijk.com/misc/posterization.png)
ISO100 14 bits + 5EV: this is not posterization (few levels but still enough to make noise dither any visible posterization):
(http://www.guillermoluijk.com/misc/noposterization.png)
A 12-bit RAW file has 4096 levels only when they are full of information. This is not the case, 5 stops underexposure provoques that at least the first 5 stops are empty. In this particular case, the ISO100+5EV only has 32 levels containing information for the entire scene as the RAW histogram shows, that's far from 4096.
Regards
Hi Guillermo,
Thanks for doing this. I am having a little difficulty seeing posterization in those images. A couple of comments:
1) Can you repeat the test at 12 and 14 bits both uncompressed to get rid of Sony's funny compression variable? I wouldn't be surprised if some of what we see is delta encoding of the noise.
2) The threshold for visibility of posterization seems to be around 1 ADU of random noise dithering, independently of where the noise comes from. In that case one could expect blocking and posterization in the deepest shadows of the ISO100 12 bit image because according to Bill Claff's site it would in theory have minimum (read) noise of 0.35 12 bit ADU; I would not expect it in the ISO 100 14 bit image because its minimum random noise would be 1.4 14 bit ADU. Nor in the ISO3200 12 bit image because its minimum random noise would be 2.6 12-bit ADU. I would not expect to see it anywhere where there is enough signal for photon noise to provide the dithering.
3) Empty stops are perfectly ok in the presence of enough dithering. Just because there are more levels does not mean that there is more information in the data. See here (http://www.strollswithmydog.com/iq-raising-iso-vs-pushing/) for a demonstration.
Jack
Post by: Guillermo Luijk on May 22, 2016, 08:51:40 am
1) Can you repeat the test at 12 and 14 bits both uncompressed to get rid of Sony's funny compression variable? I wouldn't be surprised if some of what we see is delta encoding of the noise.
Hi Jack, unfortunately the funny Sony firmware 3.0 just allows two options: 'Compressed' (which is 12-bit compressed RAW), and 'Uncompressed' (which is 14-bit uncompressed RAW). It has no uncompressed 12-bit nor compressed 14-bit.
If you (or any in the thread) have the ability to compile DCRAW, we can reproduce 12-bit uncompressed by developing a 14-bit uncompressed RAW file adding the following bold sentence in DCRAW's scale_colors() function which decimates 2 bits in the RAW data:
for (i=0; i < size*4; i++) {
val = image[0];
if (!val) continue;
val = (val >> 2) << 2
val -= black;
val *= scale_mul[i & 3];
image[0] = CLIP(val);
}
Unfortunately I don't have my PC ready now to compile DCRAW.
Regards
Post by: Jack Hogan on May 22, 2016, 01:48:13 pm
Post by: Zorki5 on May 22, 2016, 02:25:16 pm
But after the RAW encoding, the ISO100 compressed 12-bit becomes noisier because of the quantization roundings. Posterization is just the result of quantization errors, and quantization is a form of noise.
The more roundings -- the more posterization and the less noise. Always.
Do this mental experiment: start decreasing number of colors in your image (in post), until you have only one. At that point, posterization will become 100%, and noise will become 0%, so to speak. Question: at which point noise stopped increasing and started decreasing? Correct answer is "at no point"; it was this way all along.
Even if, while dropping bits, at some point you observe an effect that might look to you as an increase in noise, that would not be noise, that would be something else.
As a side note: when you blow your image to > 100%, try using "nearest neighbor", or some other scaling algorithm that doesn't do smoothing, or do less smoothing (i.e. not bicubic or Lanczos); this way, what's really going on will become more apparent.
Post by: Zorki5 on May 22, 2016, 02:31:59 pm
val = (val >> 2) << 2
Or:
val &= ~3;
;)
Post by: Bart_van_der_Wolf on May 22, 2016, 02:37:25 pm
I am having a little difficulty seeing posterization in those images.
Hi Jack,
I agree, but that is probably because a more accurate term to describe the issue would be "Quantization errors".
One could call it local posterization, but there is already enough confusion between "posterization", and "banding". I'd stick to "quantization errors" in Guillermo's clear examples, especially because it's mostly showing the effect of the lack of levels to represent the lowest bits.
Of course, Raw processors also play a role in how they convert those truncated values in the shadows to mid-tones. Mid-tones and highlights have enough shot noise to dither the smoothness in gradients so that the lack of a few bits of precision doesn't become too obvious.
Cheers,
Bart
Post by: Zorki5 on May 22, 2016, 02:42:53 pm
I agree, but that is probably because a more accurate term to describe the issue would be "Quantization errors".
+1
Agree with the rest of the message as well, but what's important for the topic of this discussion is in the above quote.
Post by: Guillermo Luijk on May 22, 2016, 02:45:03 pm
Or:C guys always wanting to obfuscate code
val &= ~3;
;)
www.guillermoluijk.com
Post by: Guillermo Luijk on May 22, 2016, 02:47:49 pm
The point is Sony is providing truly lossy RAW files, with no alternative option in the Mark I series of the A7.
Regards
www.guillermoluijk.com
Post by: Zorki5 on May 22, 2016, 02:48:55 pm
C guys always wanting to obfuscate code
Yep, Duff's device (https://en.wikipedia.org/wiki/Duff%27s_device) had been my inspiration for decades :D
Post by: samueljohnchia on May 27, 2016, 03:26:22 am
Is this test showing regular Sony 13/11 bit lossy compressed raw, or is it 12 bit readout (silent shutter, continuous shot mode, bulb exposure) that you are showing here?
Thanks!
Post by: IliasG on June 01, 2016, 06:00:20 am
.. unfortunately the funny Sony firmware 3.0 just allows two options: 'Compressed' (which is 12-bit compressed RAW), and 'Uncompressed' (which is 14-bit uncompressed RAW)...
Hi Guillermo,
Sony's ARW2/3 compressed raws are not 12 bit, (except from Bulb and silent shutter modes) and it's Dcraw which makes them look like 12bit ..
https://github.com/Beep6581/RawTherapee/issues/1997
https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2013/comment-4/sony-scaling.patch
(you have to take care of the black level vallue also ..)
In fact at the linear portion (before Sony's curve starts decreasing data density) they are 13bit in 14bit container (half levels missing ).. if you cannot modify/compile Dcraw try by first converting to DNG ..
Post by: Guillermo Luijk on June 08, 2016, 03:14:12 am
Hi Guillermo,That's frankly interesting Samuel/Ilias. I was indeed using EFCS in all shots, the Compressed were developed by DCRAW as 12 bit but the Uncompressed as 14 bit. That would explain the unexpected lack of levels in the Compressed shots. The conclusion would be not to use Compressed+EFCS.
Sony's ARW2/3 compressed raws are not 12 bit, (except from Bulb and silent shutter modes) and it's Dcraw which makes them look like 12bit ..
https://github.com/Beep6581/RawTherapee/issues/1997
https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2013/comment-4/sony-scaling.patch
(you have to take care of the black level vallue also ..)
In fact at the linear portion (before Sony's curve starts decreasing data density) they are 13bit in 14bit container (half levels missing ).. if you cannot modify/compile Dcraw try by first converting to DNG ..
However I found another issue on DCRAW 9.27: it doesn't develop properly ISO100 Uncompressed, while the rest of ISOs are OK. Raw Therapee v4.1 has the same problem but v4.1005 fixes the bug. I let Coffin know.
(http://www.guillermoluijk.com/a7/comp.jpg)
Regards
www.guillermoluijk.com