Major sRAW1>DNG bug

[Per Zangenberg]

I usually convert my CR2 files to DNG. The files from my 5D mkII are usually a bit smaller than the original CR2 files. But I recently started using a lot of sRAW1 for my wedding work and by chance I discovered that when converting an sRAW1 file to DNG the file becomes almost twice the size of the CR2 file!?!

I have submitted a bog report to Adobe, but I would like others with 5D mkII to test and see if they get the same result and if so also submit a bug report to Adobe:

http://www.adobe.com/cfusion/mmform/index.cfm?name=wishform

[digitaldog]

I recently started using a lot of sRAW1 for my wedding work...

Why?

[Panopeeper]

I discovered that when converting an sRAW1 file to DNG the file becomes almost twice the size of the CR2 file!?!

This is not a bug, in fact it is not a problem at all, except for the occupied storage space.

The DNG specification allows for two formats: raw channels (like Bayer) or "linear", a dumb name, which is in RGB format (the colors are in the camera's address space, not WBd, and the values are linear, like in raw). The sRaw format does not fit in either; it is not raw any more, but the pixels don't have all three components; neighbouring pixels share channels. The DNG converter assigns all three component values to all pixels, thereby increasing the amount of stored data (in practical terms, component values become replicated).

[Per Zangenberg]

This is not a bug, in fact it is not a problem at all, except for the occupied storage space.

And how is this not a bug? It makes the DNG format useless IMO. At least for sRAW.

[Panopeeper]

And how is this not a bug? It makes the DNG format useless IMO. At least for sRAW.

It does not make the DNG format useless, but DNG has a disadvantage in this case. In other words: the advantage of sRaw, the small files, can be fully utilized only with DPP. However, the advantage on the CF card is still there.

Added: a bug is a problem caused unintentionally. This problem is the consequence of the DNG design. Usually I am a critic of DNG, but one has to realize, that not every idea a manufacturer can come up with can be accomodated in a design, particularly if that idea did not exist at the time of making the design.

[Per Zangenberg]

It does not make the DNG format useless, but DNG has a disadvantage in this case. In other words: the advantage of sRaw, the small files, can be fully utilized only with DPP. However, the advantage on the CF card is still there.

Added: a bug is a problem caused unintentionally. This problem is the consequence of the DNG design. Usually I am a critic of DNG, but one has to realize, that not every idea a manufacturer can come up with can be accomodated in a design, particularly if that idea did not exist at the time of making the design.

Ok, thank you for the explanation. I will just stick with CR2 files from now on then.

[digitaldog]

Ok, thank you for the explanation. I will just stick with CR2 files from now on then.

sRaw, despite the name really isn't Raw. Its demosaic the raw data, separate color from luminance, and then subsample color information more than luminance information.

From DP review:
The sRAW data is coded in lossless JPEG format. After decoding, the data stream look like this in dcraw:
skip this part if you're not interested in details


o o o x o x x x .... (pattern repeats)0 1 2 3 4 5 6 7 (index)where 'o' is a valid number, 'x' is an unknown (not recorded in sRAW)
The procedure then goes on to estimate sample No.5 (interpolate) by taking the average of No.1 and No.9, and No.6 from No.2 and No.10. The resulting pattern becomes
o o o x o o o x ....0 1 2 3 0 1 2 3
Then the coefficients 0,1,2 will be used to reconstruct R,G,B for each pixel as follows:R' = C0 + C2G' = C0 - 0.19 C1 - 0.5 C2B' = C0 + C1
(letting C1 = 2Cb, C2 = 1.6Cr, it's clear that this is just a YCbCr to sRGB transform)
thenR = (R'-512)*0.497G = (G'-512)*1.146B = (B'-512)*0.828(probably in camera RGB, didn't bother to check, unbalanced)
end of skip


In short, every 2 sRAW pixels contain: 2 Y, 1 Cb and 1Cr samples, while the corresponding RAW image area (8 pixels) contains 4 Green, 2 Blue and 2 Red samples. All samples are in 14bit. This explains why 1/4 the picture area carries 1/2 the data volume, and why sRAW has very high per-pixel sharpness. However, we still have no idea how this down-sampling is performed

[sandymc]

However, we still have no idea how this down-sampling is performed

Umm, what don't we understand here? Its just a YCC encoding with subsampled chroma - many formats have used this for years. Kodak's old Photo CD, from the early nineties, comes to mind as an early digital image example. You do have to anti-alias the chroma and/or use a clever decoder to prevent artifacts, but that's well understood.

Sandy

[madmanchan]

What is the difference, then, between sraw modes 1 and 2? They have different resolutions. Clearly there is a downsampling process involved, and not just in chroma. What is that process? I think that's what Andrew's getting at.

[sandymc]

What is the difference, then, between sraw modes 1 and 2? They have different resolutions. Clearly there is a downsampling process involved, and not just in chroma. What is that process? I think that's what Andrew's getting at.

Eric,

I'd assumed, maybe wrongly, that it was mostly as laid out here by Laurent Clevy: http://lclevy.free.fr/cr2/#sraw. But I've never had cause to check whether what he writes is correct or not, but he bases most of what he writes on dcraw.

Sandy

[Scott Martin]

I agree with you Per Zangenberg. I didn't think I'd be utilizing the sRAW options on the 5Dmk2 but they have proved to be surprisingly useful in my (and many of my client's) workflows. The advantages of the smaller file size can be quite useful and if conversion to the DNG format takes away this advantage 1) people need to hear this so they can avoid DNG on these files and 2) it would be nice if the DNG conversion process would maintain their intentionally smaller file size and resolution. I would think the later would be to the advantage of the DNG format and its users. Thanks for the heads up.

[Scott Martin]

Eric, what do you think is going on here? Why are Canon's sRAW files actually larger when converted to DNG? DNG users want to know (and want to continue converting to DNG).

[madmanchan]

Sandy, the article you refer to does go into the details of the decoding process. Those are fine. I was referring to the encoding process: how does one take the raw mosaic data and generate the subsampled YCC sRAW (or, now with the 7D, S-RAW and M-RAW) image values?

Scott, DNG supports two basic flavors of image storage: (1) mosaic image data (i.e., color filter array (CFA)), in a variety of mosaic patterns, and (2) non-CFA image data, like RGB or GMCY.

Canon's compressed sRAW format does not fit either category. It is certainly not mosaic data, so it cannot be represented by option 1, i.e., mosaic DNG. Once the sRAW data has been decompressed and mapped to a RGB color space, it can be stored via option 2, i.e., non-CFA image data. However, the image is now much bigger in size, because each pixel in the image is now being represented by three separate 16-bit values, one for each color component. As observed above, this makes the resulting DNG file much bigger than the original sRAW.

While I agree that, from a file size point of view, it is rather useless to convert sRAW files to DNG, this does not mean it is a bug in DNG. It simply means that DNG, as currently specified, cannot store sRAW image data directly -- and hence has to map it to another (unfortunately larger) format that it can store. It is possible that future versions of DNG will offer additional compression or image storage abilities which will address issues like these. In the meantime, if you wish to keep file sizes small when shooting sRAW, I recommend sticking with .CR2.

[Scott Martin]

Excellent! Thanks for taking the time to reply, Eric. Your perspective is invaluable.

[sandymc]

Sandy, the article you refer to does go into the details of the decoding process. Those are fine. I was referring to the encoding process: how does one take the raw mosaic data and generate the subsampled YCC sRAW (or, now with the 7D, S-RAW and M-RAW) image values?

Eric,

Well, it would certainly be intellectually interesting to know how exactly the encoding is done, but I'm not clear that its something we need to know in practice. As long as the subsampling process was done correctly (anti-aliasing, etc), and we can decode the resulting image, it makes no difference, yes? Or am I missing something here?

Sandy

[madmanchan]

Hi Sandy, I did not mean to imply that it was practically important to know how the encoding is performed. I was merely clarifying Andrew's point. Andrew said, "However, we still have no idea how this down-sampling is performed." You then said, "what don't we understand here? Its just a YCC encoding with subsampled chroma." In my view, the things that we "don't understand here" are the details of how that encoding is performed. As you have correctly pointed out, the details of decoding it are largely public and reasonably well understood.

[lclevy]

hi,

the answer of the RGGB -> YCC encoding process is certainly described in one of the Canon Patents (demanded or granted)...

Assignees:  CANON KABUSHIKI KAISHA
http://www.faqs.org/patents/asn/444

for example this one is like the liveview
http://www.faqs.org/patents/app/20080292301  

and this one about Vignetting Correction
http://www.faqs.org/patents/app/20080291299

Laurent
(http://lclevy.free.fr/cr2)

[mpenney]

I usually convert my CR2 files to DNG. The files from my 5D mkII are usually a bit smaller than the original CR2 files. But I recently started using a lot of sRAW1 for my wedding work and by chance I discovered that when converting an sRAW1 file to DNG the file becomes almost twice the size of the CR2 file!?!

I have submitted a bog report to Adobe, but I would like others with 5D mkII to test and see if they get the same result and if so also submit a bug report to Adobe:

http://www.adobe.com/cfusion/mmform/index.cfm?name=wishform

Lightroom makes very large DNG conversions very slowly. Not sure why... probably maintains the extant RAW file inside the DNG (unnecessary).
If you just have to have DNG's use Adobe's free converter. It makes smaller files and is much quicker. Then import the dngs to Lightroom.

No matter which program you use, however, the DNG will be bigger than a RAW.

If you need to do this because you are trying to get away with an old version of photoshop which won;t open your RAW files or some other reason I think you will find it much quicker to stick with just the raw xmp file combination...make sure you opt in for the sidecar xmp rule.

[Scott Martin]

Lightroom makes very large DNG conversions very slowly. Not sure why... probably maintains the extant RAW file inside the DNG (unnecessary).No matter which program you use, however, the DNG will be bigger than a RAW.

Check your DNG conversion options. When the proprietary RAW files isn't embedded the conversion process isn't slow and the file size is definitely a little smaller. Lightroom DNG conversion shares the same code with the separate Adobe DNG converter.

[Per Zangenberg]

and the file size is definitely a little smaller.

No, the sRAW1 files are a LOT bigger when converted.