Pages: [1] 2   Go Down

Author Topic: Understanding reduction in file size from Camera Raw to DNG  (Read 18732 times)

noavscinc

  • Newbie
  • *
  • Offline Offline
  • Posts: 8
Understanding reduction in file size from Camera Raw to DNG
« on: April 18, 2014, 05:18:51 pm »

I am currently testing the conversion of Leaf camera raw files into DNGs for a photographer's archive. I am hoping to convert all of the mos files to DNGs because Leaf Capture and the Leaf Raw Converter are not being updated and because the photographer wants to have an Adobe centered workflow. In my testing I discovered that converting mos files to DNGs through ACR 8.4 and LightRoom 5.4 resulted in a reduction of file size by nearly 50%. A 44.5MB mos file became a 23.6MB DNG. From what I've read only about 15-20% of the camera raw file should be lost and all of the data lost should be proprietary.
 
Here-in lies my question, is there any way that I can track or determine exactly what sort of compression is being done to the mos file and what information is or is not traveling in the conversion to DNG?
 
These are the settings I have used for converting raw files to DNGs:
ACR:
JPEG Preview: Medium Size
Embed fast load data
Don't use lossy compression
Preserve pixel counts
Don't embed original

LIGHTROOM 5.4:
Only Convert Raw files
Delete originals after successful conversion
File Extension DNG
Compatibility Camera Raw 7.1 and later
Jpeg Preview Medium Size
Embed Fast Load Data
 
Logged

Lundberg02

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 379
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #1 on: April 18, 2014, 07:18:47 pm »

My guess: the bit depths are different.
Logged

Redcrown

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 507
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #2 on: April 18, 2014, 08:11:34 pm »

I don't know Leaf so I went to the Mamiya site and downloaded a mos raw file.

Converted it to DNG about 8 different ways and don't get your results. Not even close. So I suggest you repeat your test.

I used ACR 7.1 and 8.4, different jpeg previews, embedd fast load on and off. My 43.04mb mos file becomes a 42.84mb DNG file, give or take a few K depending on the variables.

If I use Lossy DNG compression it becomes 16.26mb.

You can't change bit depth in DNG concersion, so that's not a variable.
Logged

noavscinc

  • Newbie
  • *
  • Offline Offline
  • Posts: 8
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #3 on: April 21, 2014, 10:39:14 am »

Hi Redcrown

I tried my test again with files from 2003-2013. I even tried doing a test with a lossless and lossy conversion of a 165 MB file. Lossless = 86.5 MB and Lossy =16.1 MB. I also tested the conversions in ACR 8.3 and LR 5.3 and got the same results. Playing with jpeg previews and fast load data resulted in very minute differences in size (of the few K sort you encountered in your testing).

What sort of mos file did you download from the Mamiya site? Might you be able to direct me to the file so that I can try the same tests you preformed with the settings I have on my computer?
Logged

noavscinc

  • Newbie
  • *
  • Offline Offline
  • Posts: 8
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #4 on: April 21, 2014, 10:40:16 am »

Lundberg02

Why do you think bit depth might be a factor?
Logged

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #5 on: April 21, 2014, 11:16:10 am »

By default, Adobe products use lossless compression when converting to DNG, which typical gives +-50% data size reduction. If you want to switch that off, rather than "Camera Raw x.x and later" compatibility, you need to set custom compatibility. In DNG converter, you select "Custom DNG Compatibility", and tick uncompressed. ACR and LR have similar settings.

Sandy
Logged

noavscinc

  • Newbie
  • *
  • Offline Offline
  • Posts: 8
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #6 on: April 21, 2014, 12:40:02 pm »

Hi Sandy,

Thank you for the tip. Checking off the compatibility in ACR and selecting uncompressed resulted in a file size increase of about .2 and .4 for 165MB and 167MB raw files I tested.

I can't seem to find the setting to set up custom compatibility and check "Uncompressed" in LightRoom. Could you point me in the direction of where you found that option?

From what I've read not compressing the DNG files is beneficial because it is not always clear whether the lossy compression will result in a loss of quality or "baking in" of changes to a raw file to make it backwards compatible with certain Camera Raw Versions and thus less flexible to edits. http://dpbestflow.org/DNG#backwards-compatibility

Do you as a rule always convert your files to DNGs without the lossless compression? Do you have any other thoughts about why one might want or not want to make their files backwards compatible (forgive my ignorance on the issue)?
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #8 on: April 21, 2014, 01:29:12 pm »

There's a discussion here of where the DNG compatibility option is in Lightroom.

I don't recommend converting files to DNG if you want to ensure that there is no possible data loss - there are too many things that can go wrong.

Sandy
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #9 on: April 21, 2014, 01:30:53 pm »

I don't recommend converting files to DNG if you want to ensure that there is no possible data loss - there are too many things that can go wrong.
Other than proprietary metadata, what data loss are you referring to?
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Redcrown

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 507
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #10 on: April 21, 2014, 01:36:46 pm »

From this page: http://www.mamiyaleaf.com/sample_images.html

I went here: http://www.mamiyaleaf.com/samples_aptus-II_8.html

and downloaded the second Harley cycle image.

That "Aptus II-8" mos file gives me DNG files that are just a few K smaller in size.

Then I downloaded an "Aptus II-5" file (the one with a lime and other test objects). That mos file is 47.36 MB, and when I convert it to DNG I get 29.14 MB, which is about a 40% reduction. Not your 50% reduction, but much more significant than other raw files.

My primary experience is with Canon CR2 files from several models. I get an average of 15% to 20% reduction when converting to DNG. Not the 50% range that Sandymc mentions. I think most, if not all cameras compress their in-camera raw files. Certainly the method and degree of raw compression varies by camera maker. That would account for big variances in compression that different users see.

So, a make or model of one camera might use weaker compression than other makes or models. That apperas to be the case with Leaf. The mos file from the Aptus II-5 model has pixel dimensions of 5336 X 4000 and is 47.48MB. The Aptus II-8 file has pixel dimensions of 7304 X 5466 and is 43.04 MB. Much bigger image, but a smaller mos file.

What is the camera model of the mos file you are using? Maybe consider uploading it to Dropbox or somewhere and we can test it for you.
Logged

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #11 on: April 21, 2014, 01:38:03 pm »

Other than proprietary metadata, what data loss are you referring to?

There are a number of issues:

  • DNG conversion these days has a lot of options, as the OP is finding - it's easy to make a mistake
  • What Adobe doesn't know, Adobe doesn't transfer to DNG; what you're getting is Adobe's understanding of the file format, which may or may not be complete
  • If the DNG format  isn't capable of encoding the raw file or the current version of ACR/LR can't correctly handle the format, then DNG conversion has sometimes silently converted to linear raw format, which is a major loss of data. E.g. in the early days, for some cameras with lens correction, or the early days of X-Trans sensors.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #12 on: April 21, 2014, 01:58:53 pm »

There are a number of issues:

  • DNG conversion these days has a lot of options, as the OP is finding - it's easy to make a mistake
  • What Adobe doesn't know, Adobe doesn't transfer to DNG; what you're getting is Adobe's understanding of the file format, which may or may not be complete
  • If the DNG format  isn't capable of encoding the raw file or the current version of ACR/LR can't correctly handle the format, then DNG conversion has sometimes silently converted to linear raw format, which is a major loss of data. E.g. in the early days, for some cameras with lens correction, or the early days of X-Trans sensors.

1. True (RTFM ::))
2. What doesn't Adobe know that would affect my ability to render the raw data in a converter of my choice outside that of the camera manufacture?
3. I don't understand this if statement. I'm referring to any current raw I (or others, TMMV) have today and convert to DNG (with or without backing up the original raws).

FWIW, I have three different raw converters I like and use that fully support DNG from three different companies. Two are not Adobe.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Redcrown

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 507
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #13 on: April 21, 2014, 02:10:03 pm »

A couple more comments about DNG options:

1. About DNG lossy compression, I discovered a "bug" and discussed it here. Eric Chan acknowledged the issue, but it has not been fixed yet (as of ACR 8.4). Enough of a bug to keep me from using it.

http://www.luminous-landscape.com/forum/index.php?topic=82064.msg699884#msg699884

2. About fast load data: There was a good discussion on some forum over a year ago, but now I can't find it.

The bottom line was that embedding fast load data was not worth it. When opening DNG files one at a time in ACR there is no perceivable difference. When batch processing a large number of DNG files (via Image Processor) there may be a slight difference, but it's insignificant. Yet embedding fast load data increases file size and increases conversion time. Not a lot, but significant when spread over thousands of files.

The theory was that embedding fat load data might have had a worthwhile advantage years ago when machines were much slower, but the speed of modern machines makes it not worthwhile.
Logged

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #14 on: April 21, 2014, 02:10:25 pm »

1. True (RTFM ::))
2. What doesn't Adobe know that would affect my ability to render the raw data in a converter of my choice outside that of the camera manufacture?
3. I don't understand this if statement. I'm referring to any current raw I (or others, TMMV) have today and convert to DNG (with or without backing up the original raws).

FWIW, I have three different raw converters I like and use that fully support DNG from three different companies. Two are not Adobe.

On point (2), what Adobe doesn't know can impact on the quality of the conversion.

On point (3) the question is, how much knowledge of DNG conversion do you expect users to have? At any given point, for any given given combination of camera and LR/ACR/DNG converter version, I need to look at a converted DNG with tools that take it apart to know what Adobe's actually doing. There's no nice dialog box that comes up and warns you when there's going to be data loss. And as a direct result, there are a lot of people sitting right now with Fuji X-Trans files that have the demosaicing baked in to a version of Adobe's demosaicing code that's obsolete.  I don't know of any camera right now where this is the case, but (a) there might be one for all I know, and (b) who knows what happens when the next new camera or new version of Adobe's software comes out.

So I don't recommend converting to DNG if you want 100% assurance of no data loss.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #15 on: April 21, 2014, 02:46:53 pm »

On point (2), what Adobe doesn't know can impact on the quality of the conversion.
How? Examples please. Isn't the lossy raw just that, raw? What data loss occurs that affect how I use that raw data even in non Adobe converters?
Quote
There's no nice dialog box that comes up and warns you when there's going to be data loss.
Again, what data loss? What would I look for in the library of images I have in DNG for this data loss in the three converters I'd use on that data and compared to the camera raw original?
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

noavscinc

  • Newbie
  • *
  • Offline Offline
  • Posts: 8
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #16 on: April 21, 2014, 02:51:41 pm »

Hi sandymc,

I looked at the link you sent and was not able to determine where the options to customize compatibility are. The website says "Compatibility: Specifies the versions of Camera Raw and Lightroom that can read the file. Use the tool tips to help you choose." I can't figure out what or where these "tool tips" would be.

I have attached screenshots of the dialog options I have available to me in the export and convert options. I must be missing something very obvious...
Logged

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #17 on: April 21, 2014, 02:53:45 pm »

How? Examples please. Isn't the lossy raw just that, raw? What data loss occurs that affect how I use that raw data even in non Adobe converters? Again, what data loss? What would I look for in the library of images I have in DNG for this data loss in the three converters I'd use on that data and compared to the camera raw original?

Andrew,

As mentioned in my post above, I regard conversion to linear raw as data loss. I understand that you might not, but that's my view. There is no raw converter in the world, Adobe's included, that gets you back to raw once demosaicing already been done.

Sandy
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #18 on: April 21, 2014, 02:57:03 pm »

As mentioned in my post above, I regard conversion to linear raw as data loss.
I fully agree! But you're not helping me in understanding when and where that happens. I've got about 27K DNG's. AFAIK, none are linear. I've never asked for Linear. I think you're saying there is a case where *some* camear raws are converted to Linear without the end user being told? If so what is the trigger?
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: Understanding reduction in file size from Camera Raw to DNG
« Reply #19 on: April 21, 2014, 02:58:16 pm »

Hi sandymc,

I looked at the link you sent and was not able to determine where the options to customize compatibility are. The website says "Compatibility: Specifies the versions of Camera Raw and Lightroom that can read the file. Use the tool tips to help you choose." I can't figure out what or where these "tool tips" would be.

I have attached screenshots of the dialog options I have available to me in the export and convert options. I must be missing something very obvious...

I don't have LR installed on the machine I have with me at the moment so I can't check, but I think that the detailed options only exist if you export DNG's, not if you convert on import.

Logged
Pages: [1] 2   Go Up