Luminous Landscape Forum
Raw & Post Processing, Printing => Adobe Camera Raw Q&A => Topic started by: noavscinc on April 18, 2014, 05:18:51 pm
-
I am currently testing the conversion of Leaf camera raw files into DNGs for a photographer's archive. I am hoping to convert all of the mos files to DNGs because Leaf Capture and the Leaf Raw Converter are not being updated and because the photographer wants to have an Adobe centered workflow. In my testing I discovered that converting mos files to DNGs through ACR 8.4 and LightRoom 5.4 resulted in a reduction of file size by nearly 50%. A 44.5MB mos file became a 23.6MB DNG. From what I've read only about 15-20% of the camera raw file should be lost and all of the data lost should be proprietary.
Here-in lies my question, is there any way that I can track or determine exactly what sort of compression is being done to the mos file and what information is or is not traveling in the conversion to DNG?
These are the settings I have used for converting raw files to DNGs:
ACR:
JPEG Preview: Medium Size
Embed fast load data
Don't use lossy compression
Preserve pixel counts
Don't embed original
LIGHTROOM 5.4:
Only Convert Raw files
Delete originals after successful conversion
File Extension DNG
Compatibility Camera Raw 7.1 and later
Jpeg Preview Medium Size
Embed Fast Load Data
-
My guess: the bit depths are different.
-
I don't know Leaf so I went to the Mamiya site and downloaded a mos raw file.
Converted it to DNG about 8 different ways and don't get your results. Not even close. So I suggest you repeat your test.
I used ACR 7.1 and 8.4, different jpeg previews, embedd fast load on and off. My 43.04mb mos file becomes a 42.84mb DNG file, give or take a few K depending on the variables.
If I use Lossy DNG compression it becomes 16.26mb.
You can't change bit depth in DNG concersion, so that's not a variable.
-
Hi Redcrown
I tried my test again with files from 2003-2013. I even tried doing a test with a lossless and lossy conversion of a 165 MB file. Lossless = 86.5 MB and Lossy =16.1 MB. I also tested the conversions in ACR 8.3 and LR 5.3 and got the same results. Playing with jpeg previews and fast load data resulted in very minute differences in size (of the few K sort you encountered in your testing).
What sort of mos file did you download from the Mamiya site? Might you be able to direct me to the file so that I can try the same tests you preformed with the settings I have on my computer?
-
Lundberg02
Why do you think bit depth might be a factor?
-
By default, Adobe products use lossless compression when converting to DNG, which typical gives +-50% data size reduction. If you want to switch that off, rather than "Camera Raw x.x and later" compatibility, you need to set custom compatibility. In DNG converter, you select "Custom DNG Compatibility", and tick uncompressed. ACR and LR have similar settings.
Sandy
-
Hi Sandy,
Thank you for the tip. Checking off the compatibility in ACR and selecting uncompressed resulted in a file size increase of about .2 and .4 for 165MB and 167MB raw files I tested.
I can't seem to find the setting to set up custom compatibility and check "Uncompressed" in LightRoom. Could you point me in the direction of where you found that option?
From what I've read not compressing the DNG files is beneficial because it is not always clear whether the lossy compression will result in a loss of quality or "baking in" of changes to a raw file to make it backwards compatible with certain Camera Raw Versions and thus less flexible to edits. http://dpbestflow.org/DNG#backwards-compatibility
Do you as a rule always convert your files to DNGs without the lossless compression? Do you have any other thoughts about why one might want or not want to make their files backwards compatible (forgive my ignorance on the issue)?
-
See Reply #3: http://www.luminous-landscape.com/forum/index.php?topic=64034.0
-
There's a discussion here (http://help.adobe.com/en_US/lightroom/using/WSA58ABA51-5376-48e5-AD78-C4B453C57202.html#WS0700C48D-7ACD-4639-9825-CCB6A22C0056) of where the DNG compatibility option is in Lightroom.
I don't recommend converting files to DNG if you want to ensure that there is no possible data loss - there are too many things that can go wrong.
Sandy
-
I don't recommend converting files to DNG if you want to ensure that there is no possible data loss - there are too many things that can go wrong.
Other than proprietary metadata, what data loss are you referring to?
-
From this page: http://www.mamiyaleaf.com/sample_images.html
I went here: http://www.mamiyaleaf.com/samples_aptus-II_8.html
and downloaded the second Harley cycle image.
That "Aptus II-8" mos file gives me DNG files that are just a few K smaller in size.
Then I downloaded an "Aptus II-5" file (the one with a lime and other test objects). That mos file is 47.36 MB, and when I convert it to DNG I get 29.14 MB, which is about a 40% reduction. Not your 50% reduction, but much more significant than other raw files.
My primary experience is with Canon CR2 files from several models. I get an average of 15% to 20% reduction when converting to DNG. Not the 50% range that Sandymc mentions. I think most, if not all cameras compress their in-camera raw files. Certainly the method and degree of raw compression varies by camera maker. That would account for big variances in compression that different users see.
So, a make or model of one camera might use weaker compression than other makes or models. That apperas to be the case with Leaf. The mos file from the Aptus II-5 model has pixel dimensions of 5336 X 4000 and is 47.48MB. The Aptus II-8 file has pixel dimensions of 7304 X 5466 and is 43.04 MB. Much bigger image, but a smaller mos file.
What is the camera model of the mos file you are using? Maybe consider uploading it to Dropbox or somewhere and we can test it for you.
-
Other than proprietary metadata, what data loss are you referring to?
There are a number of issues:
- DNG conversion these days has a lot of options, as the OP is finding - it's easy to make a mistake
- What Adobe doesn't know, Adobe doesn't transfer to DNG; what you're getting is Adobe's understanding of the file format, which may or may not be complete
- If the DNG format isn't capable of encoding the raw file or the current version of ACR/LR can't correctly handle the format, then DNG conversion has sometimes silently converted to linear raw format, which is a major loss of data. E.g. in the early days, for some cameras with lens correction, or the early days of X-Trans sensors.
-
There are a number of issues:
- DNG conversion these days has a lot of options, as the OP is finding - it's easy to make a mistake
- What Adobe doesn't know, Adobe doesn't transfer to DNG; what you're getting is Adobe's understanding of the file format, which may or may not be complete
- If the DNG format isn't capable of encoding the raw file or the current version of ACR/LR can't correctly handle the format, then DNG conversion has sometimes silently converted to linear raw format, which is a major loss of data. E.g. in the early days, for some cameras with lens correction, or the early days of X-Trans sensors.
1. True (RTFM ::))
2. What doesn't Adobe know that would affect my ability to render the raw data in a converter of my choice outside that of the camera manufacture?
3. I don't understand this if statement. I'm referring to any current raw I (or others, TMMV) have today and convert to DNG (with or without backing up the original raws).
FWIW, I have three different raw converters I like and use that fully support DNG from three different companies. Two are not Adobe.
-
A couple more comments about DNG options:
1. About DNG lossy compression, I discovered a "bug" and discussed it here. Eric Chan acknowledged the issue, but it has not been fixed yet (as of ACR 8.4). Enough of a bug to keep me from using it.
http://www.luminous-landscape.com/forum/index.php?topic=82064.msg699884#msg699884
2. About fast load data: There was a good discussion on some forum over a year ago, but now I can't find it.
The bottom line was that embedding fast load data was not worth it. When opening DNG files one at a time in ACR there is no perceivable difference. When batch processing a large number of DNG files (via Image Processor) there may be a slight difference, but it's insignificant. Yet embedding fast load data increases file size and increases conversion time. Not a lot, but significant when spread over thousands of files.
The theory was that embedding fat load data might have had a worthwhile advantage years ago when machines were much slower, but the speed of modern machines makes it not worthwhile.
-
1. True (RTFM ::))
2. What doesn't Adobe know that would affect my ability to render the raw data in a converter of my choice outside that of the camera manufacture?
3. I don't understand this if statement. I'm referring to any current raw I (or others, TMMV) have today and convert to DNG (with or without backing up the original raws).
FWIW, I have three different raw converters I like and use that fully support DNG from three different companies. Two are not Adobe.
On point (2), what Adobe doesn't know can impact on the quality of the conversion.
On point (3) the question is, how much knowledge of DNG conversion do you expect users to have? At any given point, for any given given combination of camera and LR/ACR/DNG converter version, I need to look at a converted DNG with tools that take it apart to know what Adobe's actually doing. There's no nice dialog box that comes up and warns you when there's going to be data loss. And as a direct result, there are a lot of people sitting right now with Fuji X-Trans files that have the demosaicing baked in to a version of Adobe's demosaicing code that's obsolete. I don't know of any camera right now where this is the case, but (a) there might be one for all I know, and (b) who knows what happens when the next new camera or new version of Adobe's software comes out.
So I don't recommend converting to DNG if you want 100% assurance of no data loss.
-
On point (2), what Adobe doesn't know can impact on the quality of the conversion.
How? Examples please. Isn't the lossy raw just that, raw? What data loss occurs that affect how I use that raw data even in non Adobe converters?
There's no nice dialog box that comes up and warns you when there's going to be data loss.
Again, what data loss? What would I look for in the library of images I have in DNG for this data loss in the three converters I'd use on that data and compared to the camera raw original?
-
Hi sandymc,
I looked at the link you sent and was not able to determine where the options to customize compatibility are. The website says "Compatibility: Specifies the versions of Camera Raw and Lightroom that can read the file. Use the tool tips to help you choose." I can't figure out what or where these "tool tips" would be.
I have attached screenshots of the dialog options I have available to me in the export and convert options. I must be missing something very obvious...
-
How? Examples please. Isn't the lossy raw just that, raw? What data loss occurs that affect how I use that raw data even in non Adobe converters? Again, what data loss? What would I look for in the library of images I have in DNG for this data loss in the three converters I'd use on that data and compared to the camera raw original?
Andrew,
As mentioned in my post above, I regard conversion to linear raw as data loss. I understand that you might not, but that's my view. There is no raw converter in the world, Adobe's included, that gets you back to raw once demosaicing already been done.
Sandy
-
As mentioned in my post above, I regard conversion to linear raw as data loss.
I fully agree! But you're not helping me in understanding when and where that happens. I've got about 27K DNG's. AFAIK, none are linear. I've never asked for Linear. I think you're saying there is a case where *some* camear raws are converted to Linear without the end user being told? If so what is the trigger?
-
Hi sandymc,
I looked at the link you sent and was not able to determine where the options to customize compatibility are. The website says "Compatibility: Specifies the versions of Camera Raw and Lightroom that can read the file. Use the tool tips to help you choose." I can't figure out what or where these "tool tips" would be.
I have attached screenshots of the dialog options I have available to me in the export and convert options. I must be missing something very obvious...
I don't have LR installed on the machine I have with me at the moment so I can't check, but I think that the detailed options only exist if you export DNG's, not if you convert on import.
-
I fully agree! But you're not helping me in understanding when and where that happens. I've got about 27K DNG's. AFAIK, none are linear. I've never asked for Linear. I think you're saying there is a case where *some* camear raws are converted to Linear without the end user being told? If so what is the trigger?
Yes. Like I said above, if the version of DNG doesn't support something in the raw (which was the case for lens corrections on I think Panasonic cameras pre the versions of DNG with op codes), or if ACR/LR would throw up (which was the case for XTrans - before Adobe put good support in). Under those circumstances, DNG converter silently went to linear raw.
See, e.g., this thread on the Adobe forums: http://forums.adobe.com/message/5144148#5144148 (http://forums.adobe.com/message/5144148#5144148)
-
sandymc and dgitaldog:
I'm a newbie to all of this but based on the link I posted earlier it seems like the data loss and potential linearization of a file pertains to the ways in which a file is made backwards compatible by ACR. Even if you set up your preferences to preform a lossless conversion the compression process to make a file backwards compatible is lossy:
"The DNG format makes backwards compatibility possible in several different ways. Sometimes this is done without any loss of quality, and sometimes it requires “baking in” some of the changes to the raw file, which does remove some flexibility to re-edit your files. Let’s look at the issues.....Sometimes the backwards compatibility must be done by linearizing the DNG because some new features are not supported by older software. For instance, the Panasonic LX5 camera requires some image de-warping in the raw conversion in order to remove lens distortion. Older Camera Raw versions did not support this function. In order to open these new files with old software, the DNG must be created with the de-warping baked in to the DNG, which is done by selecting a Compatibility prior to the latest version, as shown in Figure 5. This is a “lossy” operation, since you’ll be making changes that can’t be undone." http://dpbestflow.org/DNG#backwards-compatibility
-
Yes. Like I said above, if the version of DNG doesn't support something in the raw (which was the case for lens corrections on I think Panasonic cameras pre the versions of DNG with op codes), or if ACR/LR would throw up (which was the case for XTrans - before Adobe put good support in). Under those circumstances, DNG converter silently went to linear raw.
Did you write there:
3. "Normal" aka CFA DNG. The DNG contains CFA (aka Bayer mosaiced) data, and whatever reads the DNG does the demosaicing. You're not locked in.
I believe I'm also using option 3. Further I've got no Fuji cameras. So outside of that model, what data loss could I find?
Adobe SHOULD pop a warning dialog, no question.
-
In order to open these new files with old software, the DNG must be created with the de-warping baked in to the DNG, which is done by selecting a Compatibility prior to the latest version, as shown in Figure 5. This is a “lossy” operation, since you’ll be making changes that can’t be undone." http://dpbestflow.org/DNG#backwards-compatibility
So it appears there is a rare condition in which without warning, conversion to DNG can be Linear. Again Adobe should pop a working. But unless you care about an old version of software support, the latest version for DNG (my default) isn't going to cause any issues?
-
I'm not sure how rare it is. It sounds like any time the ACR encounters a feature that is not supported by earlier versions of Camera Raw, it will linearize the file...Is this what you are suggesting sandymc?
-
The point is, there are circumstances under which there may be data loss in ways that would not be obvious to the casual user. In many, or even most, situations there will not be a problem, but I'm very wary of blanket assurances that DNG conversion is always lossless.
-
The point is, there are circumstances under which there may be data loss in ways that would not be obvious to the casual user. In many, or even most, situations there will not be a problem, but I'm very wary of blanket assurances that DNG conversion is always lossless.
Yes, there are ways to hose any data during conversion. You had me worried but now I see this is a very isolated case with one or two cameras with a setting that I wouldn't recommend using for DNG conversion. Unless I'm missing something more. So in the three cases you cited, it's really one. If the end user really knew what we now know, they wouldn't have an issue, they would use a more appropriate conversion setting. With these older cameras set for older software. Adobe should pop an dialog for these rare cases. No excuse for that. I wasn't aware of this so this was useful.
-
- What Adobe doesn't know, Adobe doesn't transfer to DNG; what you're getting is Adobe's understanding of the file format, which may or may not be complete
While that is true for "some" raw file formats, I'm pretty sure (post opcodes) that the DNG conversion will find a place for private maker notes and other data into the DNG file even if DNG doesn't understand that data. You'll have to ask Eric to confirm this...but what DNG (or ACR/LR) doesn't know about can't be used in DNGs anyway. So, if ACR/LR can't use the metadata, not sure what good it is.
Now, there are some issues with Bridge and Lightroom failing to show or disclose certain metadata such as focus points in a capture, but I don't think converting to DNG strips them, they just are not used in Bridge and LR.
-
Now, there are some issues with Bridge and Lightroom failing to show or disclose certain metadata such as focus points in a capture, but I don't think converting to DNG strips them, they just are not used in Bridge and LR.
I believe that a NEF converted to DNG changes or strips the focus point metadata in such a way that Aperture can no longer use it. At least is seems to in the testing I've done.
This had made me leery of the DNG format, not knowing what else might be striped out that I could want days or years later. If Lightroom were to one day display Nikon focus points, would it still be able to in a NEF converted to DNG?
-
Yes, the focus point metadata (and other Nikon-related metadata) is preserved in the DNG. At the technical level, vendor private data such as MakerNotes are basically TIFF-like tags that software can simply copy over from one container/file to another, even if the software doesn't understand the meaning of those tags (*). Just because other software does not show the metadata after conversion to DNG, it does not mean that the metadata does not exist.
Eric
(*) It would be like going to a warehouse filled with boxes and being asked to save or move those boxes. You can move those boxes from one warehouse to another even if you don't know or understand what's in those boxes.
-
Yes, the focus point metadata (and other Nikon-related metadata) is preserved in the DNG. At the technical level, vendor private data such as MakerNotes are basically TIFF-like tags that software can simply copy over from one container/file to another, even if the software doesn't understand the meaning of those tags (*). Just because other software does not show the metadata after conversion to DNG, it does not mean that the metadata does not exist.
Eric, thanks for the authoritative answer; it's good to hear from the source.
I guess what is happening is that the DNG Converter moves the Nikon focus point metadata and then Aperture doesn't know where to look to find it.