Luminous Landscape Forum
Raw & Post Processing, Printing => Digital Image Processing => Topic started by: eitanwaks on July 07, 2009, 02:51:33 pm
-
hi,
although I'm sure there are a lot of threads dealing with the subject whenever I search for "NEF versus DNG" I received zero results. I don't want to open a whole new thread on the subject that has been dealt with thoroughly. Can someone please point me to some threads that deal with the subject?
I'm contemplating converting everything to DNG so as not to have to deal with the XMP sidecar files. I would like to read more about the subject before I commit to this process. If anyone has a short opinion I would love to hear it.
Thanks for the help,
Eitan Waks
-
Here's a couple of article to get you started.
http://www.digitalphotopro.com/software/im...ook-at-dng.html (http://www.digitalphotopro.com/software/image-processing/dpp-solutions-a-look-at-dng.html)
http://www.digitalphotopro.com/gear/imagin...ive-format.html (http://www.digitalphotopro.com/gear/imaging-tech/the-digital-negative-format.html)
-
Strange, isn't it? I don't use DNG because that would make me lose the XMP sidecar files.
-- Olaf
-
I'm contemplating converting everything to DNG so as not to have to deal with the XMP sidecar files
The sidecar files are the icing on the cake. Do you prefer saving/archiving a 15-30 MB file after having changed the white balance (as an example), instead of saving/archiving a 2 KB small file?
-
Having the metadata embedded in a publicly-documented file format is indeed an advantage of switching to DNG. A wider range of applications can read/write to DNG, so you're freer to choose the right tool for the job than when you're fiddling with XMP colostomy files. Another gain is that your DNG contains an updated preview, so you can see or use the adjusted image in other applications.
A lot depends on how much ongoing work you to do the pictures after you first process them, and in what program. If you do the vast bulk of your adjustments and metadata entry via Bridge/Adobe Camera Raw when the pictures are still new, create the DNG, and back it up. Any adjustment/metadata from that point on is exposed as it's not backed up, but wouldn't be hard to recreate if ever needed.
If you continually add more adjustments and metadata to your images, using programs like Lightroom or Expression Media, then backup the DNG after its creation, so the image data is safeguarded, and backup your ongoing work by backing up the catalogue. While it's helpful to save metadata and embedded previews back to the working DNG, there's no point backing it up each time.
Going DNG doesn't mean trashing your NEFs. Some keep them online to keep the option of using raw converters such as Nikon Capture which fail to read DNGs. I back them up, so I've the image data backed up in another format, but I rarely if ever see them again. Others simply delete them.
Treat yourself to a copy of Peter Krogh's DAM Book which discusses these issues and many others at length.
John
-
Having the metadata embedded in a publicly-documented file format is indeed an advantage of switching to DNG. A wider range of applications can read/write to DNG, so you're freer to choose the right tool for the job than when you're fiddling with XMP colostomy files
Here is a surprize: the adjustments in the DNG file are stored in XMP format. There is no difference between the formats.
-
Gabor, I had never thought of the backup time advantage of a sidecar vs. a complete dng. Excellent point and for me it's another reason to stay with my original raws in my workflow.
Also for people using Minolta or Sony MRW/ARW's, since they write the lens info to the "wrong" spot in the exif this information is totally lost when converting to dng. So beware and also back-up your originals if you would ever want to use this info.
Btw, there is way to fix that such that programs like lightroom can read the correct lens info. See here for a thread on Dyxum with a link to the program and tons of explanations. (http://www.dyxum.com/dforum/identifying-lenses-in-lightroom-with-alphalensinfo_topic43251_page1.html)
-
Here is a surprize: the adjustments in the DNG file are stored in XMP format. There is no difference between the formats.
Was I talking about any difference in the XMP or about where it was? Or limit the comparison to adjustments? What surprise?
[!--quoteo(post=0:date=:name=pegelli)--][div class=\'quotetop\']QUOTE (pegelli)[div class=\'quotemain\'][!--quotec--]I had never thought of the backup time advantage of a sidecar vs. a complete dng.[/quote]
Trouble is, in most circumstances t's a bogus point.
John
-
Trouble is, in most circumstances t's a bogus point.
Actually, not so bogus...if you shoot a lot and use something like Lightroom (or even Camera Raw/Bridge) if you convert all your raw files to DNG upon import, then ANY touch of the DNG by Lightroom (saving xmp) or Bridge will change the file modification date/time. For a few files it's not a big deal. But for example on the recent trip Antarctica I shot almost 12K files a mix of 1DsMIII ad Digital Rebel. While converting to DNG would save a bit of space due to compression, the savings on almost 250 gigs of raws wouldn't be a big deal. However, if I go into the Lightroom catalog and add a single keyword (or do it in Bridge) then the next time Retrospect checks the file modification date/time, it'll see those DNG had been changed and Retrospect will feel duty bound to replace all of those 250 gigs of files with new copies.
Conversely, if I use native raws and push the .xmp file updates for metadata Retrospect will still need to copy all 12K of the .xmp files but instead of copying over an entire DNG file it only needs an 8KB .xmp file for the changes.
I still use DNG when my files have been "finish edited" meaning adding all the metadata, making selects and ranking and doing all the parametric tone/color edits....at that point I have no problem converting the raws to DNG. And if I ever have to send any raws to anybody, it's always only be DNG files with a notice that altering the metadata in my DNG files constitutes a potential copyright infringement circumvention and therefore subject to the Digital Millennium Copyright Act and may be construed as evidence of an attempt at willful infringement if we come to blows over usage...
:~)
-
If you read my earlier post, Jeff, you'll see I was essentially making the same point as you about what you call "finish edited". Before then, it makes very little sense to save every keyword or other edit back to the DNG (eg via auto XMP writing) and so trigger a backup, but that can easily be avoided by targeting the backup app at new files.
Pegelli's other point about certain exif information being "totally lost when converting to dng" is also slightly off beam. The information is preserved in the DNG, but isn't displayed in any Adobe apps. It can be read from the DNG by exiftool, and Lightroom can make exiftool do so, so it is possible to write a Lightroom plug-in to add the information to custom fields.
John
-
If you read my earlier post, Jeff...
sorry, I was primarily responding to the "bogus" comment...it was kida a red flag, ya know?
In that case, never mind....
:~)
-
If you read my earlier post, Jeff, you'll see I was essentially making the same point as you about what you call "finish edited". Before then, it makes very little sense to save every keyword or other edit back to the DNG (eg via auto XMP writing) and so trigger a backup, but that can easily be avoided by targeting the backup app at new files.
Pegelli's other point about certain exif information being "totally lost when converting to dng" is also slightly off beam. The information is preserved in the DNG, but isn't displayed in any Adobe apps. It can be read from the DNG by exiftool, and Lightroom can make exiftool do so, so it is possible to write a Lightroom plug-in to add the information to custom fields.
John
John,
Would like to disagree on the two points, above;
1 : It's not bogus depending how you look at it, for instance when LR2 came out I reprocessed several older files (and still do) based on the increased capabilities of LR2 (mainly local adjustments) so every time that happens my backup is a lot quicker with sidecars vs. dng's. I can see why in your workflow it's less usefull but calling it bogus because you don't need it is a bit narrow minded.
2: According to the person who wrote the application to transfer the lens info from the "Minolta/Sony" spot to a spot where Adobe (and other) applications can see it this information is totally lost when converting to DNG. No Adobe application actually 'dispays' this info, neither on the ARW/MRW's (because it's not in the right field for these applications) nor for the DNG's because it's no longer there. Until you come with more solid proof I tend to believe the person who is writing these applications and has inspected these files in depth trying to help the community with useful programs and advice.
-
OK, "bogus" was a touch strong, but it's very far from being a cast iron reason for preferring raw over DNG. There is no need to keep backing up the DNG after every edit - in the case of LR, once upon DNG creation captures the image data, and then the catalogue needs appropriate scheduled backup (which records more information than gets in the XMP whether embedded or colostomy).
As for the data, I've neither the time nor the interest to supply "solid proof", so feel free to believe as you wish. Last time I looked for similar detail (by rudely opening up an ex-NEF DNG) I could identify the data I wanted, so I'd be very surprised if I was wrong re your file formats. Adobe might not see fit to extract or display the information, which is a shame, but they aren't in the business of destroying metadata - stuff they don't use, they typically preserve in an unparsed area. In Lightroom's case, a lot of this is dumped in an "additional metadata" table in the SQL.
John
-
The sidecar files are the icing on the cake. Do you prefer saving/archiving a 15-30 MB file after having changed the white balance (as an example), instead of saving/archiving a 2 KB small file?
Considering the importance of that small 2 KB file (small but many hours of work on my part), and the fact all the copying happens automatically while I'm asleep, I don't really find this at all an issue.
Would it be cool if I could update the 2KB file without the entire 15-30mb DNG? Sure. But its really a non issue compared to losing or finding the side car corrupted.
-
The relevance of the physically separate storage of the adjustments is not mainly in the size of the file. The issue is rather, that the original does not need to be saved again and again; or, put differently, the original remains original.
Another aspect is, how much one trusts, that the DNG is truely equivalent to the native raw file. It is, almost always. However, the DNG image data is seldom a plain copy of the native image data. The conversion includes interpretation of the native raw data format, which is often not simple, in some cases ambiguous. The camera manufacturers tun their best to make this muddy and unstable. In clear-text: the conversion is a potential source of error.
I know of only two cases, when the Adobe DNG converter created an erroneour DNG file. I forgot all details from the first case; the second case was with the lossily compressed D3/D300/D700 images. Nikon had introduced a new trick with extreme noisy images: the lossily compressed data got shortened by another lossy step. The decoding of such data by the Adobe converter had been erroneous.
Originalton Thomas Knoll:
There is a known bug that happens in some rare cases with lossy compressed Nikon D300/D3 files. Most of the time lossy compressed files work fine. We are working on a solution for the next dot release.
Those images, which had been converted in DNG and then the NEF had been thrown away were lost forever.
-
Another aspect is, how much one trusts, that the DNG is truely equivalent to the native raw file. It is, almost always. However, the DNG image data is seldom a plain copy of the native image data.
Gabor, when converting Canon CR2's to DNG, is the RAW black point preserved or clipped to 0?
Regards.
-
Gabor, I had never thought of the backup time advantage of a sidecar vs. a complete dng. Excellent point and for me it's another reason to stay with my original raws in my workflow.
Backup is the main reason I'm sticking with .cr2's and .rw2's rather than converting to DNG. Mozy Home is a nice backup system. What it isn't is fast.
-
Gabor, when converting Canon CR2's to DNG, is the RAW black point preserved or clipped to 0?
Preserved.
The official position of Adobe is, that "the image data remains unchanged". This sounds straightforward, but it is not so simple - what about lossy compressions, about rearranging the data (because the manufacturer chooses to store it not in a straight row-column order), etc. Thus some changes to the data are not only acceptable but even unavoidable. However, removing the "blacker then black" data would not be acceptable, and the DNG specification does provide for that (even though it is erroneous).
-
OK, "bogus" was a touch strong, but it's very far from being a cast iron reason for preferring raw over DNG. There is no need to keep backing up the DNG after every edit - in the case of LR, once upon DNG creation captures the image data, and then the catalogue needs appropriate scheduled backup (which records more information than gets in the XMP whether embedded or colostomy).
As for the data, I've neither the time nor the interest to supply "solid proof", so feel free to believe as you wish. Last time I looked for similar detail (by rudely opening up an ex-NEF DNG) I could identify the data I wanted, so I'd be very surprised if I was wrong re your file formats. Adobe might not see fit to extract or display the information, which is a shame, but they aren't in the business of destroying metadata - stuff they don't use, they typically preserve in an unparsed area. In Lightroom's case, a lot of this is dumped in an "additional metadata" table in the SQL.
John
John, your first point is correct, it's neither cast iron nor bogus. It depends what you want and what your priorities are. I always get a bit stickly if someone tells me I don't need someting I find usefull, sorry for that. Also agree that backing up xmp's cannot be seen as an alternative to backing up the catalog. Both are needed!
On the second point you're right, I went back to the other forum and after some additional data and discussion the person who made the claim that data were lost looked again (and more thorough) and determined it was still there and could be used. Another myth deflated
-
OK, "bogus" was a touch strong,
There is another bogus aspect of early conversion to DNG - if you do this and then go back to tweak adjustments away from whatever default or preset you used, the preview will be rebuilt every time you click 'Done' which then takes as long again as processing the file to DNG took in the first place. This accurate preview is a huge DNG strength, but it does take time if you are fiddling. Tweaking a CR2 takes no time because, as Gabor says, only the exif is changed. I suppose you can disable the preview update, certainly you can in LR.
Best to fiddle and add metadata to CR2s and then archive to DNG to kinda 'lock it in'.
-
There is another bogus aspect of early conversion to DNG - if you do this and then go back to tweak adjustments away from whatever default or preset you used, the preview will be rebuilt every time you click 'Done' which then takes as long again as processing the file to DNG took in the first place. This accurate preview is a huge DNG strength, but it does take time if you are fiddling. Tweaking a CR2 takes no time because, as Gabor says, only the exif is changed. I suppose you can disable the preview update, certainly you can in LR.
I agree that a jpeg preview that can be displayed by non-Adobe apps can be a big plus. However, I just started playing with converting to DNG and my experience in CameraRaw is that clicking "Done" does NOT update the embedded jpeg. To do this I've had to invoke the "Update DNG previews" option. Perhaps I've missed the option to choose whether the Preview is automatically updated or not?
In any case, without the jpeg update it is much faster, but at the expense of inaccurate preview of adjustments in many applications.
I'm a big fan of Photo Mechanic for it's speed and ease of use, and I'm am considering adopting DNG's as a way to allow me to keep with it for image selection after my initial edit and corrections. Any thoughts?
Ric
-
That "Update DNG previews" is now the way to do it.
As you're finding, DNG is great when you need to work with multiple applications.
John