Luminous Landscape Forum
Raw & Post Processing, Printing => Adobe Lightroom Q&A => Topic started by: Martin Ocando on February 09, 2012, 11:20:26 pm
-
So, it appears DNG might be a better option after all. Will like to hear from Jeff, Andrew and Eric about this.
http://news.cnet.com/8301-17938_105-57371809-1/adobe-offering-new-reasons-to-get-dng-religion/
-
So, it appears DNG might be a better option after all. Will like to hear from Jeff, Andrew and Eric about this.
http://news.cnet.com/8301-17938_105-57371809-1/adobe-offering-new-reasons-to-get-dng-religion/
Mike and I cover the new DNG options in the LR4 Advanced vids...but yes, Adobe (read Thomas/Eric) have advanced DNG with the most recent SDK. You can add "Fast Load" when creating a DNG which speeds opening the raw file in LR & ACR. You can also create Lossey Compressed DNG files which are linear DNGs with compression and in LR4 you can downsample a DNG based on megapixel sizes or long edges...which means you can make a DNG that is TINY by comparison to the original and maintain the ability to make ACR/LR adjustments. An IQ 180 file that is 80MP (and about 80 MBs) can be lossey compressed and downsamped to 10 MP and end up just under 3 MBs.
So, DNG continues to advance...also note that Adobe has submitting DNG to the ISO for an upcoming TIFF-EP file format update...if/when that happens, there will be a lot less reason for NOT using DNG by the major camera makers like Nikon/Canon. At this point it's really a political question not a technical question...
-
Mike and I cover the new DNG options in the LR4 Advanced vids...but yes, Adobe (read Thomas/Eric) have advanced DNG with the most recent SDK. You can add "Fast Load" when creating a DNG which speeds opening the raw file in LR & ACR. You can also create Lossey Compressed DNG files which are linear DNGs with compression and in LR4 you can downsample a DNG based on megapixel sizes or long edges...which means you can make a DNG that is TINY by comparison to the original and maintain the ability to make ACR/LR adjustments. An IQ 180 file that is 80MP (and about 80 MBs) can be lossey compressed and downsamped to 10 MP and end up just under 3 MBs.
So, DNG continues to advance...also note that Adobe has submitting DNG to the ISO for an upcoming TIFF-EP file format update...if/when that happens, there will be a lot less reason for NOT using DNG by the major camera makers like Nikon/Canon. At this point it's really a political question not a technical question...
Awesome. Just for the speed and size improvements I'll be an early adopter. I think camera makers have no real reason/advantage, economical or otherwise for not adopting DNG natively. I mean, Adobe is not making any money because of the format, is good for them, because they can streamline and optimize their software for it, is good for us, because it gives additional advantages proprietary RAW formats does not provide, and is good for the camera makers, because they won't have to dedicate any time or money on adjusting their own formats to fit newer sensors. The key, I think, is on the demosaicing, which I believe is very specific to the maker and sensor, unless I'm mistaken.
-
The key, I think, is on the demosaicing, which I believe is very specific to the maker and sensor, unless I'm mistaken.
No...actually it's specific to the raw processor. ACR and LR at equal version are the same. DxO does a linear (demosiaced) DNG which works pretty well even though DxO's demosiacing isn't...various apps such as Capture One can save out DNG (but is limited to the DNG SDK they are using).
There are two base flavors of DNG; raw DNG which is un-demosiaced raw files converted to a DNG wrapper and linear DNG which are locked into a DNG demosiacing but still in a linear gamma (with ProPhoto RGB colors). Either DNG flavor works well in ACR/LR. There is another flavor of linear DNG, a file that started in a gamma encoded color space such as sRGB or Adobe RGB. So, a JPEG opened into ACR'LR will be limited because the file started life in a non-linear gamma linear. Still, editing a JPEG in ACR/LR is still better than editing in say Photoshop.
The new additions to DNG are the added Fast Load which can be applied to raw or linear DNGs and the lossey compressed or downsampled DNG which are both linear (demosaiced) DNGs.
I suspect we'll see some cameras come with lossey DNG built in as a replacement for JPEGs–the file size reduction can be important.
Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR. The .xmp files are tiny compared to the DNG files with the XMP built in. So if you change a DNG processing settings, the whole DNG has to be backed up based on modification date changes. A proprietary raw file with a sidecar only needs to write the tiny sidecar file on backup.
I use DNG for any file that might be an interchange file (handing off to another user). I can embed IPTC metadata as well as snapshots of various image settings. Wit raw files those live only in the .xmp files which can be lost.
-
No...actually it's specific to the raw processor. ACR and LR at equal version are the same. DxO does a linear (demosiaced) DNG which works pretty well even though DxO's demosiacing isn't...various apps such as Capture One can save out DNG (but is limited to the DNG SDK they are using).
There are two base flavors of DNG; raw DNG which is un-demosiaced raw files converted to a DNG wrapper and linear DNG which are locked into a DNG demosiacing but still in a linear gamma (with ProPhoto RGB colors). Either DNG flavor works well in ACR/LR. There is another flavor of linear DNG, a file that started in a gamma encoded color space such as sRGB or Adobe RGB. So, a JPEG opened into ACR'LR will be limited because the file started life in a non-linear gamma linear. Still, editing a JPEG in ACR/LR is still better than editing in say Photoshop.
The new additions to DNG are the added Fast Load which can be applied to raw or linear DNGs and the lossey compressed or downsampled DNG which are both linear (demosaiced) DNGs.
I suspect we'll see some cameras come with lossey DNG built in as a replacement for JPEGs–the file size reduction can be important.
Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR. The .xmp files are tiny compared to the DNG files with the XMP built in. So if you change a DNG processing settings, the whole DNG has to be backed up based on modification date changes. A proprietary raw file with a sidecar only needs to write the tiny sidecar file on backup.
I use DNG for any file that might be an interchange file (handing off to another user). I can embed IPTC metadata as well as snapshots of various image settings. Wit raw files those live only in the .xmp files which can be lost.
Go it. Gotta start using sidecar .xmp files for backup.
So it will still be better, for the time being, to keep our undemosaiced DNGs, or proprietary RAWs, since newer demosaicing algorithms might come out and we can reprocess our RAWs and take advantage of them. Am I right ?
What about when the camera supports DNG natively. It will still have to update the whole DNG when backing up, or you can attach an .xmp sidecar to a DNG ?
-
So it will still be better, for the time being, to keep our undemosaiced DNGs, or proprietary RAWs, since newer demosaicing algorithms might come out and we can reprocess our RAWs and take advantage of them. Am I right ?
More or less...yes. The thing about raw processing is we still keep seeing improvements as time goes by. Which means newer processing is likely better than older processing. Fact of life...some things keep getting better.
What about when the camera supports DNG natively. It will still have to update the whole DNG when backing up, or you can attach an .xmp sidecar to a DNG ?
There's no easy way of setting a DNG NOT to update the XMP metadata internally. You can lock the DNG which should force LR/ACR to pop a .xmp file but it gets complicated when dealing with DNGs...there a preference in ACR (not sure about LR) to ignore .xmp files which you would need to set to off.
-
As I understand it, the default in Lightroom is to have 'autowrite changes to .xmp' turned off. That means that any metadata added and any history settings are stored only in the catalog. You can update the metadata in the .xmp file (or the .dng, .tif, .jpg, or .psd) by pressing Ctrl/Cmd-S.
Mike.
-
You need to be very careful to distinguish between lossy DNGs and full-blooded ones. If not, it would be very easy to create lossy DNGs for some reason like taking on the road or saving disc space by compressing frames shot for timelapses, and then confuse them with the real negatives, even overwrite non-lossy DNGs.
A couple of features in Lightroom 4 are helpful. Library's filter panel's File Type now distinguishes between the different types of DNG, and you can examine individual images in the Metadata panel's DNG view.
But outside Lightroom, you may not be able to see them at all, and in Explorer / Finder the only indications that you've a group of lossy DNGs may be file size and that the thumbnail doesn't resolve (at least until an OS update).
I have no real reason to create lossy DNGs, so I'm not doing so. If I did so, then I would be taking extra steps to prevent confusion - probably adding a filename suffix to every lossy DNG. Whatever app I am using, I want to distinguish between the full blooded DNG "120210_0123 Blea Tarn.dng" and its lossy counterpart "120210_0123 Blea Tarn LSS.dng".
I'm not saying there isn't some progress, and I welcome fast load (even if I can't see any speed difference) and I like that the small number of JPEGs I shoot with my iPad can now be saved as a sensible-size DNG. But I do feel we shouldn't dive blindly into these waters. We've got the new capabilities, they do present their risks to the integrity of our picture collections, and have we got sufficient workflow benefits? For me, no, not yet. You?
-
I suspect we'll see some cameras come with lossey DNG built in as a replacement for JPEGs–the file size reduction can be important.
Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR. The .xmp files are tiny compared to the DNG files with the XMP built in. So if you change a DNG processing settings, the whole DNG has to be backed up based on modification date changes. A proprietary raw file with a sidecar only needs to write the tiny sidecar file on backup.
I wish Nikon would give us the option of using DNG vs NEF. I really don't know any Nikon user that uses Nikon software to process NEF files. On the backup issue, you raise an important point for those of use using cloud storage as a backup. It's a lot faster to back up changed .xmp files than a DNG file that incorporates the process directions.
-
The "need" to back up the whole DNG after processing changes is dubious. At best, it's a second rate backup that omits a good chunk of your Lightroom work. Backup the DNG upon its creation, then backup your catalogue, and you've backed up all your work.
-
You need to be very careful to distinguish between lossy DNGs and full-blooded ones. If not, it would be very easy to create lossy DNGs for some reason like taking on the road or saving disc space by compressing frames shot for timelapses, and then confuse them with the real negatives, even overwrite non-lossy DNGs.
Exactly, good example. I’m excited by the new possibilities with DNG and the fast load is huge. But I think it is very important end users understand what John is saying here and not assume that just because you have a DNG, all variants are equal.
As for the article, everything seems good but I don’t really like the title.
-
What, you don't think we should argue such things on rational grounds, do you? Sheesh.
-
The "need" to back up the whole DNG after processing changes is dubious. At best, it's a second rate backup that omits a good chunk of your Lightroom work. Backup the DNG upon its creation, then backup your catalogue, and you've backed up all your work.
John, maybe I'm a bit confused then about the DNGs. I thought that the process changes that are in .xmp files associated with the RAW are contained in the DNG if one is using that format (I don't do this right now, sticking with RAW + .xmp. From what you are implying is that there is redundancy in the processing directions.
Alan
-
John, maybe I'm a bit confused then about the DNGs. I thought that the process changes that are in .xmp files associated with the RAW are contained in the DNG if one is using that format (I don't do this right now, sticking with RAW + .xmp. From what you are implying is that there is redundancy in the processing directions.
John’s point is quite valid and about the only disadvantage I see to a DNG workflow (although IMHO not a big one).
Suppose you have a DNG that contains all this useful data. You simply make a tiny update like altering some metadata. The entire DNG has to be backed up. It takes a lot longer than updating a small XMP sidecar file. I don’t think there is any way around this. You have a single container of a document and if something changes and you want to back it up, you back up the whole thing.
I do this unattended at night using a script so it doesn’t impact my time. But it is something to consider.
-
You need to be very careful to distinguish between lossy DNGs and full-blooded ones.
Very true...there is a good thread on the LR4 beta forums talking about this issue. Anybody interested should check it out. Eric Chan has taken the time to give extensive answers to a lot of questions. Info on lossy DNG? (http://forums.adobe.com/thread/947446?tstart=0)
In addition to the JPEG compression, it turns out that lossey compressed DNGs are also mapped to a different tone curve and are not linear. So, it treats the lossey compressed DNG image data more like it treats a JPEG or TIFF that is turned into a DNG.
I think great care and experimenting should be done before people jump into the waters of mass conversion of raw files into lossey compressed files.
-
The entire DNG has to be backed up.
My point is that it does not, unless you want a level of redundancy. The DNG has only had some of your Lightroom work saved back to it, so if you do choose to make this backup you are only getting a second rate level of comfort anyway. On the other hand, if you backup the original DNG when it's created, and routinely backup the catalogue, then 100% of your work is covered.
-
My point is that it does not, unless you want a level of redundancy. The DNG has only had some of your Lightroom work saved back to it, so if you do choose to make this backup you are only getting a second rate level of comfort anyway. On the other hand, if you backup the original DNG when it's created, and routinely backup the catalogue, then 100% of your work is covered.
In my case, I just back up the entire drive that has the catalog, DNG etc. So what you are saying is, just backup the catalog. The only other issue is, as I add new DNG’s to this drive, I’m not sure how easy it would be to tell my simple backup app (SuperDuper), just back up DNG’s that didn’t exist the last time you backed up. Now if only new DNGs are added, no issue. But if 100 new DNG’s were added, plus 5 were edited, then SD is going to update 105 DNGs. Since as I said, this is done unattended, I guess it isn’t a big deal.
And I do love back up redundancy.
-
My point is that it does not, unless you want a level of redundancy. The DNG has only had some of your Lightroom work saved back to it, so if you do choose to make this backup you are only getting a second rate level of comfort anyway. On the other hand, if you backup the original DNG when it's created, and routinely backup the catalogue, then 100% of your work is covered.
This still does not answer my question above. If I convert to DNG upon import and then do LR processing, where are these process changes being stored (this in the absence of .xmp files)? Are they in the catalog or the DNG? My understanding of the catalog is that its a database pointing to the location of files but maybe there is more to it.
-
Sorry, Alan, in the catalog. The writing to the DNG is optional, like saving xmp sidecars to your NEFs.
-
Sorry, Alan, in the catalog. The writing to the DNG is optional, like saving xmp sidecars to your NEFs.
You beat me to it. I found the discussion in Martin Evening's book. Thanks! I'm still on the fence with regard to DNGs.
-
Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR.
+1 for RAW + .xmp
Unfortunately it does not embed Dng profiles,
as far as I'm aware.
__
-
Unfortunately it does not embed Dng profiles, as far as I'm aware.
Correct...it can't because a .xmp file is strict text only...here's the line from a raw file set to Camera Standard; crs:CameraProfile="Camera Standard"
Actually embedding the profile inside the DNG can be a dual edged sword...once embedded into a DNG, there is no function in ACR/LR to remove it. You can change it, but not remove a DNG profile once embedded...I've asked Thomas and Eric about this and Thomas' answer was a shrug and "why would you want to do that?"...at the time I couldn't really come up with a good answer :~) But it still bothers me that you can't removed an embedded DNG profile...prolly some really weird OCD on my part.
-
Correct...it can't because a .xmp file is strict text only...here's the line from a raw file set to Camera Standard; crs:CameraProfile="Camera Standard"
.dcp can be represented in a text form ( for example SandyMC /thank you/ created an utility that gives an example = http://dcptool.sourceforge.net )... so technically you can embed DNG profile in .xmp
-
... so technically you can embed DNG profile in .xmp
But neither ACR nor LR can...
-
Actually embedding the profile inside the DNG can be a dual edged sword...once embedded into a DNG, there is no function in ACR/LR to remove it. You can change it, but not remove a DNG profile once embedded...
Why would that be a problem?
I thought I recall Eric saying at one point, it is or could be possible to embed multiple profiles in a DNG.
-
thought I recall Eric saying at one point, it is or could be possible to embed multiple profiles in a DNG.
Yes...you can embed multiple DNG profiles. But there's no function in ACR/LR to remove any DNG profiles. The problem arises when some/any of the DNG profiles are, well FUBAR...you can't get them out of a DNG.
-
Not even through Bridge's File Info panel - the Advanced tab?
-
Alan: This was written for beginner Lightroom users, but might be of interest:
http://www.wolfnowl.com/2012/01/lightroom-file-management-and-metadata/
Mike.
-
Alan: This was written for beginner Lightroom users, but might be of interest:
http://www.wolfnowl.com/2012/01/lightroom-file-management-and-metadata/
Mike.
Very useful and clear. Thanks, Alan
-
Yes...you can embed multiple DNG profiles. But there's no function in ACR/LR to remove any DNG profiles. The problem arises when some/any of the DNG profiles are, well FUBAR...you can't get them out of a DNG.
Would this be a FUBAR profile from the beginning someone embedded by mistake or one that goes FUBAR for some reason (corruption)? I’ve never seen this but I’m not sure I’d know what to expect if after the fact, a DNG profile went FUBAR. I’d be concerned if the later were possible (what would be the cause, of just the DNG profile going FUBAR and what else in the DNG could go that route?).
-
Would this be a FUBAR profile from the beginning someone embedded by mistake or one that goes FUBAR for some reason (corruption)?
It was from somebody who had flailed around in DNG Profile Editor (not me, mind you) who posted a raw with about 1/2 dozen DNG profiles embedded into the DNG–most of them screwed up (goofy color edits). The FUBAR part was that his naming was all over the map and mislabeled in the DNG drop down menu under Profile...
John, I can see al the DNG profiles–particularly if I add snapshots using different profiles but as far as I know, Bridge treats Camera Raw XMP metadata as read only...
Whether or not a normal user can get into the DNG data and remove a profile is something for Eric to say. As far as I know, you can't.
-
John, I can see al the DNG profiles–particularly if I add snapshots using different profiles but as far as I know, Bridge treats Camera Raw XMP metadata as read only...
Just checked, Jeff, and as far as I can see you are correct.
-
Correct, Jeff, we don't have any public tools that users can conveniently use to remove profiles (e.g., by name) from DNGs.
This is easy for a software developer to whip up in the public DNG SDK, though. (Maybe I'll consider distributing my tool, but it's not very user-friendly ... )
-
Is there any way to store metadata originated by Lightroom into an XMP file when dealing with native DNG's (Leica, Ricoh, Prentax...)? I did not find such an option, but may be I just missed it.
Ario
-
No, not with LR (unlike Bridge which creates sidecars if DNGs are read only) and I really don't think this is likely to change. In any case, the xmp files do not contain all your Lightroom work on those files (eg virtual copies are only in the caalogue).
-
Thanks, so in order to avoid unnecessary backups of the entire DNG file for any change made in LR is better to live the metadata inside the catalog.
Ario
-
Correct...it can't because a .xmp file is strict text only...
... "why would you want to do that?"
It is among several points, about what to take care, when moving to a new computer.
Customized Dng profiles will not travel over automatically with Raw+.xmp
Not a big issue, just something to be aware.
__
-
John’s point is quite valid and about the only disadvantage I see to a DNG workflow (although IMHO not a big one).
Suppose you have a DNG that contains all this useful data. You simply make a tiny update like altering some metadata. The entire DNG has to be backed up. It takes a lot longer than updating a small XMP sidecar file. I don’t think there is any way around this. You have a single container of a document and if something changes and you want to back it up, you back up the whole thing.
Some may be interested to know there are block-level backup software solutions that work extremely well for things like DNG files. This type of software keeps track of each data-block of a given file and updates only the bytes that have been altered. If I add some IPTC fields to a DNG it will take a snapshot of the file as it currently exists in backup and add a new delta-value snapshot for the few data-blocks that changed. I use this kind of software for both off-site internet backups as well as a bare drive I keep in town at another site. I once updated some global IPTC values on about ten thousand DNGs to the internet storage. The software indicated my upload speed was hundreds of times faster than I know my connection is...As it turns out it calculates throughput based on file size but since it was only updating the tiniest sliver of data, all ten thousand or so DNGs were backed up online in a few hours. It's effectively the same as just backing up the sidecar xmp files.
I've left out the brand names to avoid the appearance of selling something...but this technology is out there if you want. I've been very pleased with it for a couple years now. And remember, a single backup is no backup at all. I have several copies in 4 cities...occupational hazard. Good Luck!
David