Luminous Landscape Forum
Raw & Post Processing, Printing => Digital Image Processing => Topic started by: wolfnowl on November 29, 2011, 03:26:21 pm
-
A worth read from two list members!
http://www.natcoalson.com/blog/2011/11/29/my-adobe-dng-chat-with-eric-chan/
Mike.
-
The conclusion sounds like he wasn't listening to a word Eric was saying.
-
Mike, thanks for posting this - it is a useful reference piece. And Eric, thanks for taking the time to explain all that.
-
Nice of you to find and post this, Mike.
I noticed in the comments section Gene McCullagh raised some concerns about DNG I wasn't aware of concerning xmp side car files and backing up files.
So to get this straight, if I were to edit a DNG version of my Pentax K100D PEF in ACR/LR, there would be no xmp side car and the edits would be embedded in the DNG file itself? Is this correct?
-
So to get this straight, if I were to edit a DNG version of my Pentax K100D PEF in ACR/LR, there would be no xmp side car and the edits would be embedded in the DNG file itself? Is this correct?
Correct if you saved out the settings to XMP. That will update the settings in the DNG and cause the file to be marked as modified so a subsequent backup based on changed files would require the whole DNG to be backed up–even if only the settings were changed. If you use Lightroom without saving the settings to the XMP but only save it in the database, the file would not need backup because the changes would only be in the database (which would be changed and hence get a backup). Same deal for JPEG & TIFF files whose XMP is written into the file.
-
If you use Lightroom without saving the settings to the XMP but only save it in the database, the file would not need backup because the changes would only be in the database (which would be changed and hence get a backup). Same deal for JPEG & TIFF files whose XMP is written into the file.
Jeff, that is good to know. It's a subtle aspect that I've not seen explained before.
Thanks for the information.
Alan
-
You CAN set Lightroom to automatically write changes to XMP in the preferences (it's off by default), but there's a trade-off in terms of memory usage while it's doing that. Otherwise Ctrl/Cmd-S will handle that nicely for you. But yes, as Jeff clearly explained, Lightroom is a database program and by default stores the information in its database (catalog).
Mike.
-
If you use only LR to process RAW files then does it matter if the information is stored only in the database and not in a sidecar file?
-
If you use only LR to process RAW files then does it matter if the information is stored only in the database and not in a sidecar file?
Note only "some of" the information is ever stored in the sidecar.
-
maybe off topic or thread stealing (sorry)......
is there any easy way to 'convert' an xmp based LR hierarchy to a 'put the data in the .lrcat' ? Or do I have to export everything, change settings and then reimport to a new catalog?
-
Note only "some of" the information is ever stored in the sidecar.
Not sure that matters in this context. If a user only uses LR for the RAW adjustments, exports a TIFF/PSD (or JPEG) then does everything else in PS or some other piece of software there's no need for saving out to the XMP file. Isn't that the case?
But if there's information related to edits made in LR that is only retained in the LR database then the XMP is useless anyway. Isn't it?
Does ACR work differently from LR in how it handles DNG files? This article on the Adobe site, http://help.adobe.com/en_US/Photoshop/11.0/WSD94FB319-761D-4e9a-BC8D-24DF7EBC05B4.html seems to indicate that it is possible to save edit information to a DNG from ACR to a sidecar. But if I read Jeff's explanation correctly, in LR that's not possible?
-
Does ACR work differently from LR in how it handles DNG files? This article on the Adobe site, http://help.adobe.com/en_US/Photoshop/11.0/WSD94FB319-761D-4e9a-BC8D-24DF7EBC05B4.html seems to indicate that it is possible to save edit information to a DNG from ACR to a sidecar. But if I read Jeff's explanation correctly, in LR that's not possible?
ACR can save out a separate .xmp text file from the ACR flyout menu, LR does not do that. But you don't really want to get into a DNG plus .xmp situation because it can become complicated, which one is correct? The xmp metadata in the DNG or in the .xmp file? Camera Raw actually has a preference setting to allow you to ignore .xmp for DNG files, LR doesn't.
-
Thanks, Jeff. Appreciate the clarification.
-
Something I've discovered about the way ACR handles xmp side cars with regard to keywording. I may have to hunt down a preference setting somewhere to change this so it doesn't happen but this is what I found so far.
I just started keywording all my Raw PEFs giving each visually descriptive names in Bridge CS3 (I don't do events or weddings, just individual fine art type images). Upon doing a search in Snow Leopard (not Bridge) using very unique keywords I know would only be embedded in certain images, all I get are the xmp side cars and NONE of the PEF images so I can see their previews. Bummer. I can do a search in Bridge and I get the image thumbnails associated with the xmp side car.
Wonder if I switched all PEF's to DNG and embedded their xmp side car info which contain edits, camera info and keywords that I'll be able to use the OS to do a search and get the same thumbnail results as Bridge.
-
Boo-YAH! It worked. I converted one test PEF with embedded keyword to DNG and it now shows up in an OS search using that embedded keyword.
-
The conclusion sounds like he wasn't listening to a word Eric was saying.
Ben, I can certainly understand why you would say that. Please let me clarify the statements in my conclusion.
First, the entire purpose of my conversation with Eric was to determine if the raw data from my Canon DSLRs is being modified when converting to DNG. It just took me a long time to get around to specifically asking that.
From Eric's responses, my current understanding is that with "mainstream" DSLRs like those from Canon and Nikon, the scene-referred raw data should NOT be getting modified when it's going into the DNG. In other words, the raw data in the DNG is the same as the raw data in the original CR2. But with other cameras, this will certainly not always be the case.
If after reading Eric's responses you came to a different conclusion please help me better understand. DNG is not nearly as straightforward as I thought it was.
Thanks!
-
Ben, I can certainly understand why you would say that. Please let me clarify the statements in my conclusion.
Nat the whole exchange was somewhat like you were trying to "press" Eric to assure you in something for whatever reason and Eric was skillfully trying to thread the fine line between many rocks and hard places...
-
Well I know from just messing around converting my Raw Pentax PEF's to DNG, I can affect the JPEG preview Bridge uses, not other apps that can open and allow viewing of the DNG file like say Apple's Preview. I get two completely different previews but that's what I got with the original PEFs as well.
The main question to answer is what exactly is happening to the preview used to edit by in each converter of choice. I know my Pentax's converter uses the camera settings it only has access to while other converters offer their own recipe according to their default settings.
But an added bonus of DNG is that Bridge CS3's main Preview Panel shows the effects of sharpening much like how it's shown in Photoshop at all zoom levels. ACR (my converter of choice) only shows sharpening effects at 100% view, but Bridge now shows it at any view depending on size of it's GUI on screen while the same edited PEF the DNG was converted from doesn't show the sharpening even when I have "Use High Quality Previews" chosen in Bridge Prefs.
Frankly what I gathered from Nat's entire article is the operative statement made by Eric..."It depends". That's nailing it down.
So the specifics between other camera models and brands in how that camera delivers Raw data for ACR/LR or any other converter to convert Raw to DNG is not specifically defined and so we're really back to square one when it comes to knowing exactly what happens.
But really as long as I can edit the DNG in my converter of choice and have it look the same and behave the same using that converter's editing tools, does it really matter knowing what happens under the hood?
-
But really as long as I can edit the DNG in my converter of choice and have it look the same and behave the same using that converter's editing tools, does it really matter knowing what happens under the hood?
Yes.
What happens a few generations down when you decide to refresh your image with another converter that does not support converted DNGs?
RAW converters get better over the years. Only the original RAW file has all the information. A normalized DNG does not.
cheers
afx
-
Yes.
What happens a few generations down when you decide to refresh your image with another converter that does not support converted DNGs?
RAW converters get better over the years. Only the original RAW file has all the information. A normalized DNG does not.
cheers
afx
This runs in opposition to what Adobe (and others) have stated.
If I convert a raw file to DNG, what's missing? What information is contained in the original raw that isn't carried over to the DNG?
-
Yes.
What happens a few generations down when you decide to refresh your image with another converter that does not support converted DNGs?
RAW converters get better over the years. Only the original RAW file has all the information. A normalized DNG does not.
cheers
afx
Andreas - what makes you think the best raw converters going forward will not support a widely-used format such as DNG? And why do you think there is a risk that these converters won't have all the information needed from the DNGs to do excellent conversions? It's fine to say there may be risks of this or that, but it doesn't mean much without being able to substantiate such risks in terms of their reasonableness or probabilities of occurrence.
-
Andreas - what makes you think the best raw converters going forward will not support a widely-used format such as DNG? And why do you think there is a risk that these converters won't have all the information needed from the DNGs to do excellent conversions? It's fine to say there may be risks of this or that, but it doesn't mean much without being able to substantiate such risks in terms of their reasonableness or probabilities of occurrence.
Nikon NEF files are converted slightly differently using Nikon software vs Adobe. I'm not saying that one is better than the other because I really haven't tested this (and probably don't really know what to look for in any event). If I convert and toss out my NEF files, I forever give up on the use of Nikon software to do conversions.
I guess I would also pose the question differently. Adobe made DNG an open format but I think (don't know for sure) that Leica is the only camera that saves images in DNG. The more critical question is why don't more camera mfrs support DNG?
-
If I convert a raw file to DNG, what's missing? What information is contained in the original raw that isn't carried over to the DNG?
My understanding is the proprietary data the 3rd party converter can't use anyway.
-
Alan, that's true of any RAW converter. DxO will process the same file differently from Adobe or C1. Right? The camera maker's own software will also process Raw files differently because it has access to the in camera settings that third party software doesn't make use of. I think Hassy can also record in DNG.
Andrew, that's what I would have thought. In which case it doesn't really matter. And Adobe (don't know about others) have the camera specific profiles available that are intended to mimic those other settings anyway.
-
Nikon NEF files are converted slightly differently using Nikon software vs Adobe. I'm not saying that one is better than the other because I really haven't tested this (and probably don't really know what to look for in any event). If I convert and toss out my NEF files, I forever give up on the use of Nikon software to do conversions.
You're not likely to be missing anything.
-
This runs in opposition to what Adobe (and others) have stated.
Not really. Read the transcript of what Eric said in the linked interview.
If I convert a raw file to DNG, what's missing? What information is contained in the original raw that isn't carried over to the DNG?
Just like an MP3 can not be converted back to the WAV it was created from you can not create the NEF or CR2 from a DNG (unless you did a complete embed).
Converted DNGs do not contain dead pixels for example.
You might also have a look at how the DNG converter evolves which is a clear sign that the data is not identical, there is a really nice example in this article: http://www.libraw.org/articles/2-ways-to-nowhere.html
cheers
afx
-
Not really. Read the transcript of what Eric said in the linked interview.
I did. More than once. For practical purposes, it contains the same data. Not sure the calibration data from lower end cameras being cooked into the DNG is necessarily a bad thing. Particularly if not all raw conversion software can read it.
Just like an MP3 can not be converted back to the WAV it was created from you can not create the NEF or CR2 from a DNG (unless you did a complete embed).
Yes, I know. Not convinced it's a bad thing in this context.
Converted DNGs do not contain dead pixels for example.
What happens to them?
You might also have a look at how the DNG converter evolves which is a clear sign that the data is not identical, there is a really nice example in this article: http://www.libraw.org/articles/2-ways-to-nowhere.html
cheers
afx
I'll look at the article, perhaps on the weekend, when I have more time.
[/quote]
-
Andreas - what makes you think the best raw converters going forward will not support a widely-used format such as DNG?
Check how many do so now?
And best is relative...
And why do you think there is a risk that these converters won't have all the information needed from the DNGs to do excellent conversions?
Again, excellent is subjective. It is no longer the original.
Just like I will not trash my CD collection by converting to MP3, I will not convert my images to DNG.
cheers
afx
-
My understanding is the proprietary data the 3rd party converter can't use anyway.
Actually for most raw file formats (such as CR2 and NEF) DNG will move most the proprietary metadata into the DNG safely...it just can't be used by ACR/LR. So that data isn't "gone" it's just stored. The camera maker's software COULD be written to read DNG.
Capture One can read DNGs–it can't use the processing engine in ACR/LR but it can use it's own engine for adjusting the DNGs. However, as far as I know, it can only read DNGs from cameras it already supports, so it's not a full DNG SDK implementation.
Personally, I don't toss my original raw files. For use in Lightroom it's useful to keep the original raws and use .xmp sidecar files since that's a much smaller backup. However, if I send raw files to anybody for any purpose, I only send DNG files so I can embed not only the settings but also IPTC metadata into the DNG file. In that case, DNG makes for a much better interchange format. And, DNG is still relatively new (released in 2004) and has gone through 3 major revs (now at DNG v1.3). I would expect that there are a lot more things Thomas Knoll wants to put into DNG (well, ok, I know that for a fact but can't talk about it :~).
And Leica is not the only company to adopt DNG, here's a list from the wiki page (http://en.wikipedia.org/wiki/Digital_Negative) for DNG:
Casio supports DNG in their Exilim PRO EX-F1 and Exilim EX-FH25.
Leica's Digital Modul R for the Leica R8 or Leica R9 and the Leica M8 or Leica M9 natively support the DNG format.
MegaVision E Series Monochrome back.
Panoscan MK-3 digital panoramic camera.
Pentax supports DNG in their K10D, K20D, K200D, K2000, K-7, K-x, K-r and K-5 DSLR cameras.
Ricoh supports DNG in the Ricoh Digital GR, considered a professional compact, and the Ricoh Caplio GX.
Ricoh GXR is the latest, new and solely system of mirrorless interchangeable lens camera unit use also DNG.[43]
Samsung supports DNG in their Pro815 "prosumer" camera and GX-10 and GX-20 DSLR cameras.
Sea&Sea DX‐1G underwater camera.
Seitz Roundshot D3 digital back, used in cameras such as the 6×17.
Silicon Imaging Silicon Imaging Digital Cinema SI-1920HDVR.
Sinar now uses DNG as the raw file standard for their eMotion series of digital backs.
Also note that Adobe has offered DNG to the ISO to be a part of the revised TIFF/EP standard (ISO 12234-2). That's a slow moving process though. TIFF/EP is what most all raw file formats currently are based on including Canon, Nikon, Sony, Panasonic, Olympus, Fuji.
Again, it's useful to talk about the technical issues of DNG separately from the political issues.
-
Question to Eric Chan (in case he shows up in this thread): as far as I know, most native RAW formats don't inform about each sensor's precise saturation point, i.e. the RAW value (or value range) in which RAW clipped data is encoded. RAW developers should know this value and apply it in the RAW conversion, so RAW developers have an internal table with sat points for each camera.
I guess the DNG file needs to encode the saturation point as metadata so that the RAW developer uses it to process the RAW file. Does Adobe DNG Converter have a fixed sat value for every camera? what if this value is not correct/optimum? how is the value chosen?.
Just to illustrate the consequences of chosing a wrong saturation point, a parameter the user cannot change in RAW developers since it is internal to the software:
This Canon 40D image has been developed using a wrong saturation point (in this case the one DCRAW uses internally for that camera):
(http://www.guillermoluijk.com/tutorial/satlevel/sat1.jpg)
This is the result when repeating the RAW development with a correct saturation point:
(http://www.guillermoluijk.com/tutorial/satlevel/sat2.jpg)
This is a RAW development from a Canon 7D using a version of ACR not yet fine tuned for that camera; same problem, magenta cast in the RAW clipped areas:
(http://www.guillermoluijk.com/tutorial/satlevel/canon7d.jpg)
The same problem arises when developing Fuji S3/S5 Pro RAW files using ACR.
Regards