A DNG is a DNG?

[sandymc]

The way the various pieces of software works is roughly as follows: If (for example) you were to take a level compressed NEF from a D80 and converted to a DNG:

1. DxO would give you a demosaiced, linear file - effectively a TIFF in a DNG wrapper.

2. Any of the Adobe products (LR, DNG converter, etc) will give you a DNG with the same mosaiced (Bayer pattern for the D80), compressed, raw data as the original NEF

3. C1 will give you a mosaiced (Bayer pattern for the D80) linear (decompressed into a 16-bit space) file.

All of the above are valid DNG files......

Sandy

[Panopeeper]

1. DxO would give you a demosaiced, linear file - effectively a TIFF in a DNG wrapper

Most native raws and all DNGs are TIFFs (TIFF is the wrapper, not DNG).

[Panopeeper]

Well, the whole intent and design of Camera Raw was and still is a file format import plug-in...the reason to build it in the first place was to be able to open proprietary raw file formats that Photoshop can't read

This is an explanation, not a justification.

Anyway, you speculated enough about my ACR is a nightmare of software architecture, justifiable only with the wish to sell the newer and newest version of PS/LR even if you don't need the features. All the above has nothing to do with the basic error of the architecture: that the cameras' "proprietory handling" is anchored in the ACR code and the support of a new camera can not be added without replacing ACR.

The consequence is much more than technicality: it means, that the support of a new camera can not be added to an older version of ACR, and that is the point.

[DarkPenguin]

On the plus side a task specific engine should be faster.

[Panopeeper]

On the plus side a task specific engine should be faster.

? This has nothing to do with what I posted.

[DarkPenguin]

? This has nothing to do with what I posted.

Whatever.

[Mark_Tuttle]

My understanding is that Capture One 4.5 is writing mosaiced DNG files, not linear DNG files. However, I do not believe that Capture One is embedding the same color profile into the DNG as the one that Capture One itself uses to process the images. That is why you are seeing different results.

I guess the question I have to ask myself is why Phase One bothers with DNG output at all then.  If Adobe products already read the Phase One TIFFs and will change them into DNG why would PO even bother to output their flavor of DNG that essentially doesn't provide any advantage?  I can't use the LLC corrections of PO Pro unless I output as a TIFF, I can't make corrections unless I output as a TIFF ....  what is the point?  Is there something that my blind eyes are missing?  Archival issues only arise if Adobe software can no longer read a PO Raw TIFF, and since they already do then why should PO bother? Having Capture One save as a DNG just to open back up in Capture One makes no sense either.

(and I know the next answer from Jeff is "You would have to ask Phase One")

[Panopeeper]

If Adobe products already read the Phase One TIFFs and will change them into DNG why would PO even bother to output their flavor of DNG that essentially doesn't provide any advantage?

The main advantage of DNG is, that it offers a common platform for adjusted raw data. This means, that a certain raw processor/preprocessor does not need to carry out all the functions the usual raw converters do but "add their knowledge" and let the DNG-aware raw processor do the rest.

Example: certain MFDBs create a "camera profile" describing certain characteristics of a specific camera copy. This profile needs to be involved in the raw conversion, but this is not covered by DNG. The raw (pre)processor can apply the "knowledge" from this specific source and create a DNG file, which is not "really raw" but "raw enough" to be processed by any DNG aware raw converter.

There is no written rule how far the DNG preprocessor can go, but the farther it goes, the less freedom the next stage has.

For example the "small raw" creature of some cameras creates a "raw" file, which is de-mosaiced. This step offers certain advantages, but it removes the option of choosing from different demosaicing algorythms, as some raw converters offer.

[NikosR]

This is an explanation, not a justification.

Anyway, you speculated enough about my ACR is a nightmare of software architecture, justifiable only with the wish to sell the newer and newest version of PS/LR even if you don't need the features. All the above has nothing to do with the basic error of the architecture: that the cameras' "proprietory handling" is anchored in the ACR code and the support of a new camera can not be added without replacing ACR.

The consequence is much more than technicality: it means, that the support of a new camera can not be added to an older version of ACR, and that is the point.

Well we had this discussion with Schewe and Eric Chan in another thread. Me suggesting that adding new camera support in older versions of a raw converter does not necessarily mean providing what most in the sw industry understand as backwards compatibility (which is a much more general concept and which indeed is a huge issue) but I was slugged off as not knowing what I'm talking about. They were trying to convince me that the decision not to support newer cameras in older versions (while they do offer support through their DNG converter thingy)  was not mainly a political thing. I still maintain my position that providing camera support in older versions has little to do with backwards compatibility (if you have designed your software right) and the decision not to do it is 99% a political decision but of course this might not be correct with regards to ACR sw architecture...

Then again, I might not know what I'm talking about. It is hard to argue as an outsider with people who have insider knowledge of something. It does not necessarily mean they are right, it just means they can use their insider knowledge to argue to their advantage.

[madmanchan]

The main disadvantage of linear DNG is the larger file size and the inability to use a different demosaic algorithm after the fact.

However, all of the standard operations can still be applied in full, such as white balance, colorimetric characterization, highlight recovery, etc.

[Mark_Tuttle]

The main disadvantage of linear DNG is the larger file size and the inability to use a different demosaic algorithm after the fact.

However, all of the standard operations can still be applied in full, such as white balance, colorimetric characterization, highlight recovery, etc.

Let me go backward a bit and explain how I came to ask this question.  The follow is from an email to Doug Peterson at Capture Integration, who thankfully is far more helpful that the actual Phase One staff.

My Question:

When a Phase One P45 file is processed in Capture One 4.1.2 I have Prophoto RGB as the icc output source if I am processing as a 16-bit TIFF. When I change my processing option to DNG it also shows (at the bottom just above the 'process' bar) Prophoto as the icc profile, which seems kind of odd to me because I wouldn't think that a DNG would have an imbed ICC profile. When I checked (while in TIFF process option) all of the output options there is "Embed Profile" at the top and all of the other possible profiles listed.

In any case, I am finding that if I open a Capture One processed DNG in Lightroom 2 or ACR 4.6 that any changes I have made in Capture One are not recorded in the DNG; the program seems to simply take the P45 TIFF and change it into DNG, so I suspect that any LLC or Lens Corrections have no information transferred to Adobe products either.

So I went ahead and took a P45 file and processed it in C1 4.1.2, then opened the Adobe DNG profile editor and ran the GMB card balancer, created the recipe and saved it in the file where LR2 looks when you go to the camera calibration section of the Develop module. The problem is that it only shows "Embeded" with no other calibration choice. This feature works fine if I take the P45 Tiff directly to Adobe DNG convertor and then into LR2, as it shows the DNG calibration choices I've created.

The conclusion seems to be that the DNG created by Capture One 4.1.2 is creating a different DNG file that that of Adobe DNG Convertor. I even tried taking a DNG processed in Capture One and running it through DNG Convertor but there was no difference.

In order to use the LLC and Lens Correction features of Pro, whether 3.7.9 or 4.5, it appears that the only option is to process in Pro, export as TIFF and then make corrections to the TIFF in Adobe products.

Is this an accurate assessment or have I screwed up somewhere (which is totally possible smile.gif )?

Doug's response:

When you switch to DNG for output you are using the default embeded camera profile. The fact that it shows another profile name in the summary above the process bar when DNG is selected is a bug. Unless I am mistaken the DNG standard allows for an embedded device profile to allow the receiving program to translate appropriately into whatever working space is used (profoto, adobe 98, srgb etc).

No information is passed from C1 to other programs through the DNG. It is a clean slate. The white balance is an exception as receiving programs *should* read the white balance with which the shot was ORIGINALLY shot (not what you've changed it to since).
The fact that the two programs create a slightly different DNG is not inherently against the specifications for DNG. Really DNG has not (likely will not) come out to be the universal exchange and archival mechanism everyone hoped that it would be.

So in short everything you said was right (excepting that the ICC profile listing for DNG is a bug). You would need to use C1 and output to 16bit if you wanted to use the lens correction and LCC tools but still wanted to work primarily through Adobe products.

So what I'm getting out of this, along with the responses that I've received so far is that if I am using some products then I can get "all of the standard operations can still be applied in full, such as white balance, colorimetric characterization, highlight recovery, etc.", but this doesn't happen between all products that read DNG.  I understand the differences in DNG construction now, and suspected as much from my experiences with various TIFF files in the past.

Is this a fair assessment?  If so it leads me back to wondering why Phase One bothered to incorporate DNG as an output option if C1 only passes through WB to other programs that read DNG. Unless there is some sort of 'sidecar file' that can be placed in a hierarchy and read by all DNG applications there seems to be a level of practicality missing.

[Panopeeper]

Mark,

you need to differentiate between adjustments acting directly on the data and those, which can or have to be recorded as metadata. For example vignetting could be applied directly on the raw data (non-demosaiced as well), while saturation can not be changed on that basis, it needs to be recorded in some form, like XMP.

If you upload such a DNG, I can show you, what information the DNG creator added to the data. It is certainly not correct, that No information is passed from C1 to other programs through the DNG. It is a clean slate. "no added information" would not be a "clean slate" but an erroneous, unusable DNG file. Perhaps "no information about adjustments made in the C1 session is included in the DNG file" is correct.

[madmanchan]

Mark, yes, Doug is mostly accurate and fair.

The distinction that many photographers often miss regarding DNG is the distinction between the (1) essential image data and metadata required to decode the image and (2) raw converter-specific settings, such as Camera Raw XMP metadata, or Capture One settings, or Silkypix settings.

#1 is really the equivalent of the negative. It basically says, "tell me what I need to read the image data properly" but doesn't include any raw-to-output rendering adjustments the photographer may have made (just like with film: the negative doesn't tell alone tell someone like Ansel Adams how to develop or print that negative). This is essentially what Capture One is writing out. It is sort of like Phase One's version of DNG Converter.

#2 is optional and exists for the sake of convenience. Many photographers prefer to store raw-to-output rendering settings with the negative, and DNG's structure (based on TIFF 6.0 and compatible with TIFF-EP as well) makes it possible to do so. However, it is NOT a goal of DNG to make all renderings from all raw converters look the same! Folks who think that Adobe is attempting to dictate the rendering process are gravely mistaken. Raw converters are encouraged to use their own (hopefully advanced) image processing algorithms to perform the raw-to-output rendering. There are tags in the DNG which can be used to produces a consistent baseline rendering across converters, but raw converters can certainly (and do) apply their own controls on top of that.

[Mark_Tuttle]

Mark, yes, Doug is mostly accurate and fair.

The distinction that many photographers often miss regarding DNG is the distinction between the (1) essential image data and metadata required to decode the image and (2) raw converter-specific settings, such as Camera Raw XMP metadata, or Capture One settings, or Silkypix settings.

#1 is really the equivalent of the negative. It basically says, "tell me what I need to read the image data properly" but doesn't include any raw-to-output rendering adjustments the photographer may have made (just like with film: the negative doesn't tell alone tell someone like Ansel Adams how to develop or print that negative). This is essentially what Capture One is writing out. It is sort of like Phase One's version of DNG Converter.

#2 is optional and exists for the sake of convenience. Many photographers prefer to store raw-to-output rendering settings with the negative, and DNG's structure (based on TIFF 6.0 and compatible with TIFF-EP as well) makes it possible to do so. However, it is NOT a goal of DNG to make all renderings from all raw converters look the same! Folks who think that Adobe is attempting to dictate the rendering process are gravely mistaken. Raw converters are encouraged to use their own (hopefully advanced) image processing algorithms to perform the raw-to-output rendering. There are tags in the DNG which can be used to produces a consistent baseline rendering across converters, but raw converters can certainly (and do) apply their own controls on top of that.

Thanks, Eric. Great answer.

I'm not expecting all renderings the same any more than I would expect any brand of inkjet paper to give me the same print, but I'm hoping for something in the ball park.

And I'm not expecting Adobe to attempt to take over the world ... after all, we all know Google is doing that  

My original hope had been to make LLC and Lens Corrections in Capture One, save it as a DNG, then open up a DNG Editor profile in Lightroom 2 or Creative Suite CS4 Photoshop.  Alas .....

[Panopeeper]

It is hard to argue as an outsider with people who have insider knowledge of something. It does not necessarily mean they are right, it just means they can use their insider knowledge to argue to their advantage.

I do not dispute the claim, that the present architecture of ACR does not allow for addition of modules for input file interpretation. I am sure it does not, and that is what I am criticizing.

When I posted this on the Adobe forum, Thomas Knoll's answer was something like "What do you want? A plug-in for a plug-in?" Well, yes, as the minimum solution, although ACR being a plug-in in the present plug-in architecture is a serious mistake on its own.

[barryfitzgerald]

Hmm, well DNG has started to grow on me a tad. esp for saving space and compressing native raw files down.

Bottom line, or rather question is, do we lose anything at all with compressed DNG???

So what to use, convert to linear image, or preserve raw image?

[NikosR]

Hmm, well DNG has started to grow on me a tad. esp for saving space and compressing native raw files down.

I believe that both Canon and Nikon offer losslesly compressed raws. Why would you want to use DNG for that? Unless of course you are shooting some other camera wchich does not output compressed raws.

[NikosR]

Quoting from the ACR 5.2 announcement:

'

   **With the release of Camera Raw 5.2 (and upcoming Lightroom 2.2 release) there is an important exception in our DNG file handling for the Panasonic DMC LX3, Panasonic DMC FX150, Panasonic DMC FZ28, Panasonic DMC-G1 and Leica D-LUX 4. In this release the native, proprietary files from these cameras can only be converted to linear DNG files. A linear DNG file has gone through a demosaic process that converts a single mosaic layer of red, green and blue channel information into three distinct layers , one for each channel. The resulting linear DNG file is approximately three times the size of a mosaic DNG file or the original proprietary file format.

    This exception is a temporary solution to ensure that Panasonic and Leica's intended image rendering from their proprietary raw file format is applied to an image when converted DNG files are viewed in third party software titles. The same image rendering process is applied automatically in Camera Raw 5.2 and in Lightroon 2.2 when viewing the original proprietary raw file format.

    In a future release Adobe plans to update the DNG specification to include an option to embed metadata-based representations of the lens compensations in the DNG file, allowing a mosaic DNG conversion. In the interim Adobe recommends only converting these files to DNG to allow compatibility with third party raw converters, previous versions of the Camera Raw plug-in or previous versions of Lightroom.
'


Posts by Schewe and Eric Chan in various threads had almost convinced me that the current DNG specification already allows for the proper storage of propriatery metadata (in maker notes) and that's all that is needed for any manufacturer to accept the DNG format as a generic and universal file container, a panacea for everything bad under the sun (raw file format wise).

Now we get this statement from Adobe (obviously having to do with propriatery metadata about lens based corrections). I am confused. How about you?

[madmanchan]

Nikos, the distinction is whether the "lens compensations" mentioned in Tom's note are documented or not.

(Everything I wrote above regarding maker notes remains true.)

Perhaps you can clarify what you're confused about.

[skid00skid00]

This is an explanation, not a justification.

Anyway, you speculated enough about my ACR is a nightmare of software architecture, justifiable only with the wish to sell the newer and newest version of PS/LR even if you don't need the features. All the above has nothing to do with the basic error of the architecture: that the cameras' "proprietory handling" is anchored in the ACR code and the support of a new camera can not be added without replacing ACR.

The consequence is much more than technicality: it means, that the support of a new camera can not be added to an older version of ACR, and that is the point.

Of course it can.  You wrap your new-fangled raw in dng, and older versions of acr can convert it.
And I am quite sure that you know *why* Adobe doesn't waste profits on updating old, major releases of software...