Lightroom: DNG or RAW

[Daniel Browning]

Thanks for sharing, Gabor. By the way, your posts are often rife with opportunity to quote out of context:

...any software, no matter from which corner it comes, is guaranteed error free...

 Reminds me of Knuth for some reason: "Beware of bugs in the above code; I have only proved it correct, not tried it."

But if you go to process that CR2 again using the new version of the software, then even though you didn't touch any controls, the appearance of that CR2 has changed since you last looked at it.

Thanks for the post, Eric. The appearance changing with new versions is expected and normal. The reason I quoted that thread was because it shows that the appearance changes differently for DNG than for the original raw, contradicting Nick Rains' position that archiving the DNG would be the same.

[madmanchan]

Now that I understand the lie of parametric editing I'm fine with this.

Can you please clarify? In my view, parametric editing has never been about preserving rendered results (even approximately). Rather, it is largely about preserving the original raw data intact, regardless of how the software chooses to interpret it. That is, the software can continue to evolve and improve, but the original data should not change.

[DarkPenguin]

Can you please clarify? In my view, parametric editing has never been about preserving rendered results (even approximately). Rather, it is largely about preserving the original raw data intact, regardless of how the software chooses to interpret it. That is, the software can continue to evolve and improve, but the original data should not change.

Why do you need parametric editing for that?

[Panopeeper]

"Beware of bugs in the above code; I have only proved it correct, not tried it."

I do swear that my programs are error free, but I would not bet on that.

[Nick Rains]

The reason I quoted that thread was because it shows that the appearance changes differently for DNG than for the original raw, contradicting Nick Rains' position that archiving the DNG would be the same.

Seems to me that if you use different software to try and get the same results, you may or may not be successful. I admit to being surprised that a 3 year old DNG produced different results compared to processing the original CR2 RAW. Maybe Eric has an opinion on why this might be so.

However, just to check my facts, I just did this:

Convert a file to DNG.

Load both this DNG and it's corresponding CR2 into ACR5.2

Open both into PSCS4 having synched the settings, or left them both on default as I did.

Compare using difference blending mode.

Zero difference.

So I stand by my assertion that converting to DNG makes no difference; but evidently only using exactly the same software for both DNG and CR2. Therefore it's not the DNG format that is the 'culprit', it's the software that processes it.

[madmanchan]

Why do you need parametric editing for that?

You don't.

Parametric editing is not a requirement for non-destructive workflows. However, it offers several advantages over pixel-based edits. First, the edits themselves are much more compact. Hence you can work with multiple renderings of a given raw file (e.g., a color version, a neutral B&W version, tinted B&W version, versions with different color profiles, etc.) with little overhead; syncing edits (e.g., white balance, a given split-tone setting, hue twist, etc.) across a large number of images is also very efficient.

Another advantage, in my opinion, is that the image processing application (e.g., a raw converter) can automatically arrange the image processing operations in the most effective and/or efficient order, regardless of the order in which you make the edits. This all simply falls out of the idea that the application is working with a set of instructions/parameters. This has significant implications for users because it relieves users from accidentally being, for instance, in the wrong color space when performing certain types of operations (often you want to be in a linear RGB space for certain types of operations, in a non-linear opponent space for other types of operations, etc.). Having the user perform these conversions back-and-forth in a direct pixel-based representation, whether manually or automatically via an action/macro, is quite error-prone and can have many unwanted side effects.

[Nick Rains]

Follow up.

I cannot reproduce the results of that DPReview thread.

I dug back into my archives, I archive the original RAW to DVD and it's corresponding DNG to external HD.

OK, DNG created back in 2006, opened along with it's original CRW file at exact same settings in ACR5.2 using the old ACR2.4 profiles.

Zero difference, completely black difference blend.

So, by my own experience and testing, DNG and RAW are entirely equivalent, even from old D60 files, as long you use the same RAW processor. AFAIC that's the end of it, if there is some subtle technical issue lurking here then it does not show up in the real world and I'm not really interested.

[Panopeeper]

I stand by my assertion that converting to DNG makes no difference; but evidently only using exactly the same software for both DNG and CR2. Therefore it's not the DNG format that is the 'culprit', it's the software that processes it.

This is not, what you have proven. You have shown, that ACR interprets the native raw file identically as the DNG file. Big deal.

What about trying this with other raw converters? Some of them process (or processed in the past) the raw file "normally", i.e. as the raw processor's designers/programmers decided, while processing the DNG file as Adobe decided. This is relevant regarding the color reproduction.

However, again, in my eyes this is not an argument for or against DNG. This is the question of interpretation: if you select the path of DNG created by Adobe's DNG converter, you decide for Adobe's interpretation, which is the same as if you convert the native raw file directly with ACR. If you process the raw files (native and DNG) with another raw converter, the results may or may not be identical.

I remember there was a time, when Aperture processed DNG files, but only those created from natively supported raw files; this indicated, thet Aperture supported the format but did not give a damn for Adobe's interpretation of that raw data.

[Nick Rains]

This is not, what you have proven. You have shown, that ACR interprets the native raw file identically as the DNG file. Big deal.

What about trying this with other raw converters? Some of them process (or processed in the past) the raw file "normally", i.e. as the raw processor's designers/programmers decided, while processing the DNG file as Adobe decided. This is relevant regarding the color reproduction.

However, again, in my eyes this is not an argument for or against DNG. This is the question of interpretation: if you select the path of DNG created by Adobe's DNG converter, you decide for Adobe's interpretation, which is the same as if you convert the native raw file directly with ACR. If you process the raw files (native and DNG) with another raw converter, the results may or may not be identical.
.

I have demonstrated what others have been disputing, that DNGs are not equivalent to the original RAW. They are, as expected.

As to processing in different RAW converters, well of course the results are going to be different. What's wrong with that? It's quite common to use different converters to process different files, some have strengths in different areas - that's why people like to keep their original RAWs, just in case they need to use, say, DPP for some subtle edit. In this case DNG is not the appropriate format, I've no problem with that.

I happen to use ACR almost exclusively- it's convenient to do so - and so DNG is an appropriate format to archive into - for me.

[Drazick]

papa, this is a common misconception. Most of the makers' "costly device characterisation" is stored in maker notes. Whether a maker note is stored in a CR2 container, a NEF container, a DNG container, or a TIFF/EP container does not matter; the raw conversion software can read the same (proprietary) information regardless of the container. In fact, when the Adobe DNG Converter creates a DNG from a non-DNG raw file, it copies over the maker note, even though it doesn't most of the contents of that note. Cameras today that shoot DNG files directly also store characterization data in the DNG maker note; only the makers of those cameras understanding the meaning of those notes. Those makers have "given up" nothing.

Different makers will take different approaches to a common raw format.

Example 1: Some makers like Ricoh, Leica, and Casio are opting for the DNG-only route, i.e., their cameras do not offer a non-DNG raw mode. They are fine with putting as-shot white balance (i.e., maker neutral), embedded color profiles, etc. in public tags. Whatever they feel uncomfortable with sharing goes into the DNG maker note.

Example 2: Other makers like Pentax are opting for two raw modes, one that is DNG, one that is non-DNG (for Pentax, that would be PEF).

Eric,
The question is whether DNG allows the companies hide their proprietary color LUT and yet allow the user get the same colors using any DNG RAW converter.

At the end, that's what we want, the camera colors at any RAW converter.

[madmanchan]

The question is whether DNG allows the companies hide their proprietary color LUT and yet allow the user get the same colors using any DNG RAW converter.

At the end, that's what we want, the camera colors at any RAW converter.

That depends on what you mean by "hide" and "proprietary" in this context.

Camera vendors can certainly store color profiles (with matrices and complex LUTs, curves, etc.) in DNG images.  However, as the format of these color profiles is described in the DNG spec, and both the spec and SDK are free to download and use, a developer/engineer could easily "peek" inside the file and grab the profile contents to see what's going on.

On the other hand, Adobe has provided so-called Camera Matching profiles for several years for several cameras.  These profiles are stored in the same format and also produce "camera colors".  Developers/engineers can poke inside these profiles in exactly the same way.

So, from my point of view, none of this stuff is particularly hidden to begin with.

[JRSmit]

Eric,
The question is whether DNG allows the companies hide their proprietary color LUT and yet allow the user get the same colors using any DNG RAW converter.

At the end, that's what we want, the camera colors at any RAW converter.

Not really. Actually really not. I want my colors and not those of  nikon canon or whatever make.
When the endresult is there the  camera used is of no importance.

[Drazick]

Eric,
You get very close to the manufactures color.
Yet not in every situation and set of parameters.

Let's say I'm Nikon.
I want to give my customers, owners of Nikon cameras, the option to have the same colors they get using Capture NX in ACR / Lightroom.
Yet I don't want the matrices / LUT's to be exposed.
Just be hidden as they are hidden in my own NEF files.

Does DNG have that capability?
As far as I know, it doesn't, that's the main barrier...

[Vladimirovich]

Just be hidden as they are hidden in my own NEF files.

Nikon software creates .icc profiles during its work and those profiles are being extracted during that process by people to be used by their own purposes... the only way to have undocumented info in a documented file format is to store all that in a special tag intended for storing undocumented manufacturer's info and that will be a major PR blow to DNG concept if Adobe starts to use that info itself... alternatively Adobe might update a DNG spec allowing encryption of information in documented tags (so that you can have encrypted content in documented fields) and then Nikon can give certain software developers keys to decrypt, but as history teaches that is a futile effort... the keys will be stolen

[Vladimirovich]

As far as I know, it doesn't, that's the main barrier...

the main barrier to what ? it is possible to imitate color rendering quite close w/o asking Nikon... a good software developer (certainly not that every single one or every other) can create a synthetic raw file (or several) to cover with all possible (or reasonably necessary) combinations of R G1 B G2 values, convert it using Nikon software, get resulting tiff, then create a .dcp profile with LUTs imitating the same conversion (might be a huge one though) from that raw to Adobe's internal working color space (certainly adjusted to the fact that Nikon is not generating tiffs in the same colorspace as Adobe uses internally and that Adobe deals with demosaicked RGB data, not RGBG raw values for their color transforms)... something like this... "VitNovak" @ http://forums.adobe.com/community/cameraraw did something like this for his Canon's - search for his postings.

[Tim Lookingbill]

Let's say I'm Nikon.
I want to give my customers, owners of Nikon cameras, the option to have the same colors they get using Capture NX in ACR / Lightroom.
Yet I don't want the matrices / LUT's to be exposed.
Just be hidden as they are hidden in my own NEF files.

Does DNG have that capability?
As far as I know, it doesn't, that's the main barrier...

As Eric mentioned in post #14...

...in my opinion the real benefit of DNG (or at least some standard, documented raw format, whether it be DNG, TIFF/EP, or something else) is when they are generated directly by the camera.

In the case of DNG, for instance, an in-camera DNG can contain an image checksum, which DNG readers can use to check for image integrity after images have been read off the card (or at any other time). As another example, the makers can embed 1 or more color profiles to be used for portable default color rendering across DNG readers; Casio and Pentax have done this quite effectively, without having to divulge what recipes they used to cook up those profiles in their labs.

Eric,
You get very close to the manufactures color.
Yet not in every situation and set of parameters.

That's because you have two preview renditions to compare against with each reflecting how their rendering recipe deals with the sensor's non-linear behavior to any given scene captured. You're asking too much of both the device and software with regard to consistency between the two.

If you had only the Nikon software generated preview to assess color consistency shot to shot then you wouldn't notice the inconsistencies.