Lightroom: DNG or RAW

[Nick Rains]

Have you actually read the post I linked to? You asked for evidence, and I provided one; note, that this is not the only case when the DNG converter created an erroneous output.

Of course I did. Please don't be so quick to assume things...what part of my post led you to think that? I made no reference to it at all, just to your 'tone'.

Anyway, moving on.. Yes, that was in interesting link, thanks, I will consider it further.

[papa v2.0]

Hi just read the DNG spec and have a  question

Does Adobe purport that camera manufacturers replace the native RAW file with the DNG spec to give say, JPEG and DNG instead of currently JPEG and native RAW?

While  I agree a common file spec would be useful I cannot see the camera manufacturer giving up their costly device characterisation data that easily.

[madmanchan]

papa, this is a common misconception. Most of the makers' "costly device characterisation" is stored in maker notes. Whether a maker note is stored in a CR2 container, a NEF container, a DNG container, or a TIFF/EP container does not matter; the raw conversion software can read the same (proprietary) information regardless of the container. In fact, when the Adobe DNG Converter creates a DNG from a non-DNG raw file, it copies over the maker note, even though it doesn't most of the contents of that note. Cameras today that shoot DNG files directly also store characterization data in the DNG maker note; only the makers of those cameras understanding the meaning of those notes. Those makers have "given up" nothing.

Different makers will take different approaches to a common raw format.

Example 1: Some makers like Ricoh, Leica, and Casio are opting for the DNG-only route, i.e., their cameras do not offer a non-DNG raw mode. They are fine with putting as-shot white balance (i.e., maker neutral), embedded color profiles, etc. in public tags. Whatever they feel uncomfortable with sharing goes into the DNG maker note.

Example 2: Other makers like Pentax are opting for two raw modes, one that is DNG, one that is non-DNG (for Pentax, that would be PEF).

[papa v2.0]

have I got this right?  In order to process a RAW file be it a DNG or native RAW file, the device RGB to CIEXYZ matrix or transforms need to be used.
The .NEF files for example, do not contain the Device RGB to CIEXYZ transforms, these are in the accompanying proprietary software.
For eg .ACR has to use its own 'home made' camera characterizations as these are not available in the 'maker notes', not that I could find at least.

I you could point me to where this information is I would be grateful.

[Panopeeper]

have I got this right?  In order to process a RAW file be it a DNG or native RAW file, the device RGB to CIEXYZ matrix or transforms need to be used

Not right. The transformation between the camera's color space and the target (sRGB or Adobe RGB) does not need to be done by matrix operations. Adobe decided for this way in order to make the transformation more formalistic. Other raw processors, typically the manufacturers' own products do not use this method. The other method is direct transformation (often but incorrectly called LUT).

The inferiority of ACR's color reproduction was due primarily to this method. Now, with the additional profiling option (DNG Profile Editor) this is better, but it is still questionable if this method can ever yield equally good result as the more direct transformation.

Note, that the transformation depends on the illumination; there are different matrixes for different illumination.

[papa v2.0]

I thought that the Device  RGB ( or coordinates) had to be transformed to CIEXYZ BEFORE any further colour space transformations take place.
Is this not usually done using a 3x3 matrix?

[madmanchan]

Not right. The transformation between the camera's color space and the target (sRGB or Adobe RGB) does not need to be done by matrix operations. Adobe decided for this way in order to make the transformation more formalistic. Other raw processors, typically the manufacturers' own products do not use this method. The other method is direct transformation (often but incorrectly called LUT).

The inferiority of ACR's color reproduction was due primarily to this method. Now, with the additional profiling option (DNG Profile Editor) this is better, but it is still questionable if this method can ever yield equally good result as the more direct transformation.

Note, that the transformation depends on the illumination; there are different matrixes for different illumination.

I don't quite agree ...

In most cases, the makers' default color transforms are actually using a matrix as well, i.e., in addition to lookup tables. The matrix used for initial rotation and scaling to a canonical space (usually XYZ with a D50 white point), as well as to account for per-device characterization. The LUT is mostly used to build in color preference.

I do not see why you question whether a DNG profile can achieve quality equivalent to your so-called "direct transformation." A DNG profile can perform a matrix and a LUT operation and hence can effectively perform an arbitrary 3D to 3D color transformation.

[madmanchan]

I thought that the Device  RGB ( or coordinates) had to be transformed to CIEXYZ BEFORE any further colour space transformations take place.
Is this not usually done using a 3x3 matrix?

papa, if the goal is to get fairly accurate scene colorimetry reproduction, then a 3x3 matrix to CIE XYZ space (or other standard space like RIMM) from device coordinates is generally sufficient. This is a reasonable first step before performing additional color operations, i.e., get the image into a color space with a well-understood set of primaries.

However, this is not a requirement. An image processing program can actually perform RGB color operations while in the native RGB linear camera system (or some derived space, e.g., after applying some gamma encoding). On the other hand, exposing editing operations to the user that manipulate the image in such a space is usually not a good idea, because it means that the controls will behave differently on different cameras (e.g., a RGB tone curve would behave very differently on a Nikon D70 versus a Nikon D3).

[bjanes]

Yes. The reasons are stated here:
http://www.ppmag.com/reviews/200709_adobedng.pdf

As Gabor has pointed out, the DigitalDog's referenced assay is concerned mainly with the advantages of parametric editing, which can be done with raw files as well as DNGs. The advantage of DNG is that the metadata for the edits are stored directly in the DNG rather than in a separate XMP file or database as is the case when one is editing raw files with ACR. In the latter instance, the metadata can become separated from the raw file and the edits are then lost.

With my Nikon D3 using 14 bit losslessly compressed NEFs, converting to DNG results in a slightly smaller file, unless the option of embedding the raw file in the DNG is selected, in which case the resulting DNG is approximately double the size of the NEF. If one has only a DNG without the embedded raw file, the ability to edit the file with Nikon Capture NX or other third party software that does not support DNG is lost. Currently, ACR is my preferred editor, but there are advantages to Nikon Capture NX. NX can automatically correct for chromatic aberration with Nikkor and third party lenses and for light falloff with lenses that have a CPU. With ACR, one must do these corrections manually by entering the appropriate parameters, and this can be cumbersome with zoom lenses where the parameters differ with focal length and aperture. In the future, it is likely that additional lens defects will be corrected in software and it could be important not to lose this ability. If one chooses to take advantages of DNG, I think that it is important to save or embed the original raw file.

Parenthetically, I note that Andrew still thinks that the raw file is grayscale and that color is magically created by the raw converter. This feat is only slightly less astounding than the conversion of water to wine at the wedding at Cana.  

Bill

[digitaldog]

As Gabor has pointed out, the DigitalDog's referenced assay is concerned mainly with the advantages of parametric editing, which can be done with raw files as well as DNGs. The advantage of DNG is that the metadata for the edits are stored directly in the DNG rather than in a separate XMP file or database as is the case when one is editing raw files with ACR.

I guess the last paragraph in column one discussing sidecar files wasn't obvious enough for some readers!

[bjanes]

I guess the last paragraph in column one discussing sidecar files wasn't obvious enough for some readers!

Perhaps so, or perhaps the title of the article was ill-chosen.

Bill

[digitaldog]

Perhaps so, or perhaps the title of the article was ill-chosen.

You are entitled to that opinion, that said, I find many compelling features of DNG and find it a powerful option given the alternatives.

I'll tell the magazine you might be interested in becoming an editor....

[Daniel Browning]

Can you point to any evidence that shows that the output from a DNG file is in any way different to that from it's original RAW when both are processed in the same way with the same RAW converter?

http://forums.dpreview.com/forums/read.asp...essage=30275878

[Tklimek]

Hmm....that is an interesting post.

However, Borg states that the original RAW process (NX for example) should be archived with the original raw files otherwise a similar situation would develop.  Sounds kinda hokey to me.  I think one might be able to say...no problem...archive your DNG with whatever version of ACR is being used at the time (unless I misunderstood what he said).

I think the gist of that thread on "that other forum" is that a RAW file or a DNG contains your BASE information, whatever RAW processor you use (whether camera manufacturer or 3rd party such as Adobe) will in fact change over time as improvements are made.

Does that sound correct?

Cheers...

Todd in Chicago

http://forums.dpreview.com/forums/read.asp...essage=30275878

[DarkPenguin]

Yeah, but it's bullshit.  It was fine in ACR when you could easily toss a copy of ACR next to your files and get back to the last rendering.  In the case of lightroom you have a choice of having ALL of your files change whenever you change LR (to a version that changes the rendering) or exporting rendered copies of all your files.  This kind of defeats the purpose of metadata editing.  Heck if you're using ACR doesn't it defeat the purpose of smart objects, too?

[Tklimek]

LOL.....hehehehe.... a very slippery slope indeed!

Eric Chan indicated in that other thread that if you wanted to archive/preserve an image with the enhancements so that it DID not change over time; archive a TIFF version with the parametrics baked in.

:-)

Cheers...

Todd in Chicago

Yeah, but it's bullshit.  It was fine in ACR when you could easily toss a copy of ACR next to your files and get back to the last rendering.  In the case of lightroom you have a choice of having ALL of your files change whenever you change LR (to a version that changes the rendering) or exporting rendered copies of all your files.  This kind of defeats the purpose of metadata editing.  Heck if you're using ACR doesn't it defeat the purpose of smart objects, too?

[Panopeeper]

Can you point to any evidence that shows that the output from a DNG file is in any way different to that from it's original RAW when both are processed in the same way with the same RAW converter?

http://forums.dpreview.com/forums/read.asp...essage=30275878

The original question in that form was not really relevant, i.e. this is not the real issue; nor is that post on DPreview relevant regarding the main issue.

DNG is (supposed to be) partly a format change of the original data. Although the encoding adopted by DNG in the compression is the same one adopted in CR2 files, there are small differences. The differences are greater with Nikon compressed image data; spme other formats are totally different.

The only relevant questions are,

1. if the image data remains unchanged or directly (i.e. one to one) convertible between the native format and DNG,

2. if all relevant info is converted or saved in DNG format.

Condition 1 is fulfilled mostly but not always. The exceptions are worthy dio be discussed on their own;

condition 2 is fulfilled by savng MakerNote in the DNG file.

Note, that I discounted the option of saving the entire original raw data in the DNG file, for that is plain garbage.

Back to the subject: it is NOT important, that the same result be generated from the different formats. DIfferent versions of any raw converter may yield different results from the same raw file, no matter which format.

It is important to understand, that the DNG file contains not only the "natural data" (i.e. that, what is taken from the native raw file) but its interpretation is added. Particularly, the color reproduction is an issue, which is entirely in the domain of the raw processor and can be changed. Thus there is nothing strange on the fact, that different versions of ACR or of the DNG converter generate different outputs.

Again, the main issue is, if the raw image data remains the same except for formal changes. The example I provided demonstrate, that this is not always the case. That error has been removed, but all related images converted from NEF in DNG format in that period are permanently lost if the NEF was not kept.

I repeate it: only computer illiterates would believe, that any software, no matter from which corner it comes, is guaranteed error free and one can blindly trust to throw away the original data.

[madmanchan]

I also recommending archiving the original raws as they came off the camera (whether these are camera-generated DNGs or non-DNGs) and have no issue with Gabor's view on this matter.

Regarding the dpreview thread linked to by Daniel, my take with Iliah's concern is that rendering changes are an issue in any parametric editing environment and is independent of DNG (and I stated this opinion in that thread). For example, suppose you process a CR2 file in your preferred raw conversion software today and adjust its controls the way you like. Then next year a new version of the software comes out with an improved color interpolation algorithm that reduces speckle artifacts. Presumably this is a Good Thing, in that if the algorithm is good, then your images simply look better. But if you go to process that CR2 again using the new version of the software, then even though you didn't touch any controls, the appearance of that CR2 has changed since you last looked at it. Same thing applies if, for instance, the raw converter "upgrades" to a 32-bit or 64-bit-per-component floating point pipeline (up from, say, 16 bits per component), well the internal math will now be done at higher precision so that results will be numerically different than what you got previously. Iliah suggests archiving the software version with the image. Well, ok, if you want to go that far, but keep in mind you'd have to archive the computer as well, and required associated hardware, since computers of the future may not run the old software. (Try running Kodak's raw converters for the DCS 460 on OS X or Vista ...)

[digitaldog]

For example, suppose you process a CR2 file in your preferred raw conversion software today and adjust its controls the way you like. Then next year a new version of the software comes out with an improved color interpolation algorithm that reduces speckle artifacts. Presumably this is a Good Thing...

Its a great thing! Its what makes Raw so damn useful. We didn't have that with film (keeping latent image around wasn't a good idea). Its not going to happening much if at all with rendered images. I'm all for reprocessing a hero image if indeed, it will visibly  benefit the final output.

Well, ok, if you want to go that far, but keep in mind you'd have to archive the computer as well, and required associated hardware, since computers of the future may not run the old software. (Try running Kodak's raw converters for the DCS 460 on OS X or Vista ...)

That's been my experience and its not a good one. That said, being on the Mac since 1988, I've got lots of files I can no longer access (proprietary calender programs). My own fault for not updating as I go. But that data is far, far less important than image data. Its pretty easy to export a text doc from those older calendar applications and re-import into a newer product. Not so easy with a proprietary Raw file!

I have one really old Mac laptop that can boot OS9, I'm thinking I might just archive it.

[DarkPenguin]

Doesn't matter if the rendering is improved.  All the work was based on a different rendering so the image might require manual intervention to actually be improved.  In the case of something like a smart object that could be a lot of layers and a lot of work.

Now that I understand the lie of parametric editing I'm fine with this.