DNG puzzle

[Frans Waterlander]

Indeed there are provisions for private metadata, good call.
But keep in my some FUD that’s OT is from someone who this week alone didn’t know the difference between a bug and a feature in ACR found in Adobe Elements. That post had to be shut down too.

And that from someone who this week alone didn't know the difference between using a database and using sidecars.

[digitaldog]

Yes, there is a sense of "crying wolf" a bit too often.

As for the original question, I think it's worth saying that DNG sometimes produces a smaller file size, sometimes not, and it is mostly dependent on the efficiency of the camera's compression. Some people  see this as a reason for adopting DNG, but it's better to see it as only a by-product (and not pay much attention to it).

And sometimes larger depending on what's stuffed into the container. My ARWs from a Sony RX10 are about 21MB each but with fast load data and a rendered JPEG, my DNGs of the same camera are about 26MB. And that's fine. My Canon 5DMII is the opposite however. So I suspect we need to hear back from Peter about what settings he might be using for the conversions and what the differences might be if Fast Load Previews are used; do you know how big they are? I suspect the preferences set for previews plays a role. My JPEG Previews in LR are set for Large as well.

[digitaldog]

My posts have contributed nothing?

Just remove the question mark.

[DP]

I have never used DNG files before today when I had the occasion to export some from LR. The original files were Sony RAWs which tend to be in the 80 MB range. Yet the exported DNGs were all 37-45 MB. It's my understanding that the DNG format preserves all the info in the RAW, so how can I get a 50% file size reduction? Thx.

here is what Eric Chan of Adobe (Camera Raw team) said and that is the ultimate truth:

Eric Chan :: http://forums.adobe.com/message/1210133#1210133
"...for archival purposes I recommend (and practice) safely storing the original raw files in the form that they came off the camera , whether they be DNG or non-DNG..."

"...To be clear, Adobe did __not__ create DNG in the hopes that photographers shooting non-DNG raw files would suddenly convert them all to DNG files and then throw away their original non-DNG raw files. Instead, Adobe created DNG has an example of a documented format (a set of TIFF extensions) that would improve interoperability among hardware and software vendors, as well as be suitable for archiving..."

[DP]

Nope, it's preserved in an unparsed metadata block.

we have a lot of examples when then current version of DNG Converter (or the same code in ACR/LR as they too can convert) discarded data during conversion, etc... Adobe makes enough errors in their code not to discard the original raw files (see above from Eric Chan).

[Schewe]

Be aware that DNG discards proprietary camera information.

Not true...ANY metadata that DNG does not recognize is stored in the file and is not discarded. This is a typical misunderstanding perpetrated by people who don't understand what DNG is and why it was done. DNG is simply a container format to teach the camera makers how to make robust raw files. It was basically done because Thomas Knoll got tired of trying to teach the camera makers how to store raw data.

BTW, the only reason Adobe ended up releasing DNG as a proposed standardized raw file format is because somebody needed to do something to reduce the Tower of Babel that the camera makers were making...

As for DNG files being smaller, even without using the lossy DNG option, the raw files will often be smaller as DNGs because, well, the camera makers are bad at making raw file formats and don't know how to maximize lossless compression. So, many (if not most) raw files will get smaller as DNGs.

[Jeremy Roussak]

BTW, the only reason Adobe ended up releasing DNG as a proposed standardized raw file format is because somebody needed to do something to reduce the Tower of Babel that the camera makers were making...

And their intentions were good, no doubt, but probably doomed.



Jeremy

[Schewe]

And their intentions were good, no doubt, but probably doomed.

Actually no...the results have been much better. Since DNG was released, NEF, CR2 (Nikon & Canon raw formats) are much better at doing the main job of acting as relabel containers. The other major manufactures likewise improved their formats and some manufactures actually adopted DNG as their camera format like Leica.

The main difference between the proprietary formats and DNG are that they are undocumented...the camera makers think they should keep their formats proprietary rather than fully document the format and metadata.

It's silly really because pretty much everything can be deduced with revers engineering. Unless the undocumented metadata is encrypted (remember when Nikon encrypted the white point metadata and Thomas refused to decode the encryption and therefor ACR wouldn't use the file's whitepoint metadata to correct).

The funny thing is that the first thing ACR and Lightroom do with proprietary raw files when it sees them is to convert to DNG anyway (even if you don't convert the raw and save as DNG) because that's the only way the proprietary files can be processed. In point of fact, a little birdie once told me that the major raw file formats are so close to being fully formed DNG containers that the cameras could prolly be converted to write to DNG file format as the camera file with a simple camera firmware update.

Personally, I don't use DNG because I like the tiny size of the side car file compared to the massive size of a DNG. So when I backup a changed file the backup only has to write the tiny .xmp file instead of the whole file that has been changed. I do use DNG as a handy delivery container for those rare times I must deliver a raw file format. That way the metadata settings are baked in the file and can't be lost if separated from the .xmp file.

[Bart_van_der_Wolf]

Not true...ANY metadata that DNG does not recognize is stored in the file and is not discarded. This is a typical misunderstanding perpetrated by people who don't understand what DNG is and why it was done. DNG is simply a container format to teach the camera makers how to make robust raw files. It was basically done because Thomas Knoll got tired of trying to teach the camera makers how to store raw data.

BTW, the only reason Adobe ended up releasing DNG as a proposed standardized raw file format is because somebody needed to do something to reduce the Tower of Babel that the camera makers were making...

As for DNG files being smaller, even without using the lossy DNG option, the raw files will often be smaller as DNGs because, well, the camera makers are bad at making raw file formats and don't know how to maximize lossless compression. So, many (if not most) raw files will get smaller as DNGs.

While generally true, camera makers face other priorities, e.g. storing 5-10 raw images per second.
I don't see how that in any way compares to a separate compression step that can spend more time on it, while using much more powerful hardware.

In fact, even if one were to use lossy / irreversible compression, there is still a lot of size that can be saved without perceptual losses. A fine example is Google's Guetzli JPEG compression, which shrinks JPEGs to really small sizes with hardly noticeable visual losses. But it's very slow, and can take minutes per image, depending on imagesize and computer horsepower. But as a demonstration, and for Website resources, a great invention (https://github.com/google/guetzli).

Cheers,
Bart

[RSL]

I’m with Jeff. I convert everything I’m gonna save to DNG. The only place we disagree is on .xmp files. I understand what he’s saying, and if I had the kind of volume Jeff probably works with I’d probably do the same thing. But I hate those little .xmps hanging around, so I convert to DNG. (I also keep the originals on DVDs in many cases).

And I hear what Bart’s saying. My only problems with it are these:

If you’re going to shoot movies you’re better off doing it with a movie camera. It really bugs me that my still cameras now have to be overloaded with movie bells and whistles I’ll never use.

My other beef is with “speed.” Why does anybody need to store 5 to 10 raw images a second with a still camera? That’s not what “still” means.

But, of course, people tell me all this is “progress.”

[PeterAit]

Whoo! As the OP, I feel like someone who has tossed a filleted pig into a piranha pool.

[digitaldog]

Whoo! As the OP, I feel like someone who has tossed a filleted pig into a piranha pool.

You shouldn’t. You didn't do anything wrong but ask a legitimate question. Hopefully you're satisfied with the answers about file size. As for piranha:

[Jeremy Roussak]

Personally, I don't use DNG because I like the tiny size of the side car file compared to the massive size of a DNG. So when I backup a changed file the backup only has to write the tiny .xmp file instead of the whole file that has been changed.

Exactly my reasons, too.

Jeremy

[digitaldog]

Exactly my reasons, too.

Jeremy

It's a very legitimate potential concern. That said, all my backup's take place in the middle of the night, every night, unattended (including backup to the cloud/CrashPlan) so it really has no effect on me personally. It would be kind of cool if there were a way not to force an update and backup if you did something as tiny as say embed a new keyword. But thus far, the backup hasn't been a concern but I can understand for those working with hundred of megs a day or more, it could be.
On the other hand, having my DNG's with my edits, a preview big enough to print from, my camera profiles and such all embedded in one file allows me to keep all that data in one place and easy to access; within the DNG.
And no sidecar's for 'rendered' images like JPEG, TIFF, PSD, the same issues for backup, even with a tiny edit somewhere in the document still applies.

[DP]

Not true...

true - Iliah Borg provided a lot of examples (due to constant errors in Adobe's code)

[Schewe]

true - Iliah Borg provided a lot of examples (due to constant errors in Adobe's code)

Iliah found some errors due to Adobe code and some errors because of the faulty method of encoding...and where found, Adobe fixed the errors.

[Telecaster]

I've never converted proprietary Raws to DNG, other than early on out of curiosity, but I do use two camera systems that write DNGs natively: Leica and Pentax (DNG is an option with the latter).

-Dave-

[Ivophoto]

And their intentions were good, no doubt, but probably doomed.



Jeremy

My father claimed to know a lot about humanism.
I asked him if he could explain what humanism exactly was.

I saw on his face he was uncomfortable with the question and he replied:
“Well, that depends what kind of humanism you mean. “

[RSL]

And their intentions were good, no doubt, but probably doomed.



Jeremy

The only problem with this cartoon is that the "standards" aren't "competitive." They're not even standards. How many times have you, as a Nikon shooter, decided to try Olympus's .ORF instead of Nikon's NEF? .ORF is only a "standard" for Olympus, and .NEF is only a standard for Nikon.

The only "competitive" standard is DNG.

[Jeremy Roussak]

The only problem with this cartoon is that the "standards" aren't "competitive." They're not even standards. How many times have you, as a Nikon shooter, decided to try Olympus's .ORF instead of Nikon's NEF? .ORF is only a "standard" for Olympus, and .NEF is only a standard for Nikon.

The only "competitive" standard is DNG.

True, perhaps. But it's funny anyway.

Jeremy