Michael's DNG comment

[sandymc]

This seems to be causing some confusion. On backups:

1. To backup the original image (aka the "negative") only, you only need to backup once, preferably as soon as you import the image from SD card/CF card.

2. To backup the image and the adjustments you've made (assuming LR is set up to save to the DNG), you need to backup every time you adjust the image.

Sandy

[john beardsworth]

Not really.

You need to backup the DNG when it is first created, and the catalogue on a daily or other schedule. Doing this alone is sufficient to allow 100% recovery of your images and all the work you've done on them.

If you really want to backup the adjusted DNG, then pt 2 applies - but these redundant backups will often not include all your adjustments (virtual copies, proof copies). 2 is entirely optional - in my view wasteful.

[32BT]

Could it be that some of the questions and/or confusion in this thread come from the difference between these options:

1. Use LR as the backup mediator

2. Use the OS as the backup mediator

The original question doesn't seem to specifically mention which is used. If TimeMachine on OS X is used, for example, then the questions and confusion make perfect sense to me, since it will search for "edited" files and back those up completely. And incremental backups and states as supported on the latest mac os need to be supported specifically by applications. (I have no idea whether LR makes use of this).

[john beardsworth]

I'd say it's more the result of perfectly-natural assumptions.

[Fips]

1. To backup the original image (aka the "negative") only, you only need to backup once, preferably as soon as you import the image from SD card/CF card.

2. To backup the image and the adjustments you've made (assuming LR is set up to save to the DNG), you need to backup every time you adjust the image.

For my workflow, option 1 has a disadvantage. While I only need to backup once during the import, it will cost me a lot of disk space as I also backup all the junk which is deleted later in the editing process. This is particularly of concern as some day in the future I might want to backup to some cloud where space is limited and bandwidth costly.

The perfect solution - if such thing exists - would be an incremental, blockwise backup. So even when a file, say a DNG, is changed, only the blocks on the disk which have changed are updated in the backup instead of the while file. That would combine the advantage of having all the metadata in one file while backups are as fast as with separate XMP sidecars.

[digitaldog]

For my workflow, option 1 has a disadvantage. While I only need to backup once during the import, it will cost me a lot of disk space as I also backup all the junk which is deleted later in the editing process.

Yes, good point. I don’t want to back up 100 images only to end up deleting 30. I’d rather backup after I convert DNG on import which doesn’t appear to be that much more time consuming and embeds a lot of useful data at this import stage I apply. Delete the images I don’t want, do some work and then backup. I am backing up to a cloud (CrashPlan) as well as a number of external drives and it is the cloud backup that is the biggest concern for me in this discussion of what and when to backup. The main drive that is dedicated to images and all LR files is a mirrored array. It has saved my butt a few times. After working on a decent number of images, I’ll plug in one of my rotating external drives, clone to that and move that do a machine that then backs up to the cloud. Much of this is automated (backup to external drive done with a schedule that runs automatically in the middle of the night).

I do wish we could have our cake and eat it too. That is, backup just small additions of XMP data embedded in a really big DNG.

Another ‘issue’ in the backup straight away is that I will often use different DNG profiles from a single import session. I like that these profiles are embedded in the DNG. I like that there is a high quality JPEG of the current rendering inside the DNG. I’d really like all that data to be backed up. If I were to import and immediately backup the DNGs, as you point out, not only do I back up documents that will get trashed, I backup the DNG profile used at import and I’ll end up switching that. I’ll do some rough global work on images and I’d like that rendering applied to the embedded JPEG ‘just in case’. If I move the cloned drive to another system, it doesn’t matter if a DNG profile is missing on that machine, it travels with the DNG. So the backup at start of workflow while saving some time is a tad worrisome for a guy like me that likes to wear a belt and suspenders in terms of backing up data.

[sandymc]

If you really want to backup the adjusted DNG, then pt 2 applies - but these redundant backups will often not include all your adjustments (virtual copies, proof copies). 2 is entirely optional - in my view wasteful.

Well, depends on whether you view virtual copies, etc as adjustments. Point 2 was really about adjustments to the original image. The DNG advantage of including adjustments into the file itself doesn't, of course, apply to virtual items, etc, only the original. If you use such things extensively, then the "adjustments in the original file" part of the DNG value proposition isn't really relevant to you anyway.

Sandy

[digitaldog]

The DNG advantage of including adjustments into the file itself doesn't, of course, apply to virtual items, etc, only the original.

Which is why if I find I have a VC that is a hero image, I export it as a DNG.

[john beardsworth]

Well, depends on whether you view virtual copies, etc as adjustments.

Definitions can get tiresome. Whether VCs are counted as adjustments or not (OK, then they are collections of adjustments), they're work and time you've invested. Even if you do not use VCs, and you don't use soft proofing copies, there's still a big chunk of most people's Lightroom work that never gets written to xmp and undermines its backup value. Writing xmp is principally designed for data exchange with other apps, not for backup.

[Les Sparks]

The discussion of DNG workflow, backups, etc. shows why it seems unlikely that the industry will adopt DNG as a standard raw format. All this discussion is basically related to use of Adobe software. DNG is an Adobe centric standard designed to provide raw files that work great with Adobe software. It is not designed to meet the specific needs or desires of camera makers or makers of non Adobe image editing software. An ISO type standard would have to be designed to meets the needs and desires of camera makers, image editing software makers, photographers, archivists, computer and display makers, operating system makers, etc. It is the need to gain approval of all these diverse groups that makes creating an approved standard so difficult.
Although Adobe has published the DNG standard, we need to remember that DNG is still basically an Adobe  proprietary format. Adobe can, and has, made changes to the standard has made enhancements and basically has an advantage in writing software that takes advantage of the enhancements. The history of the programing language JAVA provides a cautionary tale for anyone considering adopting DNG. Sun developed Java and made it an open standard. Microsoft extended Java to work better with Windows and was successfully sued by Sun. Sun was then sold to Oracle who then sued Google about Google's use of Java. Not saying that this will happen with DNG, but it could. Someone could purchase Adobe and see possibility of getting an economic benefit from DNG.
I agree that a standard raw format would be great and that DNG is a good starting point. But I think there are significant barriers, not all of them of the not invented here type, that have to be overcome. Any standard that does  come out of say ISO is unlikely to be pure DNG.

[Keith Reeder]

DNG is an Adobe centric standard designed to provide raw files that work great with Adobe software.

That's not the case, Les - it is incredibly easy to have a complete, and completely effective, DNG-based workflow from image ingestion to finished image/print without there being a single piece of Adobe software on your machine.

That's actually one of DNG's benefits - the flexibility it delivers and the lack of proprietary software tie-down. Quite a few Linux folk I know have DNG workflows, and they can't/won't use Adobe software.

[digitaldog]

DNG is an Adobe centric standard designed to provide raw files that work great with Adobe software.

Yes it does work great and nothing stops any other manufacturer from utilizing exactly the same file format the same way with their raw converters.

[Les Sparks]

I wasn't saying that you had to use Adobe software to use DNG. I was trying to point out that there are valid reasons why the industry might not be willing to adopt DNG as a standard. The theme here seems to be, mostly, that the Nikon and Canon don't adopt DNG because they're big bad companies. I am trying to show that there may be good reasons for them not to adopt DNG.
As far as anyone being able to adopt the format, that's OK, but as things stand now, anyone that adopts DNG is going to be playing catchup when Adobe changes the standard. For example, how many non-Adobe programs can open and use the latest version of DNG? I've seen posts here complaining that non-Adobe software can't deal with latest DNGs.
Personally, I think that a standard raw format would be nice, but not essential. I would much rather see effort put into improved lens corrections and further improvements in noise reduction and the overall raw engine than in trying to get a raw standard through a standard setting group.

[Schewe]

Personally, I think that a standard raw format would be nice, but not essential. I would much rather see effort put into improved lens corrections and further improvements in noise reduction and the overall raw engine than in trying to get a raw standard through a standard setting group.

And your attitude is part of the overall problem. You don't see proprietary, undocumented raw files as being much of a problem. However, digital conservationists do.

[digitaldog]

I was trying to point out that there are valid reasons why the industry might not be willing to adopt DNG as a standard. The theme here seems to be, mostly, that the Nikon and Canon don't adopt DNG because they're big bad companies. I am trying to show that there may be good reasons for them not to adopt DNG.

No, they are not bad companies, they have some really bad policies! Policies that are political and not in our best interest. And no, as yet I’ve yet to see the valid reasons for not adopting DNG.

As far as anyone being able to adopt the format, that's OK, but as things stand now, anyone that adopts DNG is going to be playing catchup when Adobe changes the standard.

That’s just slightly ridiculous. Do you realize that TIFF has evolved over the years and that any company who’s product supports TIFF is in the same position and can easily update their software to take advantage of the new features? Do you suppose any company worth a dine that supports TIFF will not do this (assuming the new functionally, like support for layers is useful to their customers)? The argument of playing catch up doesn’t wash. What makes you think Adobe would not be providing information to these companies? Where have the pain points been over the years as DNG (as well as TIFF and other formats) been observed? DNG like TIFF and PSD among other formats has evolved and hopefully will continue to evolve. The same is true of files like ICC profiles. It sounds like your argument is to keep these technologies stagnant.

For example, how many non-Adobe programs can open and use the latest version of DNG?

You tell us. Are you suggesting there is no backwards compatibility? And who’s converting these files with these products into a DNG that can’t be read into the products and why?

I've seen posts here complaining that non-Adobe software can't deal with latest DNGs.

Not as many as those posts complaining about non manufacturer’s raw converters that can’t deal with the latest proprietary raw. It happens every time a new camera is released.

Personally, I think that a standard raw format would be nice, but not essential. I would much rather see effort put into improved lens corrections and further improvements in noise reduction and the overall raw engine than in trying to get a raw standard through a standard setting group.

By who? Using this logic, IF Adobe didn’t have to hack every new proprietary raw file to process that data, if instead the camera spit out a DNG, Adobe would have more resources to improve NR (which they are doing any way). That is true for all 3rd party raw converter software vendors. They could all spend a lot more time on their core functionality than screw around with yet another proprietary raw file. Your argument here doesn’t wash either, sorry.

[Alan Goldhammer]

I think that the points Les makes are easily dismissed.  When I was still gainfully employed (no I wasn't laid off, just retired), I did a lot of work on barcoding of pharmaceutical packaging.  The underlying technology was privately developed and then placed with a standards developing organization that has since moved on to develop a number of other technologies that are now in use world wide.  The same thing would happen with DNG; of course Adobe developed it but it is in everyone's interest to make it fully compatible with what all users need.  The only way to do this is through a standards process where all the stakeholders have an opportunity to sit down at the table and discuss what the needs are.  Now there may be some things that get dropped and some new things that get added but that's the way these things go and it's been my experience with two groups that I worked in that the job gets done.  I don't think that Java is a good example to use because Sun/Oracle really never turned this into a standardized language.  Maybe C++ is better where there was a good process to standardized the language yet the compiler developers could tweak their part of things to give fast compile times and tighter operating directions in the final executable.  Same thing here, LR might have a different way of using the DNG than some other RAW processing tool but that really doesn't matter unless one is looking at the DNG as a final finished image.  That could be the end result of the standards process but maybe not and perhaps TIFF is suitable as an archival image format.  It really doesn't matter, what does is to get the parties talking to one another about needs and an objective goal.  This is not insurmountable and if you look at the universality of bar codes (both 1 and 2 dimensional as well as RFID chips which encode the same information structures) you see it can be done.

[madmanchan]

Most of the current formats used by the vendors are very close to DNG (TIFF).  They all use the same basic IFD structure (which came from TIFF) and have nearly all the basic tags.  There are some minor differences in lossless image compression technology, but frankly they all have roughly the same compression ratio (about 2 to 1). 

[deejjjaaaa]

And no, as yet I’ve yet to see the valid reasons for not adopting DNG.

for example necessary to disclose your developments to competition and gain their approval and delay release of techology to market ... that is assuming that DNG is controlled by ind. organization where camera companies are members... if DNG is controlled by Adobe - again Adobe may start making cameras tomorrow, MS started to make computers and Google started to make phones

[deejjjaaaa]

Using this logic, IF Adobe didn’t have to hack every new proprietary raw file to process that data, if instead the camera spit out a DNG, Adobe would have more resources to improve NR (which they are doing any way). That is true for all 3rd party raw converter software vendors. They could all spend a lot more time on their core functionality than screw around with yet another proprietary raw file. Your argument here doesn’t wash either, sorry.

I guess Adobe supports (allegedly) mr Coffin (and may be not only Adobe)... so it is not a factor... and camera profiling is something that every self-respecting raw convertor' vendor has to do anyways.

[deejjjaaaa]

That's actually one of DNG's benefits - the flexibility it delivers and the lack of proprietary software tie-down. Quite a few Linux folk I know have DNG workflows, and they can't/won't use Adobe software.

and the same exactly software will be able to handle non DNG files exactly in the same manner... because I'd assume they are already using non Adobe tools to make DNG files from non DNG raws (or you are talking about folks who use either Ricoh/Pentax or Leica gear or something)... so where is the problem and where is a tie down ?