Pages: [1]   Go Down

Author Topic: Embed original RAW into DNG after DNG creation and development  (Read 1251 times)

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Embed original RAW into DNG after DNG creation and development
« on: October 23, 2018, 10:19:35 am »

Hi,

I prefer to use Adobe products to work on my images, however I am not satisfied with the demosaicing performed by ACR for my Fuji X files, so I use Iridient X-Transformer to convert my RAWs into DNGs, and then work on the DNGs with Adobe products.

I always kept my original RAWs in another directory, I didn't embed them into the DNGs, but now I want to change the file structure I use, and it would make more sense to me to now embed the RAWs into the DNGs.  Can I do it now, after development? Obviously I want to preserve the develop settings I applied to the DNGs.

It would be best if I could do this with a command line tool, so I could easily script it for all my files, but I am open to any suggestions.

Alternatively, if this is not possible, I'd also be happy with a way of importing the RAWs into Lightroom (assume the DNGs are already imported), and automatically stack the RAWs under the DNGs (so the developed DNGs are what I see by default). Can this be done?

Many thanks!
Logged

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Re: Embed original RAW into DNG after DNG creation and development
« Reply #1 on: October 23, 2018, 10:58:14 am »

Hmm, I see an undocumented SetOriginalRawFileData function in the Adobe SDK.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Embed original RAW into DNG after DNG creation and development
« Reply #2 on: October 23, 2018, 12:20:55 pm »

I'd separate the two, archive the proprietary raws. One 'disadvantage' to DNG is that even tiny changes, to just metadata flags the entire document to be backed up. Rather than just a sidecar file. The document will be MUCH larger with the proprietary raw embedded. Now all my backups take place unattended automatically every night, even to the cloud. But if you have lots of images undergoing changes, small or large, that could be a factor. Lightroom has a option to copy the proprietary raws to another drive while converting to DNG while importing. That's probably the better option. The proprietary raws are simply copied, nothing more while conversion to DNG.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Re: Embed original RAW into DNG after DNG creation and development
« Reply #3 on: October 24, 2018, 05:19:33 am »

I want to have a single source of truth. It's confusing to have both DNGs and raw files in different places. Not all raws were converted to DNGs. When I see one raw file I can't immediately tell if it's a work file or not. I'm not concerned of files doubling in size. In total, they use up the same space, and one single file is not too big still (some people here shoot medium format, this is only 16Mpix).

Yes, it's very unfortunate that DNGs do not use sidecar files, though you can sort of force that to happen if you make your DNGs read-only. Adobe and FastRawViewer supports this somewhat well (though I hit a bug once that made me reimport the XMP manually for a few hundred images), but other software like Photo Mechanic, Capture One, etc, doesn't support this type of workflow at all, so in the end I abandoned it. My backup software is not bothered by everchanging files, it will only save the new blocks, and my files change rarely, usually just once, though it would still be better if we just had sidecar files.

It would be fine to keep the raws and DNGs separate, as long as I cataloged both, and automatically stacked the raws under the DNGs. Then there would be no confusion for me. But I can't do that manually with thousands of images and I don't know of an automated way to do it.
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: Embed original RAW into DNG after DNG creation and development
« Reply #4 on: October 24, 2018, 05:55:08 am »

I'm pretty sure there's no way to automate embedding the original RAW into an existing demosaiced DNG.

It would be fine to keep the raws and DNGs separate, as long as I cataloged both, and automatically stacked the raws under the DNGs. Then there would be no confusion for me. But I can't do that manually with thousands of images and I don't know of an automated way to do it.

In Lightroom, there is a command: Photo > Stacking > Auto Stack by Capture Time.

For stacking, photos need to be in the same folder. Maybe this is convenient, but there's also a case for keeping the raws and the DNGs  in entirely-separate folders and drives - it's similar to the point made about not embedding.

Whatever you do, you are going to find it confusing to have two copies of every photo in LR. My Syncomatic plugin copies adjustment and IPTC metadata between corresponding files, using the filenames, capture times or stacking to match them.
Logged

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Re: Embed original RAW into DNG after DNG creation and development
« Reply #5 on: October 24, 2018, 07:29:38 am »

In Lightroom, there is a command: Photo > Stacking > Auto Stack by Capture Time.

Oh, I forgot to mention I know about this, but unfortunately, it stacks them in the wrong order. I guess I could work around that if I change the capture time of the DNG slightly?
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: Embed original RAW into DNG after DNG creation and development
« Reply #6 on: October 24, 2018, 08:38:03 am »

Oh, I forgot to mention I know about this, but unfortunately, it stacks them in the wrong order. I guess I could work around that if I change the capture time of the DNG slightly?

There is an Edit Capture Time function, but it's really like I said before, whatever you do, you are going to find it confusing to have two copies of every photo in LR. Stacking might resolve one problem in folders, but let's say you're using smart collections - every smart collection is going to need an additional line to select only the DNGs. That's because stacking is local to folders and collections, so images can be stacked differently in different collections. In your situation, after creating the demosaiced DNGs I would keep the raws out of LR, either keeping them in the same folder and simply removing them from the catalogue, or moving them to a parallel folder structure and then removing them from LR.
Logged

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Re: Embed original RAW into DNG after DNG creation and development
« Reply #7 on: October 24, 2018, 08:42:55 am »

You are right. Keeping both DNG and raws in Lightroom doesn't seem to be a good idea. Thanks for talking me out of it.

Still interested in the option of post-facto embedding them in DNGs though. It's clear now that no existing product can do it, but reading Adobe DNG SDK I see that it's possible to write some custom program, in principle.
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: Embed original RAW into DNG after DNG creation and development
« Reply #8 on: October 24, 2018, 09:23:49 am »

Maybe the author of X Transformer could do it?
Logged

Damon Lynch

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 330
    • http://www.damonlynch.net
Re: Embed original RAW into DNG after DNG creation and development
« Reply #9 on: October 25, 2018, 03:48:21 pm »

How about this:

  • Use a program other than Lightroom to download and rename your files to a temporary file processing location where you can take the first step of developing them in your preferred raw converter
  • Write a script that will copy the newly created DNG files into your preferred folder structure and import them into Lightroom. Here I'm assuming there are no proprietary raw files in the existing folders, meaning you don't need to mess about filtering which files you will import into Lightroom. I'm also assuming your raw converter in step one produces filenames that you have some control over.
  • Once you've done your processing in Lightroom, use another script you've written to copy the matching raw files into your preferred folder structure, right alongside the DNG files. If you want, you can "mark" the raw files that have a matching DNG file with a filename suffix, but that's entirely optional, and personally I wouldn't bother.

This means you can keep all your files together, and not have to worry about any duplicate files in Lightroom. As long as you maintain your folder structure and keep it consistent through the years, you shouldn't run into any further difficulties.
Logged
Pages: [1]   Go Up