Some notes on restoring metadata to intermediary files in workflow

[Damon Lynch]

The problem

Sometimes image metadata is mangled by the programs we use to create our images. For example, if you assemble images into a panorama and/or an HDR image, you might find that the lens metadata is deleted or corrupted, including the lens model, focal length, aperture etc. The RAW file has the correct metadata, but not the intermediary TIFF. I've even seen a few cases in my own recent work where a DNG's metadata is subtly corrupted.

If you don't care about this metadata, then none of this is a problem. However, if you like the lens information to be displayed correctly on Web services like Flickr, or you like to use the metadata browser in Lightroom, then it's helpful to fix the problem. What follows are some brief notes, for those who may be interested. I was poking around with this stuff earlier today and perhaps others will find my findings useful.

Restoring the metadata

The restoration is a low risk operation, because we are never modifying the RAW, only using it as a source. The task is to modify the TIFF or DNG, which is then used to output the web ready jpegs, or files for your clients or what have you. There are two steps:

• Where a lot of metadata is missing or is wrong, exiftool has a handy option -tagsFromFile which allows you to bulk copy metadata from one of the source RAW files into the TIFF/DNG. (If you like shooting panoramas from vertically assembled images, be sure to restore the exif orientation tag in the TIFF/DNG after the exiftool operation).
• Where the lens model is missing or wrong in Lightroom or on Flickr (e.g. you want to see EF24-105mm f/4L IS USM instead of 24.0 - 105.0 mm), the tag to add or modify is what's known as an auxiliary XMP tag, specifically the tag Lens. There are typically quite a few tags that have something to do with lens metadata, found in exif, maker notes, and XMP metadata, but for Adobe users at least, this auxiliary XMP tag Lens is the one to modify. My assumption is that Adobe creates this tag when importing the RAW. In any case, for reference info see exiftool and exiv2. You can get the correct value to insert from one of two places: if you have a recent camera that creates Exif metadata to the version 2.3 Exif specs, it's in the Lens Model. If not, try to find it in the maker notes. I use Canon and it's easy to find. I don't know about other systems.

If step 1 is necessary, then very likely step 2 is as well. In other words, step 1 is optional, but step 2 is not.

Automation

Making these changes by hand would be mind numbingly tedious, but it's possible to automate it. I did so today by writing a Python script, and calling it from Lightroom using Jeffrey Friedl's "Run Any Command" Lightroom Export Plugin. My python script takes one or more TIFFs or DNGs and automatically figures out the corresponding RAW files from which to copy the correct metadata, and either reads & writes the metadata directly or calls exiftool to do it. No user intervention is required, apart from asking Lightroom to read the metadata again.

I don't recommend using a solution like my Python script as a general purpose solution. It works just fine for my Canon files, workflow and programming skills. However Lightroom plugins are written in Lua, and in my case I don't see myself taking the time to write one of those. If I'm not the only one who cares about this stuff, however, one of the folks who is into writing Lightroom plugins like John Beardsworth or Jeffrey Friedl could probably rustle up something fairly quickly.

Of course someone may have already written something nice already and I just don't know about it!

[Redcrown]

Damon,

Thanks for the script. Sounds handy for fixing large batches of images.

I've been using a Photoshop only technique which also restores EXIF data on the output of HDR, pano stiching, and focus stacking utilities. I simply load one of the origial images, cut and paste the composite image into it, flatten and save with whatever custom name I want.

The resulting saved composite image then has the full EXIF of the original. Sometimes you have to resize the original first to the size of the composite if the composite is larger (panos).

I have not tried your script yet, but curious how it "automatically figures out the corresponding raw file". If an HDR process starts with image001.cr2, image002.cr2 and image003.cr2, and the resulting file is named image001-003.tif, how does the script pick a raw file?

[john beardsworth]

I don't recommend using a solution like my Python script as a general purpose solution. It works just fine for my Canon files, workflow and programming skills. However Lightroom plugins are written in Lua, and in my case I don't see myself taking the time to write one of those. If I'm not the only one who cares about this stuff, however, one of the folks who is into writing Lightroom plugins like John Beardsworth or Jeffrey Friedl could probably rustle up something fairly quickly.

Thanks for volunteering me, Damon

My Syncomatic plugin can be handy for this kind of job and uses either the filename (eg 123.jpg copies metadata to 123.nef) or the capture time. I put a lot of effort into ease of use - I'm not a trained programmer and have a low tolerance for geeky interfaces (it was painful learning Lua and I've never tackled Python!).

By the way, this kind of workflow isn't just for when metadata has been stripped. It can be equally useful where there's a need to synchronise metadata. For instance, you might send someone JPEGs to do keywording and captioning in Bridge and then want to sync their work to your raw files.

John

[Damon Lynch]

I have not tried your script yet, but curious how it "automatically figures out the corresponding raw file". If an HDR process starts with image001.cr2, image002.cr2 and image003.cr2, and the resulting file is named image001-003.tif, how does the script pick a raw file?

I haven't posted the code anywhere yet. I'm not sure I will because I do think a solution like the one John has put together will be more useful to most people. Having said that my script figures out the RAW file name by using a regular expression to analyze the the name of the derivative file. This approach works because I use a consistent naming scheme for all my RAW files. Basically any naming scheme that uses unique names, consistently applied, can be subject to automation. But you need someone with coding skills to be able to figure out how.

[Damon Lynch]

Thanks for volunteering me, Damon

My Syncomatic plugin can be handy for this kind of job and uses either the filename (eg 123.jpg copies metadata to 123.nef) or the capture time. I put a lot of effort into ease of use - I'm not a trained programmer and have a low tolerance for geeky interfaces (it was painful learning Lua and I've never tackled Python!).

By the way, this kind of workflow isn't just for when metadata has been stripped. It can be equally useful where there's a need to synchronise metadata. For instance, you might send someone JPEGs to do keywording and captioning in Bridge and then want to sync their work to your raw files.

John

Hi John, I totally agree ease of use is critical and if you managed to nail that one then I think you should be proud of your work! I had seen your plugin but I didn't realize until now that it could be used to solve the kind of problem I'd outlined above too. I assume it calls on Exiftool to extract and transfer the metadata, right? Does it also handle situations where a filename has a suffix before the extension like '_SNSHDR_7' or ' Panorama' or '_DNG'? These are typically the kind of suffixes that get added in my day-to-day workflow by the various non-Adobe programs I tend to use. Fortunately these external programs add only a suffix and don't truncate the original name, which makes life simpler. 

Damon

[john beardsworth]

I assume it calls on Exiftool to extract and transfer the metadata, right? Does it also handle situations where a filename has a suffix before the extension like '_SNSHDR_7' or ' Panorama' or '_DNG'? These are typically the kind of suffixes that get added in my day-to-day workflow by the various non-Adobe programs I tend to use.

No, it's purely Lightroom, Damon. So both sets of images have to be in Lightroom, and it doesn't rebuild any missing EXIF since that's not allowed by Lightroom's SDK. It covers IPTC metadata and also synchronises adjustments. Again, this is my ease of use bias, and an assessment of what a lot of people need.

As you say, other apps or LR's Edit With often add suffixes, and people typically do things like prefix photos with their own names when outsourcing, so it's important to allow for such naming differences and the plug-in has always handled suffixes and prefixes to match series of images such as 123-edit.tif to 123.nef.

John