When Importing your Images off a CF or SD Card into Lightroom…..

[uintaangler]

Do you simply click on Copy and Move or do you click on Convert to DNG?

[Jim MSP]

I first select a folder that I want them in, then I "copy" them there.
I don't use DNG - never have, never will. I want the freedom to use multiple programs from non-Adobe suppliers on the original raw.

Jim

[uintaangler]

I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me

[wolfnowl]

Some people keep the raw files, some convert to DNG on import, some convert to DNG on import AND embed the original raw files, and some will import the raw files, prune out the 'non-keepers' and convert the rest to DNG. There are pros and cons... pick one that works for you!

Mike.

[Rhossydd]

there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me

Is that actually important to you ? If you're regularly seeing problems with file integrity you ought to look at the other aspects of the image capture system first, faulty camera, faulty card, faulty card reader etc.

I've been shooting digitally since 2000 and haven't had any issues of file corruption when importing files onto my computers.
 

[eliedinur]

I convert to DNG (with Quick Load Data) the work copies and save the backups in original Raw format. Best of both worlds.

[phcorrigan]

I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me

That verification saved me from some corrupted files. When travelling I copied them to a portable hard disk via someone else's computer and several images were corrupted. Fortunately I still had them on the SD cards so I was able to import them from there.

I do convert to DNG and I save my original RAW files as well. I ingest the files to the directory where I will keep the original RAWs, then use the free FastStone Image Viewer to batch rename them using my naming convention. I then import the renamed files into Lightroom as DNGs. FastStone is Windows only, but XNViewMP (http://www.xnview.com/en/xnviewmp) has similar capabilities and is available for Mac, Linux and Windows.

[PeterAit]

I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me

But if you don't convert to DNG and just keep the RAW files, there is zero chance that your DNG files will be corrupted - because you won't have any!

[phcorrigan]

But if you don't convert to DNG and just keep the RAW files, there is zero chance that your DNG files will be corrupted - because you won't have any!

The problem isn't corrupted DNGs, it's corrupted RAW files. Converting to DNG verifies your RAW files.

[Bart_van_der_Wolf]

Converting to DNG verifies your RAW files.

Hi Patrick,

Does it? The conversion process would only be able to detect something it understands to be wrong. So what does it detect? It won't be able to detect what's wrong in the Maker notes of the Raw file, because it doesn't know what the data there means. If there are bits changed in the image data section, it won't be able to see that because it doesn't know how the image is supposed to look.

Or is it just a DNG file integrity checker only, once the DNG is created and a checksum is calculated for later verification?

Reading a Raw file with its native, camera maker supplied, converter would seem to be able and detect issues much better. And a corrupt file (e.g. wrong size) structure would probably be detected by the operating system anyway.

And when a somehow corrupted file is detected, then what?

Cheers,
Bart

[phcorrigan]

It detects image corruption during the conversion process. While travelling I had several RAW images corrupted during a file transfer to my portable hard disk via someone else's computer. Because the DNG converter detected the corruption I was able to pull un-corrupted files off the original SD card.

Marc Rochkind's image ingestion utility Ingestamatic (http://basepath.com/site/detail-Ingestamatic.php) actually has the option to use the Adobe DNG converter to detect image corruption even when ingesting images in native RAW format. In effect, it converts the image to DNG, verifies the integrity of the image, then dumps the DNG file and imports the native RAW file.

[Bart_van_der_Wolf]

It detects image corruption during the conversion process. While travelling I had several RAW images corrupted during a file transfer to my portable hard disk via someone else's computer. Because the DNG converter detected the corruption I was able to pull un-corrupted files off the original SD card.

Hi Patrick,

I understood that, but the question is whether e.g. the native Raw converter would have given the same corruption warning. My guess is that it would have.

What DNG does allow to do, as of DNG specification 1.2, is creating an MD-5 validation hash that can be embedded in the DNG file itself. This hash refers only to the unchanging source image file so it will remain useful, even after the DNG file is readjusted or additional metadata is added. But it only serves to verify the DNG image data section integrity after the DNG has been created, a bit like storing your file(s) in a ZIP archive.

Cheers,
Bart

[phcorrigan]

Hi Patrick,

I understood that, but the question is whether e.g. the native Raw converter would have given the same corruption warning. My guess is that it would have.


Cheers,
Bart

The DNG converter can detect corruption upon import. When you import as the native RAW you are not converting anything therefore you would likely not detect corruption until you viewed or edited the image.

[digitaldog]

Do you simply click on Copy and Move or do you click on Convert to DNG?

Personally, Convert to DNG. And yes, as other's have stated,verification also saved me from some corrupted files. I NEVER format a card until after import and conversion.

[phcorrigan]

I NEVER format a card until after import and conversion.

Ditto.

[Damon Lynch]

But at the end of the day we rely on file systems that suffer from bit rot. That's the brutal reality. The problem will be solved by next-gen file systems or the use of distributed file repositories that duplicate and hash every file across multiple devices. Until then data integrity is pure luck.

[Bart_van_der_Wolf]

Personally, Convert to DNG. And yes, as other's have stated,verification also saved me from some corrupted files.

Hi Andrew, or anybody else who knows the answer,

Any Raw converter that attempts to read a corrupt file structure will report an error because it fails to make sense of the file data.

I've searched the internet, and I've not yet been able to find any information other than about the verification after converting to DNGs, so after a hash / checksum has been calculated for the image section of the DNG.

Do you know of any credible source of information that can confirm that any kind of specific Raw format integrity check is performed before/during the conversion to DNG? So anything that is more rigorous than e.g. the native Raw converter that came with the camera.

I NEVER format a card until after import and conversion.

That's good practice. Formatting cards can also shorten their useful life, because the wear-leveling data (that allows to re-use storage areas leveled with the maximum number of reliable rewrites, say 1000 times) can get overwritten (depends on the type of formatting). Deleting files may be more cumbersome than formatting, but can also allow a longer useful lifespan of a card.

Cheers,
Bart

[Glenn NK]

I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me

I've DL'd thousands of RAW files for eight years and have never witnessed any corruption in a file.

In any event, it is possible that the manufacturer of my cameras will keep improving the native software so that I could in the future, re-work my RAW images.  If they were converted to DNG, this option would not be available.  Note that for ease of PP, I do not use the camera's native software, but use LR exclusively.   There are photographers that suggest (somewhat convincingly) that software written by the camera manufacturer has a slight advantage in this respect over other software packages - converting to DNG would remove this option.

Although the DNG format is being used extensively, it is still another conversion/wrapping step that potentially carries some risk.

Corruption is more likely to happen with CF/SD cards that have faults or; rather than being formatted in-camera after each use, have had files erased.

I haven't used any cards anywhere near the number of times Bart has mentioned.

Glenn

[digitaldog]

I've DL'd thousands of RAW files for eight years and have never witnessed any corruption in a file.

You are lucky and not alone but that's not true for everyone. S%^t happens.

In any event, it is possible that the manufacturer of my cameras will keep improving the native software so that I could in the future, re-work my RAW images.

It is and if you don't mind using that, you're all set. For me DPP isn't anywhere near the top of the list. I've got a couple on the top that have no issues with DNG.

[Glenn NK]

For me DPP isn't anywhere near the top of the list. I've got a couple on the top that have no issues with DNG.

No, I don't use DPP for likely the same reasons you don't.  Just saying that if Canon actually did something with it, it might have a slight edge in the conversion process.

And maybe I'm lucky - but I haven't gotten to anywhere near the number of formats noted by Bart, and I always format after DL'ing.

My fingers will be crossed from now on.