When Importing your Images off a CF or SD Card into Lightroom…..

[phcorrigan]

Do you know of any credible source of information that can confirm that any kind of specific Raw format integrity check is performed before/during the conversion to DNG?

Take a look at pages 249-250 in Peter Krogh's "The DAM Book," Second Edition. As I said, I have personally experienced the integrity check of the DNG converter.

[digitaldog]

Take a look at pages 249-250 in Peter Krogh's "The DAM Book," Second Edition. As I said, I have personally experienced the integrity check of the DNG converter.

More here online:http://thedambook.com/dng-verification-in-lightroom-5/

[digitaldog]

Just tested the LR validation as Peter talks about. Just one small folder but one DNG did pop up as 'not valid' or something like that. Yet double click, Photoshop opens and ACR shows the image. Did one tiny tweak, saved DNG. Still pops an error Not Validated (1). Shows up in LR fine. So I selected it and asked LR to 'Convert to DNG' (you can convert a converted DNG). After that, no error about validation.

[Bart_van_der_Wolf]

No, I don't use DPP for likely the same reasons you don't.  Just saying that if Canon actually did something with it, it might have a slight edge in the conversion process.

Hi Glenn,

I haven't had a possibility to test the new DPP 4 capabilities yet, because my camera is not (yet) on the list of supported models, so I don't know if something in the image quality has improved. I do know that e.g. highlight recovery has been changed.

Things that may come in handy (especially for things like focus stacking) are e.g. the dust removal which can be stored in the file. And the lens corrections that I've tried, did look convincing. These features are already available in DPP 3.x.

DPP is not my main Raw converter, but I'm pretty confident that it can detect unrecoverable file corruption better than anything else.

And maybe I'm lucky - but I haven't gotten to anywhere near the number of formats noted by Bart, and I always format after DL'ing.

I also never experienced any Raw file corruptions while or after reading from the card, either by tethering the camera to the computer or via a separate card reader. I even restored some files that were accidentally deleted from the card to soon. So you are not the only one who's 'lucky'.

The formatting frequency is not the problem (although it could reset the write map), it's the number of times that a solid state memory cell can be written to (I believe it's typically 1000x, some cards can be written to a bit more). Wear leveling will attempt to prevent that certain cells are written to much more frequently than others, which may cause total corruption of the card if one is unlucky when the limit is reached.

My fingers will be crossed from now on.

I wouldn't worry too much, I've stopped doing that after more than some 10 years of digital capture without a single issue, especially when files are deleted instead of the card formatted. I was amazed how many files could be recovered, in addition to the few ones that I wanted to recover after all. I remember that I used both the Sandisk solution that came with my cards, and Recuva from Piriform.

Cheers,
Bart

P.S.  Some manufacturers claim 100 000 write cycles are possible with their devices. If true (hard to verify without deliberately exhausting some of those flash memory cards) that would reduce the importance of "formatting versus deleting" for most photographers a lot.

[digitaldog]

Still pops an error Not Validated (1). Shows up in LR fine. So I selected it and asked LR to 'Convert to DNG' (you can convert a converted DNG). After that, no error about validation.

These are older DNG's prior to the check sum so while they are probably fine, I'm getting this 'error' because the version is old. Probably have to consider doing a batch upgrade.

[Bart_van_der_Wolf]

Take a look at pages 249-250 in Peter Krogh's "The DAM Book," Second Edition. As I said, I have personally experienced the integrity check of the DNG converter.

Hi Patrick,

I'm not contesting the usefulness of verifying that data remains unchanged. I'm just not convinced yet that a conversion to DNG is better at detecting a Raw image data change upon initial import than e.g. the native Raw converter that came with the camera.

In fact, step 1. in Krogh's suggested Ingestion workflow is; "1. Download: Move the images from the camera to the computer."

He later expands on that step as follows: (quoting a fragment under assumption of Fair use, colored emphasis is mine)

No matter what the download type, there is a possibility of transfer error producing corrupt files. Since download programs don't do a bit-for-bit comparison with the data on the card, you will want to do a visual inspection of the images before erasing the media card. Even if the software did a bit-for-bit comparison with the files on the card, you'd still want to do a visual inspection before erasing the card -- some read errors of the card data can be recovered using data recovery software.

Important to note at this stage is that there is software that does do a bit-for-bit comparison between the data on the card and data after downloading it. The ASMP website specifically mentions ChronSync on Mac and SyncBack on PC, as examples of programs that do that.

In addition, I'd have to assume that a native Raw converter would be better at repairing a corrupted Raw file, the structure and data of which they understand intimately, than another converter. Of course, when things are that problematic that the file structure is corrupted beyond repair, then any attempt to convert it with any converter would (most likely) fail.

In step 6. of Krogh's workflow (after Renaming, Applying bulk Meta data, Applying initial image settings, and making a Backup) he gets to : "6. Convert to DNG (optional). You can make a DNG in the ingestion process and take advantage of the data validation hash for added security." Nothing is mentioned about specific image data testing before the hash is calculated.

So, AFTER conversion to DNG, which calculates a hash based on the data it finds (regardless of it's validity at that time) and the DNG converter adds it to the file, it's nice to be able and verify if at any later moment in time that image data section of the DNG is still the same (that no 'bit rot' over time has occurred).

Since DNGs will be rewritten every time when changes are made during editing of the image processing parameters, it's logical that the data verification hash only covers the original image data section in the DNG. In fact, given the frequent re-writing it's essential to verify that at least the image data section survived. Re-writing increases the risk of corruption, so it should be mandatory to check for successful re-writes.

Alternatively, when storing the original camera Raws (non-DNG) which do not get re-written (so can only ever change due to errors in file-management or 'bit-rot'), one can also use a separate hash calculation and verification application to do the same verification, for single files or in bulk, without having to use an image editor (it can run as a background process).

Cheers,
Bart

[Bart_van_der_Wolf]

These are older DNG's prior to the check sum so while they are probably fine, I'm getting this 'error' because the version is old. Probably have to consider doing a batch upgrade.

Indeed, it seems that those files were most likely just fine. It presumably was the not yet added hash check that triggered the inability to validate (it didn't say that validation failed, but just that is was not validated).

At least, the discussion allowed you to update some older files and now add the hash for later verification.
Progress was achieved ...

Cheers,
Bart

[Damon Lynch]

The real issue is that for any given temporal span, the file can be corrupted at any point. Obviously our goal is to maintain file integrity throughout the entire temporal span, from the moment it's written to the card(s) until the file's end of life. File verification at import time is fine, but that's only a miniscule fraction of a good photo's overall lifetime.

Moreover, on almost all contemporary systems the hash only tells you something has gone wrong. It does nothing to help recovery. Errors are typically silently propagated, corrupting backups. So by the time you realize you have a bad file, your duplicate copies in your backup are also corrupted.

[phcorrigan]

I'm just not convinced yet that a conversion to DNG is better at detecting a Raw image data change upon initial import than e.g. the native Raw converter that came with the camera.

The native RAW converter will convert to some other format such as JPEG, TIFF, etc. Typically if you are using Lightroom you are not using the native RAW converter prior to import, so even if it does a verification it doesn't help when importing into Lightroom.

I agree completely with Peter about a visual inspection. I also know that there are tools to to a bit-for-bit comparison on import and I do recommend using such tools.

So, AFTER conversion to DNG, which calculates a hash based on the data it finds (regardless of it's validity at that time) and the DNG converter adds it to the file, it's nice to be able and verify if at any later moment in time that image data section of the DNG is still the same (that no 'bit rot' over time has occurred).

Since DNGs will be rewritten every time when changes are made during editing of the image processing parameters, it's logical that the data verification hash only covers the original image data section in the DNG. In fact, given the frequent re-writing it's essential to verify that at least the image data section survived. Re-writing increases the risk of corruption, so it should be mandatory to check for successful re-writes.

Alternatively, when storing the original camera Raws (non-DNG) which do not get re-written (so can only ever change due to errors in file-management or 'bit-rot'), one can also use a separate hash calculation and verification application to do the same verification, for single files or in bulk, without having to use an image editor (it can run as a background process).

I agree with all of the above. All I said was that the DNG converter does an image verification during the conversion process prior to or during the import into Lightroom. Whether or not native RAW converters do the same is not an issue when importing to Lightroom since you are not using them.

[Bart_van_der_Wolf]

All I said was that the DNG converter does an image verification during the conversion process prior to or during the import into Lightroom.

Hi Patrick,

I have not found any evidence for verification prior to conversion.

Failure to import in LR is no proof, other Raw converters would probably have also failed, for the simple reason that the conversion was not possible because some of the offset tags in the file were (pointing to) garbage.

Again, any evidence to the contrary is welcome, I have not found any sofar.

Cheers,
Bart

[phcorrigan]

Hi Patrick,

I have not found any evidence for verification prior to conversion.

Failure to import in LR is no proof, other Raw converters would probably have also failed, for the simple reason that the conversion was not possible because some of the offset tags in the file were (pointing to) garbage.

Again, any evidence to the contrary is welcome, I have not found any sofar.

Cheers,
Bart

You misquoted me. I didn't say prior to conversion, I said prior to import. The issue I had was that files were corrupted in the field during transfer to a portable hard drive via someone else's laptop. In this case I used the standalone DNG converter prior to importing into Lightroom (I have since changed my procedure to convert upon import). The DNG converter detected the corruption, which I did not see when viewing the original RAW files.  It wasn't a matter of the import failing, because it happened prior to import. It was that the DNG converter was able to detect the corruption, even though it did convert the files. When I viewed them they had red lines accross them. BTW, bit-for-bit comparison wouldn't have helped here, since the files on disk were corrupted prior to import.

Other RAW converters are not an issue, since I don't know anyone who converts their RAW files to anything other than DNG when importing to Lightroom. They either import as is or convert to DNG, so anything having to do with other RAW converters doesn't apply to this discussion. In other words, it doesn't matter what other RAW converters would do since you wouldn't use them prior to or during importing to Lightroom.

As far as evidence goes, I can only refer you to Peter Krogh's book, which verifies what I said, and to the description of Marc Rochkind's ImageIngester (http://basepath.com/new/detail-ImageIngester.php) which states very clearly that it uses the DNG converter to verify RAW files. I don't know what other evidence you would like. I suppose you could contact Adobe directly if you don't believe me, Peter or Marc.