Luminous Landscape Forum
Raw & Post Processing, Printing => Adobe Lightroom Q&A => Topic started by: uintaangler on June 25, 2014, 05:35:16 pm
-
Do you simply click on Copy and Move or do you click on Convert to DNG?
-
I first select a folder that I want them in, then I "copy" them there.
I don't use DNG - never have, never will. I want the freedom to use multiple programs from non-Adobe suppliers on the original raw.
Jim
-
I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me
-
Some people keep the raw files, some convert to DNG on import, some convert to DNG on import AND embed the original raw files, and some will import the raw files, prune out the 'non-keepers' and convert the rest to DNG. There are pros and cons... pick one that works for you!
Mike.
-
there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me
Is that actually important to you ? If you're regularly seeing problems with file integrity you ought to look at the other aspects of the image capture system first, faulty camera, faulty card, faulty card reader etc.
I've been shooting digitally since 2000 and haven't had any issues of file corruption when importing files onto my computers.
-
I convert to DNG (with Quick Load Data) the work copies and save the backups in original Raw format. Best of both worlds.
-
I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me
That verification saved me from some corrupted files. When travelling I copied them to a portable hard disk via someone else's computer and several images were corrupted. Fortunately I still had them on the SD cards so I was able to import them from there.
I do convert to DNG and I save my original RAW files as well. I ingest the files to the directory where I will keep the original RAWs, then use the free FastStone Image Viewer to batch rename them using my naming convention. I then import the renamed files into Lightroom as DNGs. FastStone is Windows only, but XNViewMP (http://www.xnview.com/en/xnviewmp) has similar capabilities and is available for Mac, Linux and Windows.
-
I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me
But if you don't convert to DNG and just keep the RAW files, there is zero chance that your DNG files will be corrupted - because you won't have any!
-
But if you don't convert to DNG and just keep the RAW files, there is zero chance that your DNG files will be corrupted - because you won't have any!
The problem isn't corrupted DNGs, it's corrupted RAW files. Converting to DNG verifies your RAW files.
-
Converting to DNG verifies your RAW files.
Hi Patrick,
Does it? The conversion process would only be able to detect something it understands to be wrong. So what does it detect? It won't be able to detect what's wrong in the Maker notes of the Raw file, because it doesn't know what the data there means. If there are bits changed in the image data section, it won't be able to see that because it doesn't know how the image is supposed to look.
Or is it just a DNG file integrity checker only, once the DNG is created and a checksum is calculated for later verification?
Reading a Raw file with its native, camera maker supplied, converter would seem to be able and detect issues much better. And a corrupt file (e.g. wrong size) structure would probably be detected by the operating system anyway.
And when a somehow corrupted file is detected, then what?
Cheers,
Bart
-
It detects image corruption during the conversion process. While travelling I had several RAW images corrupted during a file transfer to my portable hard disk via someone else's computer. Because the DNG converter detected the corruption I was able to pull un-corrupted files off the original SD card.
Marc Rochkind's image ingestion utility Ingestamatic (http://basepath.com/site/detail-Ingestamatic.php) actually has the option to use the Adobe DNG converter to detect image corruption even when ingesting images in native RAW format. In effect, it converts the image to DNG, verifies the integrity of the image, then dumps the DNG file and imports the native RAW file.
-
It detects image corruption during the conversion process. While travelling I had several RAW images corrupted during a file transfer to my portable hard disk via someone else's computer. Because the DNG converter detected the corruption I was able to pull un-corrupted files off the original SD card.
Hi Patrick,
I understood that, but the question is whether e.g. the native Raw converter would have given the same corruption warning. My guess is that it would have.
What DNG does allow to do, as of DNG specification 1.2, is creating an MD-5 validation hash that can be embedded in the DNG file itself. This hash refers only to the unchanging source image file so it will remain useful, even after the DNG file is readjusted or additional metadata is added. But it only serves to verify the DNG image data section integrity after the DNG has been created, a bit like storing your file(s) in a ZIP archive.
Cheers,
Bart
-
Hi Patrick,
I understood that, but the question is whether e.g. the native Raw converter would have given the same corruption warning. My guess is that it would have.
Cheers,
Bart
The DNG converter can detect corruption upon import. When you import as the native RAW you are not converting anything therefore you would likely not detect corruption until you viewed or edited the image.
-
Do you simply click on Copy and Move or do you click on Convert to DNG?
Personally, Convert to DNG. And yes, as other's have stated,verification also saved me from some corrupted files. I NEVER format a card until after import and conversion.
-
I NEVER format a card until after import and conversion.
Ditto.
-
But at the end of the day we rely on file systems that suffer from bit rot (http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/). That's the brutal reality. The problem will be solved by next-gen file systems or the use of distributed file repositories that duplicate and hash every file across multiple devices. Until then data integrity is pure luck.
-
Personally, Convert to DNG. And yes, as other's have stated,verification also saved me from some corrupted files.
Hi Andrew, or anybody else who knows the answer,
Any Raw converter that attempts to read a corrupt file structure will report an error because it fails to make sense of the file data.
I've searched the internet, and I've not yet been able to find any information other than about the verification after converting to DNGs, so after (https://blogs.adobe.com/jkost/2013/08/dng-verification-in-lightroom-5.html) a hash / checksum has been calculated for the image section of the DNG.
Do you know of any credible source of information that can confirm that any kind of specific Raw format integrity check is performed before/during the conversion to DNG? So anything that is more rigorous than e.g. the native Raw converter that came with the camera.
I NEVER format a card until after import and conversion.
That's good practice. Formatting cards can also shorten their useful life, because the wear-leveling (https://en.wikipedia.org/wiki/Wear_leveling) data (that allows to re-use storage areas leveled with the maximum number of reliable rewrites, say 1000 times) can get overwritten (depends on the type of formatting). Deleting files may be more cumbersome than formatting, but can also allow a longer useful lifespan of a card.
Cheers,
Bart
-
I get that, but I learned an interesting thing today
If you convert your RAW files to DNG, there is a one-click way in Lightroom 5 to verify that all your DNG files are free of any defects, corruption, etc
Seems like a worthwhile trade-off to me
I've DL'd thousands of RAW files for eight years and have never witnessed any corruption in a file.
In any event, it is possible that the manufacturer of my cameras will keep improving the native software so that I could in the future, re-work my RAW images. If they were converted to DNG, this option would not be available. Note that for ease of PP, I do not use the camera's native software, but use LR exclusively. There are photographers that suggest (somewhat convincingly) that software written by the camera manufacturer has a slight advantage in this respect over other software packages - converting to DNG would remove this option.
Although the DNG format is being used extensively, it is still another conversion/wrapping step that potentially carries some risk.
Corruption is more likely to happen with CF/SD cards that have faults or; rather than being formatted in-camera after each use, have had files erased.
I haven't used any cards anywhere near the number of times Bart has mentioned.
Glenn
-
I've DL'd thousands of RAW files for eight years and have never witnessed any corruption in a file.
You are lucky and not alone but that's not true for everyone. S%^t happens.
In any event, it is possible that the manufacturer of my cameras will keep improving the native software so that I could in the future, re-work my RAW images.
It is and if you don't mind using that, you're all set. For me DPP isn't anywhere near the top of the list. I've got a couple on the top that have no issues with DNG.
-
For me DPP isn't anywhere near the top of the list. I've got a couple on the top that have no issues with DNG.
No, I don't use DPP for likely the same reasons you don't. Just saying that if Canon actually did something with it, it might have a slight edge in the conversion process.
And maybe I'm lucky - but I haven't gotten to anywhere near the number of formats noted by Bart, and I always format after DL'ing.
My fingers will be crossed from now on.
-
Do you know of any credible source of information that can confirm that any kind of specific Raw format integrity check is performed before/during the conversion to DNG?
Take a look at pages 249-250 in Peter Krogh's "The DAM Book," Second Edition. As I said, I have personally experienced the integrity check of the DNG converter.
-
Take a look at pages 249-250 in Peter Krogh's "The DAM Book," Second Edition. As I said, I have personally experienced the integrity check of the DNG converter.
More here online:http://thedambook.com/dng-verification-in-lightroom-5/
-
Just tested the LR validation as Peter talks about. Just one small folder but one DNG did pop up as 'not valid' or something like that. Yet double click, Photoshop opens and ACR shows the image. Did one tiny tweak, saved DNG. Still pops an error Not Validated (1). Shows up in LR fine. So I selected it and asked LR to 'Convert to DNG' (you can convert a converted DNG). After that, no error about validation.
-
No, I don't use DPP for likely the same reasons you don't. Just saying that if Canon actually did something with it, it might have a slight edge in the conversion process.
Hi Glenn,
I haven't had a possibility to test the new DPP 4 capabilities yet, because my camera is not (yet) on the list of supported models, so I don't know if something in the image quality has improved. I do know that e.g. highlight recovery has been changed.
Things that may come in handy (especially for things like focus stacking) are e.g. the dust removal which can be stored in the file. And the lens corrections that I've tried, did look convincing. These features are already available in DPP 3.x.
DPP is not my main Raw converter, but I'm pretty confident that it can detect unrecoverable file corruption better than anything else.
And maybe I'm lucky - but I haven't gotten to anywhere near the number of formats noted by Bart, and I always format after DL'ing.
I also never experienced any Raw file corruptions while or after reading from the card, either by tethering the camera to the computer or via a separate card reader. I even restored some files that were accidentally deleted from the card to soon. So you are not the only one who's 'lucky'.
The formatting frequency is not the problem (although it could reset the write map), it's the number of times that a solid state memory cell can be written to (I believe it's typically 1000x, some cards can be written to a bit more). Wear leveling will attempt to prevent that certain cells are written to much more frequently than others, which may cause total corruption of the card if one is unlucky when the limit is reached.
My fingers will be crossed from now on.
I wouldn't worry too much, I've stopped doing that after more than some 10 years of digital capture without a single issue, especially when files are deleted instead of the card formatted. I was amazed how many files could be recovered, in addition to the few ones that I wanted to recover after all. I remember that I used both the Sandisk solution that came with my cards, and Recuva from Piriform.
Cheers,
Bart
P.S. Some manufacturers claim 100 000 write cycles are possible with their devices. If true (hard to verify without deliberately exhausting some of those flash memory cards) that would reduce the importance of "formatting versus deleting" for most photographers a lot.
-
Still pops an error Not Validated (1). Shows up in LR fine. So I selected it and asked LR to 'Convert to DNG' (you can convert a converted DNG). After that, no error about validation.
These are older DNG's prior to the check sum so while they are probably fine, I'm getting this 'error' because the version is old. Probably have to consider doing a batch upgrade.
-
Take a look at pages 249-250 in Peter Krogh's "The DAM Book," Second Edition. As I said, I have personally experienced the integrity check of the DNG converter.
Hi Patrick,
I'm not contesting the usefulness of verifying that data remains unchanged. I'm just not convinced yet that a conversion to DNG is better at detecting a Raw image data change upon initial import than e.g. the native Raw converter that came with the camera.
In fact, step 1. in Krogh's suggested Ingestion workflow is; "1. Download: Move the images from the camera to the computer."
He later expands on that step as follows: (quoting a fragment under assumption of Fair use, colored emphasis is mine)
No matter what the download type, there is a possibility of transfer error producing corrupt files. Since download programs don't do a bit-for-bit comparison with the data on the card, you will want to do a visual inspection of the images before erasing the media card. Even if the software did a bit-for-bit comparison with the files on the card, you'd still want to do a visual inspection before erasing the card -- some read errors of the card data can be recovered using data recovery software.
Important to note at this stage is that there is software that does do a bit-for-bit comparison between the data on the card and data after downloading it. The ASMP website (http://dpbestflow.org/data-validation/transfer-validation#using) specifically mentions ChronSync on Mac and SyncBack on PC, as examples of programs that do that.
In addition, I'd have to assume that a native Raw converter would be better at repairing a corrupted Raw file, the structure and data of which they understand intimately, than another converter. Of course, when things are that problematic that the file structure is corrupted beyond repair, then any attempt to convert it with any converter would (most likely) fail.
In step 6. of Krogh's workflow (after Renaming, Applying bulk Meta data, Applying initial image settings, and making a Backup) he gets to : "6. Convert to DNG (optional). You can make a DNG in the ingestion process and take advantage of the data validation hash for added security." Nothing is mentioned about specific image data testing before the hash is calculated.
So, AFTER conversion to DNG, which calculates a hash based on the data it finds (regardless of it's validity at that time) and the DNG converter adds it to the file, it's nice to be able and verify if at any later moment in time that image data section of the DNG is still the same (that no 'bit rot' over time has occurred).
Since DNGs will be rewritten every time when changes are made during editing of the image processing parameters, it's logical that the data verification hash only covers the original image data section in the DNG. In fact, given the frequent re-writing it's essential to verify that at least the image data section survived. Re-writing increases the risk of corruption, so it should be mandatory to check for successful re-writes.
Alternatively, when storing the original camera Raws (non-DNG) which do not get re-written (so can only ever change due to errors in file-management or 'bit-rot'), one can also use a separate hash calculation and verification application to do the same verification, for single files or in bulk, without having to use an image editor (it can run as a background process).
Cheers,
Bart
-
These are older DNG's prior to the check sum so while they are probably fine, I'm getting this 'error' because the version is old. Probably have to consider doing a batch upgrade.
Indeed, it seems that those files were most likely just fine. It presumably was the not yet added hash check that triggered the inability to validate (it didn't say that validation failed, but just that is was not validated).
At least, the discussion allowed you to update some older files and now add the hash for later verification.
Progress was achieved ...
Cheers,
Bart
-
The real issue is that for any given temporal span, the file can be corrupted at any point. Obviously our goal is to maintain file integrity throughout the entire temporal span, from the moment it's written to the card(s) until the file's end of life. File verification at import time is fine, but that's only a miniscule fraction of a good photo's overall lifetime.
Moreover, on almost all contemporary systems the hash only tells you something has gone wrong. It does nothing to help recovery. Errors are typically silently propagated, corrupting backups. So by the time you realize you have a bad file, your duplicate copies in your backup are also corrupted.
-
I'm just not convinced yet that a conversion to DNG is better at detecting a Raw image data change upon initial import than e.g. the native Raw converter that came with the camera.
The native RAW converter will convert to some other format such as JPEG, TIFF, etc. Typically if you are using Lightroom you are not using the native RAW converter prior to import, so even if it does a verification it doesn't help when importing into Lightroom.
I agree completely with Peter about a visual inspection. I also know that there are tools to to a bit-for-bit comparison on import and I do recommend using such tools.
So, AFTER conversion to DNG, which calculates a hash based on the data it finds (regardless of it's validity at that time) and the DNG converter adds it to the file, it's nice to be able and verify if at any later moment in time that image data section of the DNG is still the same (that no 'bit rot' over time has occurred).
Since DNGs will be rewritten every time when changes are made during editing of the image processing parameters, it's logical that the data verification hash only covers the original image data section in the DNG. In fact, given the frequent re-writing it's essential to verify that at least the image data section survived. Re-writing increases the risk of corruption, so it should be mandatory to check for successful re-writes.
Alternatively, when storing the original camera Raws (non-DNG) which do not get re-written (so can only ever change due to errors in file-management or 'bit-rot'), one can also use a separate hash calculation and verification application to do the same verification, for single files or in bulk, without having to use an image editor (it can run as a background process).
I agree with all of the above. All I said was that the DNG converter does an image verification during the conversion process prior to or during the import into Lightroom. Whether or not native RAW converters do the same is not an issue when importing to Lightroom since you are not using them.
-
All I said was that the DNG converter does an image verification during the conversion process prior to or during the import into Lightroom.
Hi Patrick,
I have not found any evidence for verification prior to conversion.
Failure to import in LR is no proof, other Raw converters would probably have also failed, for the simple reason that the conversion was not possible because some of the offset tags in the file were (pointing to) garbage.
Again, any evidence to the contrary is welcome, I have not found any sofar.
Cheers,
Bart
-
Hi Patrick,
I have not found any evidence for verification prior to conversion.
Failure to import in LR is no proof, other Raw converters would probably have also failed, for the simple reason that the conversion was not possible because some of the offset tags in the file were (pointing to) garbage.
Again, any evidence to the contrary is welcome, I have not found any sofar.
Cheers,
Bart
You misquoted me. I didn't say prior to conversion, I said prior to import. The issue I had was that files were corrupted in the field during transfer to a portable hard drive via someone else's laptop. In this case I used the standalone DNG converter prior to importing into Lightroom (I have since changed my procedure to convert upon import). The DNG converter detected the corruption, which I did not see when viewing the original RAW files. It wasn't a matter of the import failing, because it happened prior to import. It was that the DNG converter was able to detect the corruption, even though it did convert the files. When I viewed them they had red lines accross them. BTW, bit-for-bit comparison wouldn't have helped here, since the files on disk were corrupted prior to import.
Other RAW converters are not an issue, since I don't know anyone who converts their RAW files to anything other than DNG when importing to Lightroom. They either import as is or convert to DNG, so anything having to do with other RAW converters doesn't apply to this discussion. In other words, it doesn't matter what other RAW converters would do since you wouldn't use them prior to or during importing to Lightroom.
As far as evidence goes, I can only refer you to Peter Krogh's book, which verifies what I said, and to the description of Marc Rochkind's ImageIngester (http://basepath.com/new/detail-ImageIngester.php) which states very clearly that it uses the DNG converter to verify RAW files. I don't know what other evidence you would like. I suppose you could contact Adobe directly if you don't believe me, Peter or Marc.