Bit rot :(

[projectsbin]

Today I discovered two (out of ca. 10000) RAW photos corrupted in my library. I am very concerned with that. I use Mac and Apple (actually Toshiba) 512GB SSD.

I was able to recover them from an old backup, but that is not the point.

With no sophisticated error checking and correction it is very inefficient to manage big data sets and backups (damaged file where actually present in many backups).

How do you manage files to minimize the risk of silent data loss?

[francois]

…
How do you manage files to minimize the risk of silent data loss?

Very simply, I use multiple backups…

[projectsbin]

Maybe I was not specific enough: I have multiple backups and in 80%+ of them the file is present already in corrupted state. So the question is if there is maybe any checksumming app to do periodical disk checkups?

I am really intrigued by why Apple and other producers did not decide to use ECC memory and ZFS file system to be able to recover from that sort of problems. My cheap server has both.

[francois]

Maybe I was not specific enough: I have multiple backups and in 80%+ of them the file is present already in corrupted state. So the question is if there is maybe any checksumming app to do periodical disk checkups?

I seem to remember that Lloyd Chambers developed some Mac software to  to test data integrity. I've no personal experience with his software, though.

diglloydTool

I am really intrigued by why Apple and other producers did not decide to use ECC memory and ZFS file system to be able to recover from that sort of problems. My cheap server has both.

Sorry for my simplistic answer…

Mac Pros use ECC memory. ZFS has fallen out of favour a long time ago… I've heard different conflicting stories about it but in the end the "why" is not important anymore; an Apple supported version of ZFS is very unlikely. FWIW, GreenBytes offer a free community edition of ZFS on the Mac.

[projectsbin]

Francois,

Thank you for your informative reply. I will search for some data integrity testing tool.

Unless there is a chain of memory and media ECC, there will happen bit flips in a "normal scale" periods.

So, multiple backups most probably (that is just very inefficient). I don't know how much my "unlimited" cloud storage of choice provider will like that...

[jrsforums]

Francois,

Thank you for your informative reply. I will search for some data integrity testing tool.

Unless there is a chain of memory and media ECC, there will happen bit flips in a "normal scale" periods.

So, multiple backups most probably (that is just very inefficient). I don't know how much my "unlimited" cloud storage of choice provider will like that...

The problem we all have is if the corruption...or even deletion...accidently starts at the top of the backup chain or hierarchy.  The only solution is mutilple layers of backup, where older versions are not overwritten, but save in an archive.

Goodsync (PC & Mac, I believe) has that capability..

[Bart_van_der_Wolf]

Francois,

Thank you for your informative reply. I will search for some data integrity testing tool.

There are probably many tools that generate CRC numbers or hash codes for any input file, and some allow to compare against a database you can build with earlier generated code keys. However, you can also use a file synchronization/Backup tool (e.g. FreeFileSync) to compare files based on file content. That will prevent the need to try if files are readable, and the reading action for comparing the checksums will trigger the hard disk sector readability check which might relocate a sector when the sectors get marginal for some reason.

Another possibility is to use an archive program to produce a compressed (if possible) version of your file. The archiving program (e.g. 7Zip), allows to only test the integrity of the archive, without the need to open the file(s). Do note that some files grow when trying to compress them, but there are often parameters that can tweak the compression method to avoid that. an added benefit is that one can compress/archive multiple files that form a set, just watch out for archives that grow too large. A detected error may render all files inside unrecoverable. A benefit is that you can password protect each archive.

Cheers,
Bart

[RobSaecker]

Francois,

Thank you for your informative reply. I will search for some data integrity testing tool.

ImageVerifier: http://basepath.com/site/detail-ImageVerifier.php

Also available on the App Store.

[sty]

ZFS is currently the only production quality filesystem that can detect bit-rot and heal the data on the fly, when you open the file. This in conjunction with ECC memory gets you as robust system as possible. But then of course if there's something between the zfs fs and your editing machine (network perhaps?) corruption can easily happen somewhere else.

I'm running my lt4 catalog on my pc and all the raws are on zfs server, works very fine and monthly 'scrub' run of the data usually shows 1-2 bit flips in images which get corrected. Cosmic rays for sure!

And backups of course.

But the proper way to do it on mac:
(The stable is too old, pool version 8. Developement branch is not scary )
https://code.google.com/p/maczfs/wiki/DevelopmentOverview

[terence_patrick]

I have also experienced bit-rot, on some old SATA drives stored in a safe, but in my case the images I lost were not that important but it was still upsetting. I started investigating solutions, but realized it might be a massive time and money hole and just do my best to make multiple backups and worry more about shooting new work and less on trying to preserve some outtakes from the past. I've had a habit now of maintaining a separate drive of just "hero" images that also get stored in the cloud. Outtakes and non-client work just get the routine triple backup with fingers crossed.

[Bart_van_der_Wolf]

I have also experienced bit-rot, on some old SATA drives stored in a safe, but in my case the images I lost were not that important but it was still upsetting. I started investigating solutions, but realized it might be a massive time and money hole and just do my best to make multiple backups and worry more about shooting new work and less on trying to preserve some outtakes from the past. I've had a habit now of maintaining a separate drive of just "hero" images that also get stored in the cloud. Outtakes and non-client work just get the routine triple backup with fingers crossed.

Hi,

Don't forget to periodically re-write the data (e.g. copy to new disk), because the magnetism slowly fades away.

Cheers,
Bart

[wolfnowl]

Lightroom 5 also has a 'validate DNG' option that almost nobody mentions...

http://thedambook.com/dng-verification-in-lightroom-5/

Mike.

[francois]

Lightroom 5 also has a 'validate DNG' option that almost nobody mentions...

http://thedambook.com/dng-verification-in-lightroom-5/

Mike.

I guess that this useful feature will be mentioned in Lu-La Guide to Lightroom 5.

[Bart_van_der_Wolf]

Lightroom 5 also has a 'validate DNG' option that almost nobody mentions...

Hi Mike,

Besides having to create DNGs, it apparently only looks at the raster image data itself. That still means that if the DNG file itself becomes corrupted, it might be impossible to access the file as a whole (let alone the raster image data within). So while a nice idea, it doesn't cover all potential issues.

IMHO the best test is still with a checksum on all bits, which will ensure a bit-by-bit identical file. That's especially crucial with Raw files, which after all contain our original source data. A utility that allows to compare files bit-by-bit, stored at two or more locations, will flag issues with an extremely high probability. Good synchronization or backup tools allow to do such time consuming checks in the background and at a user definable interval.

Another example of such a time proven tool is SyncBack. Even the free version allows to do a more thorough test for file integrity, and also offers a slew of other possibilities.

Cheers,
Bart

[jonathanlung]

I'm relatively small-fry when it comes to backups, but ZFS got mentioned here a few times

D800 shooting to SD/CF (backup arrangement). Ingestion into primary machine (a Mac laptop) and displayed to check for problems. Images then transferred after processing to a computer running Debian with ZFS; each ZFS pool is replicated onto at least two disks. CF cards read via camera if/when an error is detected with the SD card (so far unnecessary -- unless I've been experiencing silent corruption). After comparing SD card contents and copy on ZFS, the SD cards and CF cards are formatted. The Debian machine is only used for storage and is usually offline. Scrub once a month. Only one of the seven drives that it's had has reported errors so far (and then the drive died).

ZFS snapshots used to reduce the likelihood of human error wiping out everything.

Haven't figured out how to economically and quickly run off-site backups; currently using sneaker-net for large files and git for smaller data.

Fortunately, the one time I experienced the need for off-site backup (a fire), a then-mid-sized hard drive could comfortably store everything that needed backing up (and more). Nowadays, D800 NEFs are far too unwieldy (as are the corresponding multi-gigabyte layered edits) for the lowly likes of me.