Pages: [1]   Go Down

Author Topic: Backups and data rot  (Read 7168 times)

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Backups and data rot
« on: January 07, 2013, 04:43:43 pm »

Morning All,I am interested in if and how people check for data rot amongst their files on drives and disks etc.

I ask as I recently suffered the following problem.
My pc is a 3yr 12gb ram, upgraded yesterday from a 120gb ssd to 500gb ssd OS os +programs C: drive.
A 120 gb ssd scratch disk for windows and Photoshop.
My D: drive consists of 4 x 1 tb WD red Nas drives onboard in raid 1+0 giving approx 1.86 gb. These drives replaced 4 x 3yr old samsung drives in the same configuration approximately a month ago.
I also have a 2yr old WD NAS 4 Tb Sharespace configured as raid 5. I use this as a backup target.
I have a 1.5tb external Seagate drive I use as a backup target as well.

My C: drive is my working drive, when files are finished processing, I transfer them to my master archive on the D: drive.

I run Memeo Backup premium. I run two identical plans apart from the backup destinations. I also do blu ray copies of 25gb folders in the master archive. I brought back copies of my files after I replaced the samsung drives with new WD Nas drives a month or so ago. I did have some problems with non essential files being restored. I am in contact with Memeo re this as a support request.

Now, after upgrading the C: drive yesterday, giving the system a clean copy of Win7 ultimate and replacing programs, my on board raid D: drive developed a fault. One drive was reporting an error and the raid was reporting itself as degraded. After pulling cables on at a time to isolate the individual drive (they are now labelled as to physical position) I found eventually the drive disappeared from the setup. After replacing cables and reseating them all properly, the drive was found and I could get it seen fine with no errors being reported in the bios or the raid manager. After booting into windows, the raid manager reported it was rebuilding the volume with my archive on it. After several hours, it reported a success! I suspect I may have loosened a cable whilst moving things in the case during the ssd installation.

One thing that struck me the day before the ssd was replaced though was I was tinkering with LR4 and happened to noticed an image in my catalog suffering data rot. When I looked at the image with an external editor, the rot was still there. When I brought the image back from both backups the rot was still there. So, what to do? The image was not important, it was one a friend had given me of a wedding he shot and we were using it for testing only.

The important question is though, How do I test the others for data rot? If they are changing due to it and then being backed up what to do? I have the backup software keep old versions as well but this was in them as well.

Is there any software that can scan for this sort of corruption?

Logged
Cheers

Onslow

RobSaecker

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 300
    • robsaecker.com
Re: Backups and data rot
« Reply #1 on: January 07, 2013, 07:32:27 pm »

The important question is though, How do I test the others for data rot? If they are changing due to it and then being backed up what to do? I have the backup software keep old versions as well but this was in them as well.

Is there any software that can scan for this sort of corruption?



ImageVerifier, by Marc Rochkind.

http://basepath.com/site/download.php?prod=ImageVerifier
Logged
Rob
photo blog - http://robsaecker.com

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Re: Backups and data rot
« Reply #2 on: January 07, 2013, 07:59:50 pm »

ImageVerifier, by Marc Rochkind.

http://basepath.com/site/download.php?prod=ImageVerifier

Thanks for that Rob... I'm having a read right now. :)
Logged
Cheers

Onslow

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: Backups and data rot
« Reply #3 on: January 07, 2013, 08:28:40 pm »

ImageVerifier, by Marc Rochkind.

http://basepath.com/site/download.php?prod=ImageVerifier

Ditto. Slow but it works. And when you're done, a sigh of relief.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Re: Backups and data rot
« Reply #4 on: January 07, 2013, 08:43:02 pm »

Ditto. Slow but it works. And when you're done, a sigh of relief.

Ah, good. Cheers for that. Hearing something good about it makes me a bit more confident it can assist me. I realise now that data rot on all my backups (not just my images) is turning out to be a concern....

My backup software verifies the reads/writes but as bits change on the raids, who knows if the changes are mirrored onto the mirrored drives? That is how I ended up with that bad file in my archive I suspect.

I have just bought LR and the LuLa videos here (C2P2AS, LR Intro, Adv and Updates, Where are my *&^ Files) I'd like to get this issue sorted before I start setting up my master catalog. I now need to work out how to adjust my workflow to take account of these issues.

Logged
Cheers

Onslow

John.Murray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 886
    • Images by Murray
Re: Backups and data rot
« Reply #5 on: January 08, 2013, 12:26:34 am »

+1 ImageVerifier

Although Memeo looks like a nice product - it's a bit unclear what backup method they are using.  With Online backup solutions becoming increasingly popular, along with the huge increase in backup size - a common strategy being used is disk block based backup, as opposed to the older and more familiar file based backup.

The advantage of block based backups (especially for online backup) is that only those disk blocks (typically 4k) that have changed since the previous backup are recorded - this vastly cuts down the amount of data to be transferred. The issue here is the possibility of one of those blocks being "changed" as the result of some form of corruption.  This can be due media errors, bit also can be the result of an application or operating system bug as well - the bottom line is one block participating in the storage of an image getting corrupted, then being persisted to your backup . . . .

How do you recover from this situation?  Go back to a previous version, of course, which begs the question; how many previous backups do you have access to?

This is exactly why I'm not a fan of online "cloud" backups - they limit the number of previous backup version to one.

Don't mis-understand, I think block based incremental backup is the way to go, especially with huge datasets, but only if you can access all backup versions.  In any case, you should also supplement this with a file based backup from time to time - which would be an ideal time to run ImageVerifier

update:  Upon re-reading your post, I see you are storing backups to a NAS - be aware the backups based on the Windows 2003/Vista/Win7 API are block based and will overwrite previous backups to any network location, including your NAS ....  If you plan on relying on your NAS as a "backup", a file based backup strategy - where the each file gets written across, is your best option...
« Last Edit: January 08, 2013, 12:45:23 am by John.Murray »
Logged

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Re: Backups and data rot
« Reply #6 on: January 08, 2013, 01:19:06 am »

+1 ImageVerifier

Although Memeo looks like a nice product - it's a bit unclear what backup method they are using.  With Online backup solutions becoming increasingly popular, along with the huge increase in backup size - a common strategy being used is disk block based backup, as opposed to the older and more familiar file based backup.

The advantage of block based backups (especially for online backup) is that only those disk blocks (typically 4k) that have changed since the previous backup are recorded - this vastly cuts down the amount of data to be transferred. The issue here is the possibility of one of those blocks being "changed" as the result of some form of corruption.  This can be due media errors, bit also can be the result of an application or operating system bug as well - the bottom line is one block participating in the storage of an image getting corrupted, then being persisted to your backup . . . .

How do you recover from this situation?  Go back to a previous version, of course, which begs the question; how many previous backups do you have access to?

This is exactly why I'm not a fan of online "cloud" backups - they limit the number of previous backup version to one.

Don't mis-understand, I think block based incremental backup is the way to go, especially with huge datasets, but only if you can access all backup versions.  In any case, you should also supplement this with a file based backup from time to time - which would be an ideal time to run ImageVerifier

update:  Upon re-reading your post, I see you are storing backups to a NAS - be aware the backups based on the Windows 2003/Vista/Win7 API are block based and will overwrite previous backups to any network location, including your NAS ....  If you plan on relying on your NAS as a "backup", a file based backup strategy - where the each file gets written across, is your best option...

Hi John, I am also John Murray. My screen name though is Onslow after 2 other forums also had John Murrays.... Ah well. We must be common... I know there are 5 John Murrays in my town I'm aware of.

I use the Nas and the external HD as online backup targets. I also have 2 offline HDs that rotate through. Unfortunately, the rot error is there as well. Blurays are ok but time consuming. In my backups, I have access to 2 versions. I am now considering buying a few disks and rotating them through once a week. Hopefully, this may limit damage to only the most recent.....

I have around 10,000 images in my archive I wish to keep plus around another 10,000 I'm slowly adding to the archive.

Cloud based storage to me is ok for small data sets but not for my purposes....

Logged
Cheers

Onslow

Peter F.

  • Newbie
  • *
  • Offline Offline
  • Posts: 4
Re: Backups and data rot
« Reply #7 on: January 08, 2013, 05:18:15 pm »

I have used Memeo and Acronis. I found neither one very satisfactory; Memeo did not seem robust enough to copy multiple terabytes and Acronis is buggy. I have since switched to Macrium Reflect Professional v.5 which I would recommend. I have restored files that are 8 years old without any problems. You should definitely be validating your backups periodically using the backup software (you may be doing this already.) I use a Netgear ReadyNAS Pro with 8 1Tb disks in RAID 10 and the RAID helps resolve disk corruption issues.
Logged

andyptak

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 469
Re: Backups and data rot
« Reply #8 on: January 08, 2013, 06:28:21 pm »

Peter - care to give more information about Macrium? Not too much info regarding operation or deployment on their site. Thanks
Logged

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Re: Backups and data rot
« Reply #9 on: January 09, 2013, 05:09:41 pm »

Peter - care to give more information about Macrium? Not too much info regarding operation or deployment on their site. Thanks
+1 on this.

I am reluctant to change my backup software until I can be as sure as I can be about the new products improvements over my current solution. Do you have any more information Andy?
Logged
Cheers

Onslow

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Re: Backups and data rot
« Reply #10 on: January 09, 2013, 05:29:08 pm »

+1 ImageVerifier

Although Memeo looks like a nice product - it's a bit unclear what backup method they are using.  With Online backup solutions becoming increasingly popular, along with the huge increase in backup size - a common strategy being used is disk block based backup, as opposed to the older and more familiar file based backup.

The advantage of block based backups (especially for online backup) is that only those disk blocks (typically 4k) that have changed since the previous backup are recorded - this vastly cuts down the amount of data to be transferred. The issue here is the possibility of one of those blocks being "changed" as the result of some form of corruption.  This can be due media errors, bit also can be the result of an application or operating system bug as well - the bottom line is one block participating in the storage of an image getting corrupted, then being persisted to your backup . . . .

How do you recover from this situation?  Go back to a previous version, of course, which begs the question; how many previous backups do you have access to?

This is exactly why I'm not a fan of online "cloud" backups - they limit the number of previous backup version to one.

Don't mis-understand, I think block based incremental backup is the way to go, especially with huge datasets, but only if you can access all backup versions.  In any case, you should also supplement this with a file based backup from time to time - which would be an ideal time to run ImageVerifier

update:  Upon re-reading your post, I see you are storing backups to a NAS - be aware the backups based on the Windows 2003/Vista/Win7 API are block based and will overwrite previous backups to any network location, including your NAS ....  If you plan on relying on your NAS as a "backup", a file based backup strategy - where the each file gets written across, is your best option...
Hi John,
Whilst the Nas is a backup target, I also have another external HD as a target. My Memeo product currently backs up 2 plans. One plan for each target. Identical backup sets.

I am having problems with the software which is reported to them as a support request. I have general errors when restoring data. They seem to be scattered through the recovery files selection. The files are physically on the targets and I can restore them manually, however, this is painful when restoring individual files scattered throughout the archive.
I'm not particularly happy with it but as I said in an earlier post, I'm reluctant to jump ship unless the fix is better than my current one. Macrium was recommended below and the website looks enticing, but I'm not sure yet.
Logged
Cheers

Onslow

DougJ

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 73
Re: Backups and data rot
« Reply #11 on: January 09, 2013, 10:38:49 pm »

One thin worrying in the thread and it is my impression from the thread that NAS storage is being viewed as a backup solution. 

I do believe that ther is a wealth of expert opinion to the effect that NAS is about redundancy and backup to tape/CD/DVD/additional HD is backup.

In my case, my image files are on a HD devoted to images as part of my desktop setup.  These are synched to my NAS, and the NAS is copied over to external USB HDs.  To be clear, there are three copies of every image file.

Ciao,

Doug

Logged

Onslow

  • Full Member
  • ***
  • Offline Offline
  • Posts: 208
Re: Backups and data rot
« Reply #12 on: January 10, 2013, 12:10:18 am »

One thin worrying in the thread and it is my impression from the thread that NAS storage is being viewed as a backup solution. 

I do believe that ther is a wealth of expert opinion to the effect that NAS is about redundancy and backup to tape/CD/DVD/additional HD is backup.

In my case, my image files are on a HD devoted to images as part of my desktop setup.  These are synched to my NAS, and the NAS is copied over to external USB HDs.  To be clear, there are three copies of every image file.

Ciao,

Doug



G'day Doug,
My Nas is a backup target, as is my external HD and several others. Multiple copies of the files exist. I'm worried though about rot in the archive and spreading through the backups as it has in one example I have found.
Logged
Cheers

Onslow

DougJ

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 73
Re: Backups and data rot
« Reply #13 on: January 11, 2013, 01:49:31 am »

Hi Onslow,

Can't offer any useful comment on the rot you are chasing.  I did want to offer a more general warning that if a NAS box is used as the only form of back-up and the software on the NAS fails for whatever reason, even if that has not led to corrupted file structures, then the chances of recovering your files are very small.

Ciao,

Doug

Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com
Re: Backups and data rot
« Reply #14 on: January 11, 2013, 03:34:49 pm »

Thanks Rob, checking it out now.
Logged

RobSaecker

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 300
    • robsaecker.com
Re: Backups and data rot
« Reply #15 on: January 12, 2013, 12:46:53 pm »

The ultimate answer for this is something like ZFS, where everything on the disk is checked for integrity. But until that comes to mainstream file systems, IV is reasonable substitute.
Logged
Rob
photo blog - http://robsaecker.com

Walt Roycraft

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 442
    • roycraftart
Re: Backups and data rot
« Reply #16 on: January 23, 2013, 08:20:50 am »

Has anyone used digiloyd's Integrity Checker?
Logged
Walter Roycraft
http://www.roycraftart.c

jonathan.lipkin

  • Full Member
  • ***
  • Offline Offline
  • Posts: 158
Re: Backups and data rot
« Reply #17 on: January 25, 2013, 10:33:20 am »

Image Verifier is great for tiff, jpeg and dng, but will not verify hasselblad fff.
Logged

dreed

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1715
Re: Backups and data rot
« Reply #18 on: January 25, 2013, 10:45:32 am »

...
One thing that struck me the day before the ssd was replaced though was I was tinkering with LR4 and happened to noticed an image in my catalog suffering data rot. When I looked at the image with an external editor, the rot was still there. When I brought the image back from both backups the rot was still there. So, what to do? The image was not important, it was one a friend had given me of a wedding he shot and we were using it for testing only.

The important question is though, How do I test the others for data rot? If they are changing due to it and then being backed up what to do? I have the backup software keep old versions as well but this was in them as well.

Is there any software that can scan for this sort of corruption?

The problem is, where did the rot occur?
Did it occur when you imported and didn't notice it?
Did it occur during the rebuild of the RAID?
Did it occur when you created the backup?

The only times that I've noticed bit rot occurring is on import. Not all SD/CF card adapters have good electronics for large data transfers. It stands to reason that other USB interfaces for hard drives may also suffer but they'd quickly sink in reviews if so.

What's the solution to this?

Embedding a checksum of the file in the file itself. To my knowledge, only the most recent DNG file format supports this but not as fully as it should. In that way, the camera can embed a checksum into the image when it creates it (before writing to card) that everything else van verify. No amount of fancy filesystem tricks (ZFS, ReFS, RAID) is going to save your image if the image doesn't get onto your computer correctly in the first place.
Logged
Pages: [1]   Go Up