Data Storage on HDD's - Longevity Question

[jvora]

Hello :

Once data is saved on a HDD, what is the estimated time before any of the data become in inaccessible/unreadable ?

Is it possible to use HDD to archive data for long term ?

Of course, the HDD would have to be stored in a appropriate environment with as little magnetic interference as possible.

My question stems as there are Docking Stations available that would permit one to swap HDD like one does DVD/CD - For example, http://newertech.com/products/voyager.php is one make.

As much as DVD are used for archiving, they are slow for saving and the retrieving of data - Plus they multiply in the hundreds in a rather short time specially when copies of the data are made - Maybe HDD here could help !

Any thoughts ?


Jai

[DarkPenguin]

I would check with Seagate or WD.  They should have white papers indicating the expected life span.  But if you bought a few large drives you could just rotate through them.  Errors should eventually become obvious.

Seagate also has a dock/disk setup that doesn't look too bad.  It does the same thing in a better form factor for a lot more money.

[John.Murray]

Great article:

http://en.wikibooks.org/wiki/Minimizing_ha...e_and_data_loss

and who better to discuss hard drive failure then google?

http://research.google.com/archive/disk_failures.pdf

hth - John

[jvora]

Thanks guys - I am downloading the pdf doc and heading to the other link !


Jai

Great article:

http://en.wikibooks.org/wiki/Minimizing_ha...e_and_data_loss

and who better to discuss hard drive failure then google?

http://research.google.com/archive/disk_failures.pdf

hth - John

[Panopeeper]

Thanks guys - I am downloading the pdf doc and heading to the other link !

Totally useless excercise.

The longevity of the signals on the disk surface is really irrelevant, as the practical problems are not that you lose a bit. Rather, the electronics or the mechanics of the drive can fail much sooner than any signal deterrioration. This means, that you can throw away all the hundreds of gigabytes, except if you are preapered to pay generously for the effort of saving some of it, while there is no guarantee that anything can be saved.

There is NO substitute for multiple savings on different medium.

[Jack Flesher]

Totally useless excercise. ~~~ Rather, the electronics or the mechanics of the drive can fail much sooner than any signal deterrioration.

Precisely.  And why I cycle new drives into my main system every 18-24 months or so, then cycle the older generation to perform tertiary back-up until the next string of replacements comes along. I also periodically cycle the tertiary back-up to make sure they are operable.  Anything older than 2 generations ultimately gets drilled (faster than erasing) and recycled.

[Panopeeper]

Some more things:

use external hard disks for archival, and disconnect them immediately after writing.

This not only increases the lifetime, but protects it from

- power surge,

- malicious software.

Furthermore, keep in mind, that a power surge from a lightning can easily jump over turned off connections, pass the transformer and the fuse as well. Real protection means unplugging, not only turning off, at least in areas, where lightning is a real risk.

I am saving everything what I find important on two external hard disks and on DVD+R.

[Ray]

I'm amazed at how lucky I seem to be with my archived images on optical media. I've still got lots of CD-Rs full of scans I made about 10 years ago, which I haven't bothered transferring to DVD or external hard drive because the images are not important; for example, comparisons between Sigma and Tamron lenses made on my old Minolta before I switched to Canon, or early scans with my first scanner, the Nikon LS-2000, which have since been superceded by more skillful scans made on a better scanner.

I never come across images which can't be read. I wonder if I'll be repeating this message in 20 years time.

However, I wouldn't like to give the impression that I've never had the experience of not being able to read a particular disc with a particular CD/DVD player for reasons other than lack of longevity. Sometimes a disc has simply not been recorded properly or there is an incompatibility issue with the CD/DVD reader.

I'm referring to discs that I know were readable immediately after recording. They're all still readable, as far as I know. I check one ot two now and again just out of curiosity.

[DarkPenguin]

I'm referring to discs that I know were readable immediately after recording. They're all still readable, as far as I know. I check one ot two now and again just out of curiosity.

Well then you're lucky.  I've lost some of my favorite shots when my backup media failed.  Now I use multiple external HDs and Mozy Home.

[John.Murray]

What really hasn't been answered here is the archive life of the magnetic patterns on the unpowered hard disk media itself.  From what I've read, "maybe 5 years" . . . .  others give longer estimates - unfortunately everything is anecdotal.  The 5 year estimate implies for me the need to refresh, or copy the archive at least every 2 years.  I also keep two archives; now organizing my data to match my chosen Lightroom based workflow, this means a disk to disk copy of data for a given archive pair every year - I'm currently using 250GB as my "standard" archive size, exporting LR catalogs from my working storage.

While powered up, lightning, malware, etc, are certainly a threats - probably swamped by static discharge, or getting dropped when unpowered.  I recently recovered a customer's external hard drive that was inadvertantly plugged into an 24 volt laptop external supply instead of the correct 12 volt supply (same connector) .  The onboard controller on the drive was destroyed.  After locating and replacing the controller, the data on the drive was fine . . . .

[giles]

What really hasn't been answered here is the archive life of the magnetic patterns on the unpowered hard disk media itself.  From what I've read, "maybe 5 years" . . . .  others give longer estimates - unfortunately everything is anecdotal.

And unfortunately will remain so IMHO: it's not terribly interesting to do a scientific study of a particular technology (e.g. circa 2008 disk drives) and report in 2018 when if magnetic disks are still in use, they'll have undergone 2-4 generations of technical evolution.  So anecdotes are the best I expect to see until storage technology stabilises, which it shows no sign of doing just yet!

Giles

[dalethorn]

I've been using external HDD's for many years with 100 percent success.  Given the tremendous coercivity that modern drives have, and the fact that airport scanners don't affect them, the only question is whether the permament lubricants in the mechanism will last 10, 20 or more years.  I suspect they will, since they're not the liquid/grease variety used in automobiles.

My advice would be:
1. Get the best quality drives - Toshiba would be my first pick.
2. Get the highest capacity available.
3. Get USB only, for maximum compatibility.
4. Make sure to use the FAT32 format, or whatever is assured to be the lowest common denominator.
5. Verify your copies to the external drive, i.e. after copying, dump the computer's system cache and content-DIFF the external files with the originals.

Lastly, if you're making only one archive copy of your files, you're in trouble before you begin.  You'll need each file to be on at least two separate external HDD's.

[giles]

I've been using external HDD's for many years with 100 percent success.  Given the tremendous coercivity that modern drives have, and the fact that airport scanners don't affect them, the only question is whether the permament lubricants in the mechanism will last 10, 20 or more years.  I suspect they will, since they're not the liquid/grease variety used in automobiles.

My advice would be:
1. Get the best quality drives - Toshiba would be my first pick.
2. Get the highest capacity available.
3. Get USB only, for maximum compatibility.
4. Make sure to use the FAT32 format, or whatever is assured to be the lowest common denominator.
5. Verify your copies to the external drive, i.e. after copying, dump the computer's system cache and content-DIFF the external files with the originals.

Lastly, if you're making only one archive copy of your files, you're in trouble before you begin.  You'll need each file to be on at least two separate external HDD's.

Here's my additional 2c, as devil's advocate in a couple of places, although anyone who's bored and follows 1-5 above will be in pretty good shape.

1. highest quality drives -- all the manufacturers have lemon models now and then.  So I'd buy a mix of brands where possible.  Hitachi and Seagate have both been good, historically; Western Digital has been good in the last few years.  Don't know about Toshiba (who?).  Samsung are a bit the new kid on the block, and I haven't heard anything bad but they're not selling into the market where I would hear much.  Fujitsu are also a minor player.

2. highest capacity -- actually, I'd be careful on this one.  The highest capacity drives sometimes contain the newest technology, which is great if it works and not so great if the teething problems haven't been sorted.  I suggest comparing $/GB and thinking about the second largest capacity.  (e.g. right now for notebooks there are 500GB drives, but at least one manufacturer's are a non-standard height; 320GB 2.5" drives are "safe" to buy without a lot of research.)  All that said, I am taking two 500GB external drives as part of my Antarctica kit.

3. USB2 is definitely the lowest-common-denominator interface right now, so yeah.  But beware stringing too many drives with USB hubs; not all hubs are as good as one might wish.  The *drives* inside your USB enclosure should be SATA, and if the enclosures you choose offer eSATA for only a small additional cost, that's insurance, especially if you have access to desktop systems with unused SATA interfaces.

4. FAT32 ... dunno.  Yeah, maybe.  "It depends."  If I were going to leave a drive for years, yes.  But I wouldn't.  I'd be copying the data every few years minimum, and so personally I use OS X's native file system.  For me there's enough safety that I have multiple machines and I travel with people who also use Macs.  Good enough.

5. When verifying your external copies, calculate a checksum.  Store the checksum with the file.  The checksum allows two things: firstly verification that you can still read all files on the drive correctly, and secondly as a tie-breaker if your multiple drives don't agree what the data is that they've stored.  (Yes, I have had a drive fail with silent data corruption: I'd rather see black smoke.  Data corruption is bad, unreported data corruption is worse.  Being able to verify my files was good.)  [So if I've done this for my audio data, I'd better get on to it for my photography files, right?  On it ...]

Cheers,

Giles

[Ray]

Well then you're lucky.  I've lost some of my favorite shots when my backup media failed.  Now I use multiple external HDs and Mozy Home.

Have we had this conversation before? Two questions. (1) Before your back-up media failed, are you certain that the disc or individual images actually were once  readable, that is, recorded properly? (2) If you are sure the images were once readable, did you try opening them in another and different brand or model of CD/DVD drive on a different computer system?

[OldRoy]

Interesting to note that unless I recall incorrectly (read the thread a couple of days ago) nobody has suggested RAID (chose mode) or tape drives as a safe storage mode. Obviously neither inherently addresses the issue of off-site copies but I wonder how many people actually store backups off-site: not many I'd guess.
On the HD failure question I can't recall how many HD's I've owned since my first (a 20Mb unit in a Mac SE in about '86) but I've never had a total failure on one. In my previous working environment I have trouble recalling HD failures too, although that may be a case of faulty soft memory. However I recall numerous cases of missing data when automated backup strategies which everyone assumed "just worked" were called upon to produce the goods. In one case (major newspaper group) the last good backup was actually several months out of date. Something had been reconfigured and... but that's large companies for you.
I'd say that the working data set plus two HD backups, with one slightly out of step but off-site and periodically updated would cover most non-mission-critical situations. Then maybe periodic optical disc copies if cheap and capacious enough. Personally I trust optical storage media less than HDs. The idea of "cloud" based backups is interesting I think.
FWIW
Roy

[Farmer]

Interesting to note that unless I recall incorrectly (read the thread a couple of days ago) nobody has suggested RAID (chose mode) or tape drives as a safe storage mode. Obviously neither inherently addresses the issue of off-site copies but I wonder how many people actually store backups off-site: not many I'd guess.
On the HD failure question I can't recall how many HD's I've owned since my first (a 20Mb unit in a Mac SE in about '86) but I've never had a total failure on one. In my previous working environment I have trouble recalling HD failures too, although that may be a case of faulty soft memory. However I recall numerous cases of missing data when automated backup strategies which everyone assumed "just worked" were called upon to produce the goods. In one case (major newspaper group) the last good backup was actually several months out of date. Something had been reconfigured and... but that's large companies for you.
I'd say that the working data set plus two HD backups, with one slightly out of step but off-site and periodically updated would cover most non-mission-critical situations. Then maybe periodic optical disc copies if cheap and capacious enough. Personally I trust optical storage media less than HDs. The idea of "cloud" based backups is interesting I think.
FWIW
Roy

RAID is not a backup.  RAID is for uptime and convenience.  In fact, RAID is bad for backup because unless you have the correct controller and get the disk order correct (some controllers don't care, but most will) then you can't access the data - certainly not easily.

My system is pretty straight forward.  I have a NAS RAID 5 as my working copy so I can access the data from anywhere on my network, or offsite if I like.  That is backed up to a single eSata drive which is rotated with a single eSata drive offsite.  The single drives can be ready by any system (and in the cases they're in, either eSata or USB) or they can be taken out and connected internally.

That gives me 1 copy with redundancy and high access and availability and 2 backup copies with universal ease of connectivity and 1 of them offsite.

It's easy, reliable, effective and safe.  The RAID could easily be replaced with a single drive or a more simple mirror and doesn't need to be on a NAS.

[giles]

Interesting to note that unless I recall incorrectly (read the thread a couple of days ago) nobody has suggested RAID (chose mode) or tape drives as a safe storage mode.

RAID is about availability, not backup or archiving.

Tapes for the sort of data volumes we're talking about are not cost effective, and tapes still need to be read and re-written periodically both to keep them "good" and to account for changing tape technologies.  I used to use tapes, but that was a long time ago, expensive even then, and maxed out at 4-8GB (for the DDS2 tape drive I was using), so I shifted to optical media and am now shifting to external hard disks.

And yes, I do have offsite storage, as does another regular poster here -- we live ~15km apart and trade cupboard space

Giles

[ Edited for clarity. ]

[Jack Flesher]

RAID is about availability, not backup or archiving.

Not that it matters but...

RAID was originally about performance with increased storage capacity and the concept of RAID-0 was born in the late 80's.  Of course, those older generations of drives were pretty unreliable, so they implemented mirroring or RAID 1. After that, various redundancy strategies using a dedicated parity drive (2,3 and 4) were devised until ultimately the spread parity of RAID-5 was conceived, offering speed, redundancy and easy maintenance with faster on-the-fly repairability.  RAID 6 and 7 were offshoots of 5 that further spread redundancy.  Originally RAID stood for Redundant Array of Inexpensive Disks, but it was anything but inexpensive, so the "Inexpensive" got re-coined to "Independent."  Most Geeks I know that had to maintain arrays, claim that RAID is really RASD -- Reliable Array for Sh***y Disks  

Today, probably the optimal arrangement if money is of no concern, is a RAID 0 array for performance *backed up* to a RAID 5 array of equal size, AKA RAID 0+5.  A less expensive strategy and essentially as good would be RAID 0+1, but the "1" side can get complicated as the 0 array gets large, so you end up with RAID 0 plus a multi-disk or JBOD array for the *back up* mirror.  

My main point is that since RAID *can be* redundant if you chose the proper drive arrangement, by default that makes it a viable back-up or archive schema...    

Cheers,

[Farmer]

My main point is that since RAID *can be* redundant if you chose the proper drive arrangement, by default that makes it a viable back-up or archive schema...

The main danger is that people without too much experience or knowledge assume that because they have RAID they have backup and that's not necessarily the case.  Even using RAID 6 you could lose 3 drives or lose the controller or accidentally delete the contents or format it or corrupt it and so forth.  That's not backup.  Backup is *another copy* and preferably both on and offsite for real protection.

[Roy]

For my needs, backup should protect against equipment failure, human error, fire, theft, and natural disaster. The key is a frequently refreshed, off-site, off-line copy, and workflow that ensures there are always two or three copies of new data on-site until a copy goes off site.