Hard earned advice against RAID...

[PatrikR]

For long term, tape is arguable the best, and most reliable solution.

[{POST_SNAPBACK}][/a]

Hi,

What kind of tape are you using? LTO, VXA, AIT, DAT or what? I called HP and they recommended LTO tape systems for longterm storage, like 30 years, but wouldn't confirm or know if their drives would work with Macs. The LTO is reliable and fastest tape and most popular in servers they told.

The VXA seems very durable but slow. There's a demo where they cook, freeze and dunk VXA tapes into hot coffee and so on with 100% recovery. Impressive. [a href=\"http://www.exabyte.com/technology/tested/index.cfm]VXA[/url]

Do you use the tape on a Mac or PC? Most professional tape drives are SCSI or SAS... Since there are no LTO kits for Mac does anybody know if they work with Mac Pro if I buy a $600 Atto Scsi card?

These investments are just growing and growing. Now I'm wondering if I need to invest almost $4000 US to get a tape drive, scsi cards, software and tape media to back up my drives. This is getting very expensive when it seems that there's always need for new equipment. But back ups are a must.

Patrik

[Ray]

I don't trust DVD's or CD's, I did backups on them a few years ago and although they are still being read you don't want to see the error logs
[a href=\"index.php?act=findpost&pid=146296\"][{POST_SNAPBACK}][/a]

Are you suggesting, Frank, that those errors degrade the image in any visible way?

I'm surprised that after 10-12 years of backing up thousands of images on CDs and DVDs, I've never had any major problem retrieving data, even from the very earliest CDs, many of which were non-descript, bargain blanks.

Stories of unreadable discs due to bit rot seem very odd to me. When people have trouble like that, I can't help wondering what the circumstances really are. Perhaps they've used labels with corrosive glue. Perhaps they've stored the discs in a room with other corrosive chemicals. Perhaps they happen to live in a heavily industrialised, polluted city like Beijing. Perhaps they accidentally left the unreadable disc(s) in a parked car to bake in the sun, or perhaps more likely, the unreadable disc(s) were never recorded properly in the first instance.

I've had a number of hard drive failures during that 12 year period, but not one CD or DVD failure of a disc which I know was recorded properly in the first instance. I have had trouble, of course, getting a successful recording (haven't we all) and still have a stack of toasters about a foot high.

[rainer_v]

i back up with two discs which i have in seperate locations, and i have backups of all stuff on 2 sets of dvds, also on two places. actual images go in a raid in my g5. on my laptop i copy them manually on external drives if on location.
i trust more in harddrives and dvds. i dont know about the tape storages, but some years ( 10 or so ) ago i stored a lot of ( digital ) music data on DAT backup tapes and nearly one of them was readable some years later. most CD backups of that time stil are readable. i listened that tape is not good for long term because the magnetic information fades away slowly by slowly. also this never was a great problem with analog data, with digital it can become one.

[Frank Doorhof]

Digital = digital so the quality is the same.

The alarm went of when I used an old CD and it took me 20 minutes to read the whole CD while normally that's less than a minute.
Everything was 100% ok.
But when I checked the error correction it was enough to let me destroy my CDs and got the third harddrive which now replaces my DVDs

And to be honest backuping on DVD is a big problem, I shoot in one session between 1-2GB so that would mean ALOT of DVDs
Harddrive space is cheap and very reliable.

[feppe]

Are you suggesting, Frank, that those errors degrade the image in any visible way?

I'm surprised that after 10-12 years of backing up thousands of images on CDs and DVDs, I've never had any major problem retrieving data, even from the very earliest CDs, many of which were non-descript, bargain blanks.

Stories of unreadable discs due to bit rot seem very odd to me. When people have trouble like that, I can't help wondering what the circumstances really are. Perhaps they've used labels with corrosive glue. Perhaps they've stored the discs in a room with other corrosive chemicals. Perhaps they happen to live in a heavily industrialised, polluted city like Beijing. Perhaps they accidentally left the unreadable disc(s) in a parked car to bake in the sun, or perhaps more likely, the unreadable disc(s) were never recorded properly in the first instance.

I've had a number of hard drive failures during that 12 year period, but not one CD or DVD failure of a disc which I know was recorded properly in the first instance. I have had trouble, of course, getting a successful recording (haven't we all) and still have a stack of toasters about a foot high.
[a href=\"index.php?act=findpost&pid=146351\"][{POST_SNAPBACK}][/a]

My experience is the polar opposite: I've had one HDD fail out of 20 (guesstimate) I've used, while I have whole spindles of DVDs which are unreadable. And my HDDs are on 24/365 and in constant use for years. The DVDs are the cheapest I can find, but I store them appropriately.

But that's not the reason why I back up to HDDs. Backing up 100+ gigs of images and 100+ gigs of music to DVDs is about as appealing as pouring battery acid on my eyes. While backing up a few DVDs a week incrementally is ok - ie. if you don't have to start from zero, and/or only backup the latest work on DVDs -, it still leaves verification.

Verifying those 200+ gigs of data every month is a colossal waste of time. With HDDs I can do full bit-to-bit verification each and every time I backup with only a marginal increase in time. And if you don't verify your backups, you're likely to get two shocks when you lose your data: one when you do, and second when your backups are DOA. And test recoveries are much faster, also. HDDs over DVDs as backups are a no-brainer.

[Steve Kerman]

From PatrikR's VXA link:

VXA is the only tape format in the world to write and read data on tapes in individually addressed data Packets.

I have to say that I am absolutely dumbfounded to read that.

Let me tell you a story that I thought was ancient history:

Many years ago I had a QIC tape backup system from Emerald Systems.  To give some idea of the timeframe, the tapes each held 60 megabytes.  I used their tape system to back up the development files of my company.

One day while editing a file, I happened to destroy it.  So, I went to the most recent backup tape to attempt to recover a recent version of the file.

A short way into reading the backup tape, the drive reported an unrecoverable read error.

At which time it stopped.

It didn't continue reading past said read error, which was on a part of the tape that didn't contain my lost file.  It simply stopped, and refused to read anything at all past that first error on the tape.

After some angry calls to Emerald Systems, I was told that the design of the QIC tape standard was such that drives couldn't read past an error.

Frankly, I thought that I was being fed a line of BS, and never again did any business with Emerald Systems.

As an engineer in the microelectronics field, I am absolutely dumbfounded to read that not only was what Emerald said about QIC tapes apparently true, but that this design flaw has continued to exist in the multiplicity of tape standards that have been developed since that time.  It seems to me to be utterly obvious that a backup tape system has to be able to ignore bad spots on the tape and recover all the readable data that is beyond the bad spot.  How this situation could have been designed in in the first place, and then continued to exist for these many years, is stupidity of a level that exceeds my ability to comprehend.


Reviewing what I just wrote, I see that it was quite a rant!  I guess the loss of that data after having spent thousands of dollars on a backup system that didn't work still stings.    I guess the moral of this story is not to trust your precious data to backup tapes that can't be read if there are any bad spots.  Which would appear to imply that VXA tapes are the only tapes which might be suitable.

[Dustbak]

You would not believe how many companies are extremely rigid in backing up every bit of data but have never performed a (full) recovery as a test.

Many of them are in for a real surprise whenever they need to recover. Most of the time separate data is very well doable but when it comes to recovering servers with stuff like active directory or exchange db servers it is a totally different story.

[Monito]

You would not believe how many companies are extremely rigid in backing up every bit of data but have never performed a (full) recovery as a test. Many of them are in for a real surprise whenever they need to recover. Most of the time separate data is very well doable but when it comes to recovering servers with stuff like active directory or exchange db servers it is a totally different story.[a href=\"index.php?act=findpost&pid=146400\"][{POST_SNAPBACK}][/a]

Good point.  Another point in favor of external drives is that one should every few years (5?) copy everything off of the old drives onto the current sweet spot capacity drive, verify the integrity of the data, and then box and put into storage the old drives.

Suppose that two years ago, a 200 GB external drive cost $200.  I wouldn't be surprised if three years from now a 2,000 GB (2 TB) drive will cost $200.  So copy ten of your shelf archive drives onto the 2 TB drive.  Verify.   Get your corresponding ten offsite archive drives and copy them onto a second 2 TB drive.  Verify it and take it and the ten old drives offsite again.

This way you keep the bits and the technology current.  If any one of the 20 drives fails to deliver its precious images, you can save the day with the other copy before you have a duplicate failure ten or 15 years down the line.

[Steve Kerman]

Good point about saving old archives.  I've been through a couple of episodes of going to IT with, "Remember Frank, who left two years ago, who designed the FPGA for the ZZXY system that we're selling million$ of each month?  Well, it seems he didn't properly release the source code to Document Control before he left, so we need to recover the copies that were on his PC, which you archived when he left."  To which IT responds, "Oh, we only keep those past-employee archives for a year; we recycled that tape a long time ago."  

[nicolaasdb]

one word.....backup!!!!!!!!

a raid is not a backup ever!!!!

but  feel your pain!! try to have your drives recovered.. there are companies that can get your info back,,,,but it is expensive

[Ray]

My experience is the polar opposite: I've had one HDD fail out of 20 (guesstimate) I've used, while I have whole spindles of DVDs which are unreadable. And my HDDs are on 24/365 and in constant use for years. The DVDs are the cheapest I can find, but I store them appropriately.
[a href=\"index.php?act=findpost&pid=146374\"][{POST_SNAPBACK}][/a]

That seems very strange indeed. Whole spindles of DVDs that are unreadable? Have you investigated possible causes?

I could appreciate that you might have unwittingly bought whole spindles of reject and faulty DVDs from some unscrupulous dealer, perhaps DVDs that were missing their protective coating.

I could also appreciate that you might have been in a hurry and recorded whole spindles of DVDs without verification of data (which can sometimes take as long as the initial recording process), or that you might have recorded the unreadable DVDs with incompatible burner and/or software which caused the discs to be unreadable on other burners.

I've also experienced inexplicable, apparent unreadability of the occasional disc on other DVD players. It's happened a couple of times that RAW images I've recorded to DVD on my laptop when travelling, have been unreadable on a particular desktop computer back home but perfectly readable on the laptop and also perfectly readable on another desktop computer with a more up-to-date DVD reader/burner.

I've also tried to play the occasional pirated DVD movie (bought in Asian countries - can I say this without incriminating myself?) which would not play well (or at all) on my stand-alone DVD player, but which would play apparently perfectly on my computer, presumably because the computer DVD player and software had better error correction.

It's so easy after reading horror stories about bit rot and general articles about optical media not being as reliable as they're cracked up to be, to make the first assumption when things go wrong that it's the disc itself which is faulty or unrealiable. I could almost bet my bottom dollar that in 9 cases out of 10, the causes of the problems are user error and/or incompatible software and hardware.

So let's be clear about this, Feppe, just for the record. Are you saying that you've got whole spindles of DVDs that you know were initially recorded properly and were readable after recording, on at least one machine, that were placed in proper storage for a period of time and subsequently became unreadable on any of several computers that you tried; that is, unreadable period?

[SeanPuckett]

Images are business data.  Treat it like any other business treats its data -- or should.  If you're not doing what the banks do, you're taking the same risks with your life.

Fast, redundant online storage -- RAID 5 at a minimum.
Reliable nearline storage -- External hard disk backups, done daily.
Permanent off-site storage -- Archived data on hard disks or DVDs somewhere else.

Redundant online storage protects you from hardware failure.
Nearline storage with daily backups protects you from user failure.
Off-site storage protects you from acts-of-god.

There is no alternative; there is nothing less responsible to your future self and business interests than to shirk competent and adequate data protection.  No excuses.

(Guess what industry *I* used to be in.)

[rainer_v]

From PatrikR's VXA link:

I have to say that I am absolutely dumbfounded to read that.

Let me tell you a story that I thought was ancient history:

Many years ago I had a QIC tape backup system from Emerald Systems.  To give some idea of the timeframe, the tapes each held 60 megabytes.  I used their tape system to back up the development files of my company.

One day while editing a file, I happened to destroy it.  So, I went to the most recent backup tape to attempt to recover a recent version of the file.

A short way into reading the backup tape, the drive reported an unrecoverable read error.

At which time it stopped.

It didn't continue reading past said read error, which was on a part of the tape that didn't contain my lost file.  It simply stopped, and refused to read anything at all past that first error on the tape.

After some angry calls to Emerald Systems, I was told that the design of the QIC tape standard was such that drives couldn't read past an error.

Frankly, I thought that I was being fed a line of BS, and never again did any business with Emerald Systems.

As an engineer in the microelectronics field, I am absolutely dumbfounded to read that not only was what Emerald said about QIC tapes apparently true, but that this design flaw has continued to exist in the multiplicity of tape standards that have been developed since that time.  It seems to me to be utterly obvious that a backup tape system has to be able to ignore bad spots on the tape and recover all the readable data that is beyond the bad spot.  How this situation could have been designed in in the first place, and then continued to exist for these many years, is stupidity of a level that exceeds my ability to comprehend.
Reviewing what I just wrote, I see that it was quite a rant!  I guess the loss of that data after having spent thousands of dollars on a backup system that didn't work still stings.    I guess the moral of this story is not to trust your precious data to backup tapes that can't be read if there are any bad spots.  Which would appear to imply that VXA tapes are the only tapes which might be suitable.
[a href=\"index.php?act=findpost&pid=146375\"][{POST_SNAPBACK}][/a]

this is exactly what happened to my DAT backup tapes. fortunately i had also CDs, one of the first you could burn yourself..... some were not working but most dd although i didnt care at al how i strored them, because i thought i have everything save on tapes.

[feppe]

They were not backup disks, so I didn't verify them. The most likely reason is:

I've also experienced inexplicable, apparent unreadability of the occasional disc on other DVD players. It's happened a couple of times that RAW images I've recorded to DVD on my laptop when travelling, have been unreadable on a particular desktop computer back home but perfectly readable on the laptop and also perfectly readable on another desktop computer with a more up-to-date DVD reader/burner.
[a href=\"index.php?act=findpost&pid=146563\"][{POST_SNAPBACK}][/a]

They were readable, but that was with the burner I burned them with. Now I have a new burner, which refuses to read a few spindles - but works on most of the brands burned on the old recorder.

In any case, it doesn't matter. I've given the reasons why HDD kicks DVDs' ass when it comes to backing up. Having to worry about unreadable media on some readers is yet another reason to avoid DVDs.

[Bruce MacNeil]

My 2 cents/

I use a large raid with Aperture.

2 raids really. One for library - one for vault.

When I work on a new project - I export that Aperture project to external bare drives that are kept off-site. These drives use MS-ds file system so I can read them with a Mac or windows machine.

I have experiment on the reliability by:

1) rebuilding library from vault
2) reading raw data from the archive drives with Max and with PC.

Things seem to work and I have everything to date always live.

The raids are 8TB each and I buy 500 GB external drives for about $100.

I also use subsumation.

[Fred Ragland]

...I also use subsumation.
[a href=\"index.php?act=findpost&pid=146711\"][{POST_SNAPBACK}][/a]

Hmm...subsumation, to

[Bruce MacNeil]

Hmm...subsumation, to
[a href=\"index.php?act=findpost&pid=146719\"][{POST_SNAPBACK}][/a]

In  the sense that the child subsumes the parent. Drive capacity adequate for 2 years of work. Then, in 2 years the new purchase (there is always a new purchase) will be large enough to contain the older system with tremendous room to spare.

In this manner I have every dig files created since 1995 ready and at hand.

[Ray]

They were readable, but that was with the burner I burned them with. Now I have a new burner, which refuses to read a few spindles - but works on most of the brands burned on the old recorder.

Just as I thought. There's an important distinction to be made between hardware  failure or material breakdown/instability and software/firmware incompatibility. Computers are always giving trouble for inexplicable reasons. One usually succeeds in fixing the problems through software/firmware updates or, in the case of an unreadable CD/DVD, simply by replacing the DVD drive.

I've been storing images on CDs and DVDs for about 15 years. In fact, I was getting my old Kodachromes transferred to Kodak's Photo-CD system before I'd bought my first computer. All those early recordings are perfectly readable on my current computers. But that has not always been the case. The second CD drive I bought, upgrading a 4x CD reader to a 20x reader about 10 years ago, refused to read many of those images. Naturally, I assumed the discs were already beginning to suffer from bit rot, although I couldn't see any blemishes on the surface, and of course I complained to Kodak on their toll-free number before I tried the discs on a friend's computer and discovered the problem did not lie with the discs but with the CD drive.

In any case, it doesn't matter. I've given the reasons why HDD kicks DVDs' ass when it comes to backing up. Having to worry about unreadable media on some readers is yet another reason to avoid DVDs.

No, I think you're wrong. It does matter. So far you've confirmed that you've had one hard drive failure but have not confirmed that you've had any DVD failures. For all you know, those apparently unreadable discs, which were once readable on the drive that burned them, are still in perfect condition.

I assure you it's far more worrying to have images stored on a medium which is rapidly deteriorating physically than it is to have images which may occasionally be subject to software incompatibilities during attempted access.

[feppe]

No, I think you're wrong. It does matter. So far you've confirmed that you've had one hard drive failure but have not confirmed that you've had any DVD failures. For all you know, those apparently unreadable discs, which were once readable on the drive that burned them, are still in perfect condition.

I assure you it's far more worrying to have images stored on a medium which is rapidly deteriorating physically than it is to have images which may occasionally be subject to software incompatibilities during attempted access.
[a href=\"index.php?act=findpost&pid=146822\"][{POST_SNAPBACK}][/a]

As said, I've given the list of reasons why HDDs are better than DVDs as backup earlier, and the reasons are none of the ones you list above - and are more pertinent.

Besides, NTFS and FAT32 are industry standards, and a HDD can be read by PCs, Mac or Linux machines. But once again, that's not really a problem as one can migrate all the backups to new media when it becomes available in a matter of minutes. Unlike DVDs, which will take hours and hours.

[Ray]

As said, I've given the list of reasons why HDDs are better than DVDs as backup earlier, and the reasons are none of the ones you list above - and are more pertinent.

Besides, NTFS and FAT32 are industry standards, and a HDD can be read by PCs, Mac or Linux machines. But once again, that's not really a problem as one can migrate all the backups to new media when it becomes available in a matter of minutes. Unlike DVDs, which will take hours and hours.
[a href=\"index.php?act=findpost&pid=146831\"][{POST_SNAPBACK}][/a]

Fair enough! We all have our preferences for a variety of reasons, some of which might be sheer convenience. I've no argument there. I'm just concerned with the facts.

My claim is, I haven't lost any data that has been correctly recorded on optical media since I first started using it 15 years ago. But I have lost data due to failure of hard drives. So have you, it seems.

You claim you have a number of DVDs that are not readable on your current DVD drive but you haven't confirmed whether this is due to DVD drive/software incompatibility or deterioration of the disc.

My concern is you are confusing issues; the convenience and time-saving factor of backing up to hard drives as opposed to the ultimate, long term reliability of optical media.