Long term storage

[MBury]

Hello there,

I'm well underway with enlarging my digital photo-library, but I've run into a few problems. Size, back-up and storage.

The storage problem is easy enough to solve, just a question of adding a disk or more when required. However, with todays disks, making back-ups is remenisent of the times I worked with 3.5"s. It is like playing DJ but then with dvd's.

I've made it a good habbit of burning my raw files to DVD just after I've transfered them from CF/SD to disk ( and will not delete them from the cards before verification is succesful ). For good measure, I make two copies! And that was a lucky strike, because after trying to read a DVD burned a couple of month ago, it was unreadable, and I'd needed to resort to the 2nd copy.

The situation for now is as follows:

1 copy of all raw files reside on a hard-disk
2 copies of all raw files reside on two seperate series of DVDs.

However, I don't trust the DVDs for long term storage. Are there any solutions known? Ones that are bettter than DVDs. I remember worm-disks that use to have shelve lives beyond 30 years but their capacity was limited.

Before anyone starts asking, no I did not use cheap bulk DVDs.

Does anybody have an idea?

Regards,

[Eric Myrvaagnes]

Hello there,

I'm well underway with enlarging my digital photo-library, but I've run into a few problems. Size, back-up and storage.

The storage problem is easy enough to solve, just a question of adding a disk or more when required. However, with todays disks, making back-ups is remenisent of the times I worked with 3.5"s. It is like playing DJ but then with dvd's.

I've made it a good habbit of burning my raw files to DVD just after I've transfered them from CF/SD to disk ( and will not delete them from the cards before verification is succesful ). For good measure, I make two copies! And that was a lucky strike, because after trying to read a DVD burned a couple of month ago, it was unreadable, and I'd needed to resort to the 2nd copy.

The situation for now is as follows:

1 copy of all raw files reside on a hard-disk
2 copies of all raw files reside on two seperate series of DVDs.

However, I don't trust the DVDs for long term storage. Are there any solutions known? Ones that are bettter than DVDs. I remember worm-disks that use to have shelve lives beyond 30 years but their capacity was limited.

Before anyone starts asking, no I did not use cheap bulk DVDs.

Does anybody have an idea?

Regards,
[a href=\"index.php?act=findpost&pid=80141\"][{POST_SNAPBACK}][/a]

Light Impressions, in their latest catalog, advertises "archival" gold CDs and DVDs. They claim 300 years for the CDs and 100 years for the DVDs. I haven't tried them, but I'm considering it. As far as I know, Wilhelm Research hasn't tested DVDs, however.

Eric

[61Dynamic]

At this point in time there is no real long-term solution for storing data. Redundant hard-drives are the best bet and will continue to be so for the foreseeable future.

I've made it a good habbit of burning my raw files to DVD just after I've transfered them from CF/SD to disk ( and will not delete them from the cards before verification is succesful ). For good measure, I make two copies! And that was a lucky strike, because after trying to read a DVD burned a couple of month ago, it was unreadable, and I'd needed to resort to the 2nd copy.

The problem with CD/DVDs is you can't trust them. In order to cut costs companies selling them use inferior materials that can corrode in as little as a month (as you noticed). some discs use better materials allowing a shelf life of years but the unfortunate reality is that you can't tell (with an exception I'll get to in a moment) which discs have that and which don't. To make things worse, the low-buck Office Max/CompUSA/etc branded discs are simply discs branded with those names from the lowest bidder meaning one batch may last years but the next batch may crap out next week.
 
The exception is MAM-A Silver discs. They use higher quality materials which are supposedly able to last a few years at least.

As Eric pointed out, there are gold archival discs both from Delkin and MAM-A advertising 100-300 years of life. While easily believable they will last longer than a standard disc, they are un-tested and may not last as long as they claim.

[Mike Boden]

The long-term storage medium that most enterprise businesses use is tape, such as LTO or DLT. Unfortunately, this system isn't very cheap. But if you want more reliability than DVD, it's another option.

[RonBoyd]

This conversation has been going on for a long, long time. A Google search will give you an idea. Try for example:

http://www.google.com/search?hl=en&lr=&rls...%22+%2Barchival

Several years ago, I spent many a butt-numbing evening trying to find a definitive answer and only became more and more depressed.

CDs and DVDs are not the optimum storage method but, then again, neither are hard disks (for the same reasons). And RAID is an even a worse idea. Tape is another medium that fails the test of usefullness for long term storage. I finally came to the conclusion that the only safe storage method -- one that has been relied upon for thousands of years -- is, what we call today, a hard copy. In other words, make a quantity of the best print you can and spread them around. Oh sure you can save in a digital manner but resolve to continually be updating -- at least every five years copying the files to a new "improved" device. 30 year shelf life. Ha! What in thirty years is going to be your play-back machine? You can bet there will be a way to "take a picture" of an item far into the future, even if it would be that old-fashioned method called Scanning.

Of course, if you stumble across a break-through, I want to hear about it right away.

Ron

[ericstaud]

The DVD's look to be the same price as hard drives at about .50 per GB.  I had a hard drive with about 80 CD-R's worth of images fail.  Imagine the project of downloading the 80 CD-R's into a new hard drive.  

Now I have a two Hard drive system of Files_01 and Files_01b.  I use Chronosync to back 01 to 01b.  They are both spun down most of the time.  01b is kept in a drawer.  With the Firmtek 2 bay device I just buy trays as I add pairs of drives.  I have two of the two-bay units, and 8 drives in trays.  I like that if a drive fails it is a simple operation to recover the files.  I have written dates on the outside of the drives.  I plan to migrate them every three years.  By then I will probably have pairs numbered from 01 to 07 (14 drives total).  At that time they might all fit onto one drive, who knows?  Those two new drives will be called Files_08 and Files_08b.

As my cataloging improves I find that the metadata and organization I add to an earlier drive can easily be added to it's backup as well.

It is certainly hard to project what kind of back-up scheme will serve you best in the future.  Planning for having your work databased should play into your plans.  The DAM book is really required reading for all photographers.  You may not use all the methods from the book, but considering them and understanding their purpose and use is very helpfull.

Best of Luck,

Eric

[dobson]

I use paired hard disks. They're cheaper than CD's/DVD's, (if you look around $0.25/Gb), and they're so much faster. If I had money and wanted to be really fancy I would set up a RAID to automatically save everything to two disks so that if anything failed at any time there would be no worry.

Phillip

[ericstaud]

"If I had money and wanted to be really fancy I would set up a RAID to automatically save everything to two disks so that if anything failed at any time there would be no worry."

Hi Phillip,

The mirrored RAID is dreamy in a lot of ways.  But if your directory gets corrupted, I think both drives are at risk (and corruption might be more common than the mechanical failure a RAID would protect against).

Also, I threw away 4,500 jpegs a few nights ago in a fog of trying to organize things.  It took about 10 minutes to realize I should not have emptied the trash.  It took me 2 minutes to get out the back-up drive and restore the files.  This is another dis-advantage of mirrors raids.

I think the paired disks you use now might be better.  Do you do anything for off-site storage?  That is the thing that bugs me when I leave the house, especially when I leave town.  

I am considering setting up a big box of drives at a friends studio or house with a VPN (virtual private network box) hooked up the their router.  This would act as though I had networked drives sitting in my office.  The 1mb/second DSL upload I have from home might be fast enough to back-up images off-site overnight.  Has anyone else considered this route?

-Eric

[Ronny Nilsen]

I am considering setting up a big box of drives at a friends studio or house with a VPN (virtual private network box) hooked up the their router.  This would act as though I had networked drives sitting in my office.  The 1mb/second DSL upload I have from home might be fast enough to back-up images off-site overnight.  Has anyone else considered this route?

-Eric
[a href=\"index.php?act=findpost&pid=80184\"][{POST_SNAPBACK}][/a]

Yes I do that, I have an external USB drive on my computer at work that have a copy of all images, in addition to exernal USB disk backup at home of course. And all images is copied on two computers at home, so that an accidental delte on one of them don't wipe everything clean (the backup would be deleted at the next nigthly sync).

[Woodcorner]

I am considering setting up a big box of drives at a friends studio or house with a VPN (virtual private network box) hooked up the their router.  This would act as though I had networked drives sitting in my office.  The 1mb/second DSL upload I have from home might be fast enough to back-up images off-site overnight.  Has anyone else considered this route?
[a href=\"index.php?act=findpost&pid=80184\"][{POST_SNAPBACK}][/a]

I am using RAID 5 firewire drives for local storage. Never had any fatal failures. If a single drive fails, I'll just swap the defective harddrive for a new one. Cost is around 0.6 USD per GB

For major catastrophies (more than one drive fails at a time, fire, flooded office, theft, and the like), I am backing up the RAID drive on a daily basis to a network drive co-located at my ISP. This machine is located in a highly secure datacenter, with UPS systems, access control etc.
I am very happy with EMC Retrospect software to do the backup job, as this software allows me to backup to disks and do progressive backups. So even if I delete a file locally and do a backup afterwards, I will be able to recover it easily for any given point in time.

The time needed to do a backup over the internet might nevertheless be crucial. Backing up multiple GB over a DSL line could prove to be impossible. You'll need a lot of bandwidth to do so!

Cheers,

Andrew

[GregW]

I sense many here have the same frustrations I do.  It seems quite difficult to find a robust solution at a reasonable price particularly as I am just an amateur.  I'm very grateful I don't have an interest in digital video otherwise I'd be completely exasperated.  

My compromise workflow goes like this:

Camera > CF Card > Powerbook > External HDD > DLT (archival tape about 200GB uncompressed) x2 one for home and one for the office safe.

The cards don't go back in to the camera until the files are on the external HDD.  The weak link is that it takes about an hour to dump a 100GB's on the tape so I only do this at weekends.  That kind of makes sense as I tend to mostly shoot at weekends.  If I were making my living with this I would do it daily.

DLT systems are much cheaper than they used to be.  I spent the first 5 years of my professional life implementing large enterprise systems (SAP).  Storage costs do seem to come down in line with data volume growth.  USD 1000 gets you something in the 50GB to 200GB range, USD 3000 for 1TB and if you want more that means auto loading which puts you nearer USD 5000!!

No one has mentioned it yet but Blueray will offer about 50GB on each disc which should take some of the pain out of burning multiple DVD's.  I know nothing about Blueray to be honest.  I assume it will have the reliability and longevity issues as CD's and DVD's.

[MBury]

Thanks thus far,

I've thought of the tape-library also, but besides it being very expensive ( the 100GB+ ones ), they are not meant to last forever! The idea is that tapes circulate in a special cycle which means data is at most a week or so old before it is replaced.

The problem is that most back-up and secure data storage solutions are geared towards constantly changing data, whith media that are refreshed daily to weekly.

I will look into a combo of the solutions you've all mentioned above. The first thing will be to buy a separate FW800 or Esata raid-enclosure to store the raw-files to, in addition to burning them on DVDs. What got me realy intrested though is the storage in a data-center. With ADSL or ADSL2, upload-speeds are sufficient to allow storage online, provided it is done at short intervals to prevent 3 to 4 GB + transfers.

To be honest, with all the digital revolution going on, I'm rather surprised that none of the big companies like Canon/Kodak/Nikon have addressed this problem, because it will only get bigger!

regards,

[RonBoyd]

To be honest, with all the digital revolution going on, I'm rather surprised that none of the big companies like Canon/Kodak/Nikon have addressed this problem, because it will only get bigger!

[a href=\"index.php?act=findpost&pid=80203\"][{POST_SNAPBACK}][/a]

Sounds more like an opportunity for two kids in a garage (ala Google and YouTube) to me. ($1.6 billion. Hmmmm.)

My digital image storage is now approaching a terrabyte (1,000 gb) in size. And that's only figuring a single copy -- no back up. So you are correct, the problem is only going to get worse (or better if you are the kid in the garage).

ericstaud mentioned the DAM book (by Peter Krogh). I couldn't agree more with how important this book is to anyone with large quantities of digital files (his companion forum only slightly less so). A problem even bigger than storage is the issue of retrieval which, of course, goes hand-in-hand with the subject problem.

I have read this thread with great interest and am aware of each of the solutions offered. However, no one seems to have an answer that I am looking for -- when can I put this stuff away and stop worrying about it disappearing while I go on with life?

Ron

[61Dynamic]

CDs and DVDs are not the optimum storage method but, then again, neither are hard disks (for the same reasons).

Not quite. Hard drives can fail but for entirely different reasons than optical discs. The predominant cause for hard drive failure is cause by software; read/write errors. A redundant system (say, RAID 1 at least) reduces the chance of data loss drastically. At that point, you only need to worry about physical failure. Strong impacts, bearing grease drying up (takes a looooong time), etc.

With a properly set up system, the biggest problem facing HDD longevity is possibility of the connection interface (I.E. IDE, SATA, FireWire, etc) becoming obsolete.

"If I had money and wanted to be really fancy I would set up a RAID to automatically save everything to two disks so that if anything failed at any time there would be no worry."

Hi Phillip,

The mirrored RAID is dreamy in a lot of ways.  But if your directory gets corrupted, I think both drives are at risk (and corruption might be more common than the mechanical failure a RAID would protect against).
[{POST_SNAPBACK}][/a]

You are thinking of RAID 0 where two drives are combined to make one large drive. In that situation, one drive failing would cause data loss on both (unless ZFS file systems become common but I digress...). RAID 1 is where two drives are mirrored and in that case, if one drive fails, the other remains usable.

[a href=\"http://www.acnc.com/04_01_00.html]This visual[/url] of RAID systems might help.

No one has mentioned it yet but Blueray will offer about 50GB on each disc which should take some of the pain out of burning multiple DVD's. I know nothing about Blueray to be honest. I assume it will have the reliability and longevity issues as CD's and DVD's.

No one can be sure of that yet. Sony was advertising BlueRay's ability to be built out of bio-degradable materials a wile back (think corn) so if companies keep their tradition of being vague about build materials then it could be more difficult than it is now. Of course that assumes BlueRay will see any success. At this time it's a toss-up between BlueRay and HD-DVD.

Another fun issue with the next-gen optical discs is their sensitivity too scratches. Due to the density of the data put on the discs, they are very susceptible to smudges and scratches. In fact, on an episode of DL.TV Robert Heron (from PC Magazine) was saying they couldn't even get the DVD drive to read the disc at one point and found that it was due to a very slight fingerprint smudge on the disc.

[nma]

RAID is not an archival solution.  RAID 5 will provide some redundency for single drive failures. Unfortunately, that is not the only failure mode.  Power suplies, even redundent supplies can fail during the write operation corrupting the array.  This is not theory; it is hard experience.

Caveat emptor.

There does not seem to be a good solution at this time.

[GregW]

Thanks thus far,

I've thought of the tape-library also, but besides it being very expensive ( the 100GB+ ones ), they are not meant to last forever! The idea is that tapes circulate in a special cycle which means data is at most a week or so old before it is replaced.

Sony claims about 10-15 years based on corporate re-use cycles.  More if it's used for archival.  It's certainly not approaching the life of a high quality injet print these days but long enough until there is a better solution to migrate to.

Off topic I know, but will our current RAW formats be supported in 10 years, let alone 100!

[RonBoyd]

Off topic I know, but will our current RAW formats be supported in 10 years, let alone 100!
[a href=\"index.php?act=findpost&pid=80251\"][{POST_SNAPBACK}][/a]

Actually, not so off-topic. That is exactly the point that Adobe is trying to make with its DNG format. In any event, the same question can be asked of TIF, JPG, or any other current format. I am much more optimistic in this area but still it is a concern.

Ron

[UURickRose]

I was using an external hard drive for a long time but didn't have it backed up and my drive went bad.  Therefore, I had to come up with a better solution.

I set up a very nice NAS (Network Attached Storage) from Infrant Technologies - you should look them up.  I bought (4) 500 Gig SATAII Hard Drives and installed them in less than 5 minutes using RAID 5.  The box support Gigabit Ethernet too and I'm getting exremely fast throughput on my LAN.  The Infrant server rules and their support is great - althought I didn't need any.  Check out their forums.  http://www.infrant.com

My entire set up cost me $1500.

If you're superparanoid, you could always buy 2 boxes.  This way you have a complete hardware failover solution in place.

I too have about 1 Terrabyte of data and there is no way I have time to burn that many DVDs.  I'll stick with my NAS.  If a drive fails, you simply replace it and let the RAID set rebuild itself (the advantage of RAID).  I'm not worried about power supplies failing.  If it does, I'll replace it.  Again, the ultra paranoid can run these in parallel which is supported natively within the box.

If you're super ultra paranoid - you should think about off-site disaster recovery too - meaning back your stuff up off-site.  Think natural disasters, fires, etc...

[DiaAzul]

Adding more experience from an enterprise/ internet hosting perspective.

1/ The number one lesson is that ALL storage media will fail at some point in time. Therefore,  it is essential to keep pulling up your backup copies and testing that you can read the data on a regular basis. This could be monthly, quarterly or whatever depending on how paranoid you are and the quality of the storage media that it is stored upon.

2/ Tape is very slow both for (a) finding the data you are looking for ( reading and writing the data

3/ There is no reason not to use CDs/DVDs as storage so long as you make multiple copies of your data and test the media on a regular basis. Reburning discs that go bad as you go.

4/ There is no reason not to use mirrored disks...with the caveat that failure of the raid controller will render BOTH disks in a mirrored array useless and you will loose all your data (I worked for an organisation where this did happen in a very public and expensive way). If you are going to use disks for backup then the systems need to be independent ...raid is a solution for near real time protection, not for long time archiving.

5/ Make sure that you keep a copy off site and that it is checked/refreshed on a regular basis (rotation of off-line discs for instance). Fire, flood and theft are as big a risk as equipment failure.

At the end of the day there is no one perfect solution, it will always be a trade off between time, cost and amount of protection for the data. However, the key to succesful archiving is knowing when an archival medium has gone faulty and replacing one of perhaps two or three of the copies before you loose all of them.

[Woodcorner]

One always has to put the backup strategies into the right perspective. Not long ago, without digital cameras, most of us only had one copy of our precious images: a single slide or negative. We kept them in expensive archival folders and cabinets with the same risk of theft, floods, fires.

To me, things have become a lot easier as I am able to make *identical* copies of my valuable images. As others have said before, the only thing I am worried about is the longevity of file formats and interface designs for harddrives and so on. But I am also not quite sure how long my Velvias will last as well...  

Cheers,

Andrew