Cloud Backups

[feppe]

I think there is a lot of misinformation about RAID.  Earlier in this thread someone said that if your RAID card goes bad, your data is unrecoverable.  That isn't true of all implementations.

I didn't say such thing. I'm aware of modern RAID capabilities, but that doesn't change the fact that RAID is not backup.

One suggestion, that I haven't heard mentioned is to make sure to replace hard drives before they get too old.  It is kind of wasteful, but I rarely have trouble finding someone to take and use my old drives for USB drives or something similar.  Another use for old hard drives is to put them into RAID 0 as a fast scratch disk.  I would say that I rarely keep a data drive more than 12-14 months.

For those of us who don't like throwing good money away, it's much cheaper to use RAID for redundancy and speed, and rotate two or more copies of backups until the drives start failing. And replace a drive at the first sign of failure.

For those of you considering fike's suggestion of putting old drives into a RAID 0, ensure it is indeed a scratch disk with nothing critical on it. If one of your drives fails in RAID 0, all data in the whole array is hosed for good. In other words, the failure rate increases with the number of drives in the striped array. And if you have old likely-to-fail drives in the array, that's a recipe for disaster.

Such old drives should be put into a scratch disk striped array, or a redundant RAID array so the failure is not catastrophic.

I agree with your conclusion regarding Amazon's data backup pricing, but one point missed, is reliability:  these folks know how to warehouse data!  I probably wouldn't store *every* file in my possession there, but certainly the inportant ones....

I'd like to raise a very recent Microsoft subsidiary Sidekick debacle. Latest news is that they might be able to recover "most, if not all" data, but lesson learned: catastrophic failures do happen to even the most experienced people and companies.


As recap: I'm not saying not to use clouds or RAID, but don't put your eggs in one basket. Any one system is not enough to protect your priceless data. A friend of mine lost years of photographs, and doesn't have a single photo of his son growing up. Data backups should be taken seriously, as it is easy to ignore and forget, but recovery is often impossible without proper backups.

[John.Murray]

I'd like to raise a very recent Microsoft subsidiary Sidekick debacle. Latest news is that they might be able to recover "most, if not all" data, but lesson learned: catastrophic failures do happen to even the most experienced people and companies.
[/quote]

We'll probably never know *exactly* what happened to the data managed by MS's recently purchased Danger unit - but suffice to say, there are bad, better and best practices to everything, including Data Warehousing.

IMHO I see fairly robust competition ahead for cloud backup and data services;  Amazon S3 http://aws.amazon.com/s3/ , Microsoft Azure http://www.microsoft.com/windowsazure/ and potentially Google  (no formally defined services covering our needs - but here is what they currently offer http://code.google.com/apis/gdata/ )  All of these providers offer geographically dispersed data storage, no eggs in one basket (data center) ala Danger.....

Interestingly the most open about methodology is Microsoft.  Here's one specific example

http://blogs.msdn.com/olavt/archive/2009/0...ial-images.aspx

[craigwashburn]

This was a timely article as my website host apparently disallows uploading backup files to them (I even paid more for the extra space, but they don't want my money apparently).

3 GB of data is a slow week here... much of that is working data though, PSDs etc and not delivered selects.  These are backed up daily across the network.  I work in an aging studio though and flood or fire is always in the back of my mind.  So I had been storing business data, and extensive keeper image portfolio (LZW compressed tiffs) encrypted off-site on my web host.  Currently around 30 GB of data.

I'm now using IDrive for this - $50 / yr for 150 gb.  Transfer rate is holding steady at 472 kb/s, which I believe is about all my connection can achieve.  2 mb/s would be nice but not available here.   Mozy is technically unlimited, but I've heard the transfer rates can be spotty.

S3 seems a little expensive for equal service and is more complicated to use.

One worry I still have though is a fire that wipes out the current job being worked on.  Insurance will cover the monetary damage, but there's always reputation damage and the opportunity cost from the lost images.  Given the slow connection speed it's not realistic to upload the post-production dailies offsite, but perhaps I will schedule a backup of raw selects from each job everyday, so at least the raw files are protected.

[alain]

This was a timely article as my website host apparently disallows uploading backup files to them (I even paid more for the extra space, but they don't want my money apparently).

3 GB of data is a slow week here... much of that is working data though, PSDs etc and not delivered selects.  These are backed up daily across the network.  I work in an aging studio though and flood or fire is always in the back of my mind.  So I had been storing business data, and extensive keeper image portfolio (LZW compressed tiffs) encrypted off-site on my web host.  Currently around 30 GB of data.

I'm now using IDrive for this - $50 / yr for 150 gb.  Transfer rate is holding steady at 472 kb/s, which I believe is about all my connection can achieve.  2 mb/s would be nice but not available here.   Mozy is technically unlimited, but I've heard the transfer rates can be spotty.

S3 seems a little expensive for equal service and is more complicated to use.

One worry I still have though is a fire that wipes out the current job being worked on.  Insurance will cover the monetary damage, but there's always reputation damage and the opportunity cost from the lost images.  Given the slow connection speed it's not realistic to upload the post-production dailies offsite, but perhaps I will schedule a backup of raw selects from each job everyday, so at least the raw files are protected.

For fast off-line storage an external disk will be a cheap and easy solution.

Either a 2.5" usb drive, I've seen 500GB for €75 and only weigthing 170 gr, easy to take with you and store in you're car or another place.  Even a usb stick could be big enough and those are very small.

You could use 3.5" drives which can hold up to 2 Tb, given the size a esata docking is very nice to have.  A lot heavier and I've seen 1.5 TB samsung drives for €89, but those weigth 650 gr without an enclosure.    I use those last one's for my primary backup.  One is always stored off site.

[natas]

I think there is a lot of misinformation about RAID.  Earlier in this thread someone said that if your RAID card goes bad, your data is unrecoverable.  That isn't true of all implementations.  Another person said that you need to tear down and recreate your RAID array if you want to add a drive for increasing size. That, too, isn't true for all implementations.  It is true that older RAID systems had both these faults, but these issues have been addressed.

RAID is a practical every-day solution for redundancy so that your backups don't have to be done hourly.  If you want to work from a laptop, they aren't practical, but doing all your photo work from a laptop isn't a very practical solution anyway (i'm not even going to go into it).

One suggestion, that I haven't heard mentioned is to make sure to replace hard drives before they get too old.  It is kind of wasteful, but I rarely have trouble finding someone to take and use my old drives for USB drives or something similar.  Another use for old hard drives is to put them into RAID 0 as a fast scratch disk.  I would say that I rarely keep a data drive more than 12-14 months.

I agree that the Amazon service pricing is a bit out of reach for most of.  I am not sure that whether I need to use it in addition to a decent backup and recovery plan that includes offsite storage.

I use http://www.backblaze.com

It's 5/month, unlimited usage and works on both windows and mac. It took me about 2 weeks to get all 500GB of my data transfered but man its so much cheaper. If you need to restore you have 3 different options.

[robertvine]

By far, the most likely event that will cause you to lose data is a drive failure. RAID and WHS type data redundancy is a good way to insure yourself from a drive failure because your data is duplicated as soon as it is stored on the system and if you do get a drive failure you can recover from it without having to download all your data off the internet.

This won't protect you from the worst case scenario though, if your house/studio burns down then RAID isn't going to save your data. You need an offsite solution to protect against this, the cheapest way to do this is to have an external drive that you backup to. The problem I found with this is that I usually created a backup every month so a lot of data goes unprotected for a long period. You could also have a failure in your backup drive which could go undetected until you need that data.

My WHS is set to run a backup at a certain time every day so new data does not sit on the system long until it is backed up off site. A professional service should protect your data from a drive failure by continually checking the data that is stored on it's servers and resolving any issues it finds.

Thanks for recommending Crashplan, I'm going to move from S3 to it soon.