Cloud backup of (lots of) photographs

[Jim Kasson]

For several years, I’ve been running a backup protocol that keeps three backups of workstation photographs and other documents on Synology local NAS boxes, with a rotating pool of hard disks – recently, 6TB ones – physically transported to a safe deposit box for disaster recovery.

I’d like to rotate the offsite disks every month, but I seem to only do it three or four times a year. This leaves me open to loosing several months work in the event of a site-wide disaster. I am now considering cloud backup to replace, or possibly enhance, off-site storage of physical disks. I don’t intend to take any of the NAS boxes out of service, so this would end up being what some people call a hybrid backup scheme.

In the past, I have been wary of upload and restore times for cloud backup. However, in the last year I’ve upgraded my Internet connection to a 50 Mb/s (bidirectionally) AT&T IPFlex circuit, which means that, if the connection speed is the limiting factor, backing up or restoring 6 TB (the size of my current photo collection) in less than two weeks. In addition, there are some cloud backup providers that provide seeding and disaster recovery with shipped disks.

I’ve done some initial testing with Amazon’s S3 service and with Dropbox Pro. I get about 18 Mb/s upload speed on large files with S3 without using the concurrent transfer feature. I am unclear on how to use overlapped transfers in unattended operation without writing some code, which I would prefer not to do. Dropbox Pro seems to max out at a little over 3 Mb/s up, although I’ve seen very fast downloads from them. I have not yet tried Google Drive.

I am intrigued by Amazon Glacier, with its very low storage costs, but whether it is implemented with tape or not, it has tape-like characteristics, and all my backup software for the last five or six years has assumed direct access storage devices (DASD in IBM-speak, pronounced “dazz-dee”). I never liked the tape-oriented backup software that I used in the nineties and oughts, but haven’t looked at current offerings.

I haven’t made up a formal list of requirements, but here’s an off-the-top-of-my-head start:
Low cost, high transfer rate In both directions, security/privacy (super-strong passwords, maybe two-factor, SSL uploads and downloads, server-side storage redundancy, support, encrypted server-side storage, etc), good client-side backup software.

My initial reaction was to put company stability in there, but now I’m not so sure. If the service provider goes belly up, I could switch to a new provider, and make more trips to the safe-deposit box in the meantime.

Here are some questions for everybody who reads this:

Do you have a similar problem? Are you looking at similar solutions? What services should I be considering? For those who have implemented something like this, what were the lessons learned? What client-side software do you like?

Pretty open-ended, but I figure I’m not alone in considering this, and maybe we can all learn from each other.

A few links to what I've already done:

Some general thoughts: http://www.kasson.com/bleeding_edge/?p=969
Experiences with Dropbox Pro: http://www.kasson.com/bleeding_edge/?p=1003
Experiences with S3: http://www.kasson.com/bleeding_edge/?p=980
Firewall performance considerations: http://www.kasson.com/bleeding_edge/?p=1018

Jim

[Jim Kasson]

More results:

I'm only getting 4 Mb/s upload rates for 100 MB files to Google Drive using Goodsync.

Jim

[Alan Goldhammer]

Jim,

Thanks for posting your experience.  I've been looking at Amazon S3 storage but haven't tried it yet.  I use NovaBACKUP as my client for backup and it can run cloud storage directly.  I have Verizon FIOS here and the upload speed is supposed to be 20MB/sec.  I've been using Mozy Home for cloud storage as I don't have nearly as much data to store as you or probably others where photography is more full time.  It's more expensive than Amazon S3 but it does have a good web interface and just the other day I had to download a couple of work files that I needed and it all went smoothly. 

[tastar]

If you are using Synology NAS devices for backup, buy a new one - and use it for your remote backup destination. A DS-411J is around $450.00 and it will store up to 18TB of data in RAID 5 using 6TB drives. And, out of the box, Synology devices support remote backup and you can use their free DDNS service if you don't have static IP addresses. You can do the initial backup to the new Synology on-site and then move it off-site - to a place with a reasonably fast internet connection - and then run a daily sync backup over the internet (you can schedule it to run at night when you're not using your system). If your base system ever blows up, you can bring the remote system back to the base and plug it into your network and you're up and running. Our experience is that the Synolgoy to Synology backup is very reliable and a lot less hassle than traditional tape and/ or disk backups.

Tony

[Jim Kasson]

Looks like CloudFront S3 Browser, which supports concurrent S3 file operations, will support high upload data rates. It can easily max out my firewall, which happens at 25 Mb/s. To see what it can really do, I'm going to have to upgrade the firewall. S3 Browser is not a completer solution; I still need some way to automate the process.

Details on CloudFront S3 Browser testing here: http://www.kasson.com/bleeding_edge/?p=1041
Details on the firewall performance here: http://www.kasson.com/bleeding_edge/?p=1018

Jim

[Jim Kasson]

If you are using Synology NAS devices for backup, buy a new one - and use it for your remote backup destination. A DS-411J is around $450.00 and it will store up to 18TB of data in RAID 5 using 6TB drives. And, out of the box, Synology devices support remote backup and you can use their free DDNS service if you don't have static IP addresses. You can do the initial backup to the new Synology on-site and then move it off-site - to a place with a reasonably fast internet connection - and then run a daily sync backup over the internet (you can schedule it to run at night when you're not using your system). If your base system ever blows up, you can bring the remote system back to the base and plug it into your network and you're up and running. Our experience is that the Synolgoy to Synology backup is very reliable and a lot less hassle than traditional tape and/ or disk backups.

Tony, that's a possibility, but I need to consider the logistics. I'd have to find someone with an Internet connection at least as fast as mine who didn't mind sharing it. Probably, it would be an "I'll do it for you if you'll do it for me" kind of thing. I'd want them far enough away that the same earthquake that gets me wouldn't get them (yes, I know I'm raising the ante over the protection I get today at my bank, but if I'm going to do this, I probably should do it right). That means I'd have to make my peace with their use on my bandwidth. 

It's an intriguing idea, but I think I'll keep it on the back burner for now.

Jim

[fdisilvestro]

Just a few comments from my experience:

I use CrashPlan, which provide an automated and unattended service for backup with adequate encryption and protection. I believe other photographers in this forum use it too. The bandwith might be limited (I don't have the figures) but they offer the seeding and disaster recovery with shipped disk in some countries. You can also define a secondary location in your network for an additional backup. The data is deduplicated, so the actual storage required is less than the original files and it has an impact in the bandwidth too.

Tape backup: I would not dismiss tape backup yet. Current offerings such as Ultrium LTO 6 are very capable and the technology continue to evolve.

Amazon S3 or Glacier: Two different products for very different purposes. I would use S3 for files that have to be constantly accessed / modified, the same as Dropbox (BTW Dropbox runs on Amazon S3), but it might be too expensive for a last resource backup. Glacier looks better for your purposes, even if it does look tape based. I think that there are applications for automating the backup with Glacier, but I don't have direct experience.

One more thing, Amazon S3 and upload / download speed: There are options for very high speed transfers, but I think they would be too expensive for this need.

[Jim Kasson]

I use CrashPlan, which provide an automated and unattended service for backup with adequate encryption and protection. I believe other photographers in this forum use it too. The bandwith might be limited (I don't have the figures) but they offer the seeding and disaster recovery with shipped disk in some countries. You can also define a secondary location in your network for an additional backup. The data is deduplicated, so the actual storage required is less than the original files and it has an impact in the bandwidth too.

Thanks, Francisco. I just tried CrashPlan, and got only 10 Mb/s upload speeds for 100 MB files.

Amazon S3 or Glacier: (BTW Dropbox runs on Amazon S3)

I didn't know that. Dropbox Pro is a lot cheaper than S3's published prices. Either Dropbox is getting a real deal from Amazon, or they're counting on their customers not using anywhere near the full TB. Or maybe they're like the guy in the joke: lose a little bit on every sale, but make it up with volume.

Amazon S3 and upload / download speed: There are options for very high speed transfers, but I think they would be too expensive for this need.

Yeah, I was afraid to ask. Anyway, it looks like regular S3 is fast with concurrent file ops.

Jim

[Jim Kasson]

I just tried CrashPlan, and got only 10 Mb/s upload speeds for 100 MB files.

I asked CrashPlan about the speed, and here's what they said:

Jim, though your ISP offers much faster speeds, the upload bandwidth you are seeing is determined at our end by the number of active users sharing the same server point as you.

Average user speeds range from 2-5Mbps, so you are currently receiving double.

Jim

[deejjjaaaa]

I asked CrashPlan about the speed

but you can use their software to backup to your own remote location, if you have a computer running there (for example relatives ?)... so it might be a much faster upload from your place vs backup to CP's own storage.

[deejjjaaaa]

Thanks, Francisco. I just tried CrashPlan, and got only 10 Mb/s upload speeds for 100 MB files.

you are not shooting 100 GB a day of photos if the whole amount is 6 Tb of your lifetime work, so what is the problem with such speeds, except that you will be uploading the initial data for a while ?

[Jim Kasson]

you are not shooting 100 GB a day of photos if the whole amount is 6 Tb of your lifetime work, so what is the problem with such speeds, except that you will be uploading the initial data for a while ?

It's not just finished work I'd like stored, but WIP as well, and when I'm working on this series, the captures alone can be 30 GB/day -- each capture is 6000x64000 pixels. But you're right, I could let that slide, since I'm only getting at best a once a month snapshot for my offsite backup now. But, taking the worst of the CrashPlan quoted rates, 2 Mb/s, it would take me over a month [Edit: over nine months; I forgot to multiply by 8.] to upload what I've got now.

Maybe now that I've seen the S3 rates, I'm just greedy.

[Added later: CrashPlan is cheap enough that it might make sense just as an extra backup site.]

Hmm...

Jim

[sbay]

I’d like to rotate the offsite disks every month, but I seem to only do it three or four times a year. This leaves me open to loosing several months work in the event of a site-wide disaster. I am now considering cloud backup to replace, or possibly enhance, off-site storage of physical disks. I don’t intend to take any of the NAS boxes out of service, so this would end up being what some people call a hybrid backup scheme.

Have you brainstormed about what you could do make the offsite disk swapping more consistent/frequent? I think due to bandwidth caps and slow upload speeds cloud storage will have significant disadvantages.

[jjj]

There are alternatives to US backups. Hubic are a French company and do 10TB for 10 Euros/month or 100GBs for 1€ .

[Jim Kasson]

Progress report

I upgraded my firewall and am now getting 47+ Mb/s upload and download speeds on Verizon speed test. S3 Browser is uploading files at a sustained rate of 5 MB/s, or 40 Mb/s.

Jim

[jjj]

That's a good upload speed. I have 150MB/s upload but a pathetic 10MB/s upload. And that's the best you can get in the UK.

[Jim Kasson]

That's a good upload speed. I have 150MB/s [download?] but a pathetic 10MB/s upload. And that's the best you can get in the UK.

That would be b/s (bits per second) not B/s (bytes/second) right?

It's not cheap. I'm paying $1K/mo, and I got a deal -- take 10 trunks of our VoIP service, and we'll charge you less for both than you would have paid for data alone. Also, I had to trench, bury the fiber conduit, install the pull boxes and the termination box.  AT&T did pull the fiber (including spares), terminate them, and install the transceivers and an immense Cisco router. Where I live there's no cable service. Or DSL, for that matter. Before I got the fiber, I was paying $750/mo for two T1 lines.

Jim

[ned]

The first service I tried was Acronis. I sent them a usb drive, they uploaded and that was it. Or so I thought. Without going through all of the details my whole archive was corrupt from the start. Bad customer service, etc etc. Then I tried Carbonite, they throttled my upload after a certain about of data was uploaded. No go. My third choice seems to be working out fine, it's called backblaze. $5.00 p/m unliminted and no throttling. I have 1.8TB uploaded, and when I check it's still there :-)  Last I checked I was getting a little over 7mb/sec upload speed which is as good as I have ever had with Comcast. Good luck.

[jjj]

That would be b/s (bits per second) not B/s (bytes/second) right?

It's not cheap. I'm paying $1K/mo, and I got a deal -- take 10 trunks of our VoIP service, and we'll charge you less for both than you would have paid for data alone. Also, I had to trench, bury the fiber conduit, install the pull boxes and the termination box.  AT&T did pull the fiber (including spares), terminate them, and install the transceivers and an immense Cisco router. Where I live there's no cable service. Or DSL, for that matter. Before I got the fiber, I was paying $750/mo for two T1 lines.

Jim

Yikes, that's expensive! We pay £62/$90 for 152Mb/s [not MB/s  ] unlimited data [some ISPs here cap data], two HD recorders [Tivo boxes] more TV channels than we ever watch including lots of HD and a phone line. Didn't have to dig any trenches either.  
Over a decade ago a friend in Japan mentioned his 'slow' 25Mb broadband was being replaced by a 100Mb service. Must ask him what he gets these days.

[deejjjaaaa]

I'm paying $1K/mo

ouch - that's for a one residence (your house) ?