Pages: [1]   Go Down

Author Topic: Archive Backup Solutions?  (Read 3568 times)

Chris Barrett

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 730
    • www.christopherbarrett.net
Archive Backup Solutions?
« on: April 01, 2016, 08:49:00 am »

Like most of you, I have a huge archive of big files, not to mention terabytes of video.  My active storage is comprised of a few 8 drive RAIDS, both internal and SAS.  I've been looking at updating my offsite backups and am considering a couple things.

One of the more common standards in the film industry is LTO.  LTO drives and cartridges are pricey, but surely worth the piece of mind.  Alternately, what about cloud based storage, ala Backblaze?  I've never used one of these services and wonder how well they work?

I've probably got about 30TB worth of files I'd like to copy to the best solution.

Opinions?

CB

gebseng

  • Full Member
  • ***
  • Offline Offline
  • Posts: 173
Re: Archive Backup Solutions?
« Reply #1 on: April 01, 2016, 09:20:27 am »

In case you have not read this blog entry by Jim Kasson yet: http://blog.kasson.com/?page_id=3060

His central argument: more hard drives!

I think it makes sense. 30TB in HDs will set you back less than USD 1,000.-, while the LTO drive alone costs 1,500.- minimum, not to mention the cost of the tapes. Also, a tape based backup solution only makes sense if you check the data integrity every few months.

In my own experience, backing up to online backup services like Backblaze or Crashplan does not exceed a speed of 2MBit/sec.

best,

geb
« Last Edit: April 01, 2016, 09:27:46 am by gebseng »
Logged

JoeKitchen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5022
Re: Archive Backup Solutions?
« Reply #2 on: April 01, 2016, 09:22:26 am »

Like most of you, I have a huge archive of big files, not to mention terabytes of video.  My active storage is comprised of a few 8 drive RAIDS, both internal and SAS.  I've been looking at updating my offsite backups and am considering a couple things.

One of the more common standards in the film industry is LTO.  LTO drives and cartridges are pricey, but surely worth the piece of mind.  Alternately, what about cloud based storage, ala Backblaze?  I've never used one of these services and wonder how well they work?

I've probably got about 30TB worth of files I'd like to copy to the best solution.

Opinions?

CB

I have been thinking about this for while too and would sleep easier at night with an off site cloud storage backup.  However, the amount of data that would need to be initially backed up is quite high. 

I saw that Backblaze has an option where they will send you a 3 TB hardrive to fill and return.  If they had something that was more like 8 or 10, that would be great. 

I too would be interested in hearing options. 
Logged
"Photography is one percent inspiration and ninety-nine percent

Christopher

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1499
    • http://www.hauser-photoart.com
Re: Archive Backup Solutions?
« Reply #3 on: April 01, 2016, 09:24:33 am »

I would go with a large Sinology with RAID 6 or their similar system.   


Christopher Hauser
ch@chauser.eu
Logged
Christopher Hauser
[email=chris@hauser-p

Chris Barrett

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 730
    • www.christopherbarrett.net
Re: Archive Backup Solutions?
« Reply #4 on: April 01, 2016, 10:02:28 am »

In case you have not read this blog entry by Jim Kasson yet: http://blog.kasson.com/?page_id=3060

His central argument: more hard drives!


That all seems reasonable enough.  I have three fast RAIDs in the studio each is 8 drives in RAID 5.  This allows for easily accessible and fast backups.  I think for the archive, I may just do bare drives in a safe deposit box, refreshing those at some interval.

Christopher

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1499
    • http://www.hauser-photoart.com
Re: Archive Backup Solutions?
« Reply #5 on: April 01, 2016, 10:27:14 am »

I would not go for single drives! Especially as you will never know in what condition each drive is. I would still go for a RAID 6 a cheaper solution would be a type of case with software RAID as you probably won't have to access that backup all the time.


Christopher Hauser
ch@chauser.eu
Logged
Christopher Hauser
[email=chris@hauser-p

Conner999

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 932
Re: Archive Backup Solutions?
« Reply #6 on: April 01, 2016, 10:43:09 am »

Another vote for single drives, via a 'toaster' or bare-drive dock.  Nice, simple, secure can be read by any similar computer, can be installed in a JBOD enclosure latter if desired, easily shipped if need be, etc. Store offsite in secure packaging. 

Our philosophy (YMMV) is our offsite / archive storage has to be the 'dumbest' - that meaning the simplest, most idiot-proof, most caveman level of tech in our system. No manufacturer's RAID software, hardware or cloud access / 3rd party business longevity between the images and our getting at them / adding to them as time passes.
Logged

alatreille

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 428
    • Between the Buildings
Re: Archive Backup Solutions?
« Reply #7 on: April 01, 2016, 11:44:43 am »

That all seems reasonable enough.  I have three fast RAIDs in the studio each is 8 drives in RAID 5.  This allows for easily accessible and fast backups.  I think for the archive, I may just do bare drives in a safe deposit box, refreshing those at some interval.

Hi Chris. 
I went through this exercise a year ago and came to the conclusion that you have.

I'd ideally like to have this and the cloud based backup, but to upload that number of terabytes would take a very very long time.

Another thing I'm contemplating for the future, is to partner with another business that has high data demands.  Each person can place a raid box in the others business location and set up a private network between the boxes for nightly backups of the working drives.
I'm still working through the nitty gritty of it, but it feels to me like something that might be work exploring.
You could also do this with a box at home.

If anyone has done this, please share your experience or knowledge.

Cheers
Andrew

Andrew

Logged
Architectural Photographer
http://www.andrewlatreille.com

Endeavour

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 393
Re: Archive Backup Solutions?
« Reply #8 on: April 01, 2016, 11:54:45 am »

backups are useless unless you can restore from them.

How long are you planning on having the backups available? Short term (12-24 months?) or long term 5-10 years+
How easily will you be able to get the data off again in 5+ years time?

Imagine you archived your business a few years ago to SCSI drives, you're going to have fun today trying to get them connected up in a hurry.
Optical discs (cds, dvds, blurays etc) can rot & degrade
cloud storage? how can you be sure your data will still be there in 5+ years time

I dont have a perfect answer for you, just to think about the purpose of your backups.
A simple RAIDed NAS solution may be fine for a while and gives multiple longterm connectivity options (USB, ethernet etc)
External HDD will need to be looked after and stored correctly - they are prone to shocks and knocks
Tapes are very slow and also require good handling
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Archive Backup Solutions?
« Reply #9 on: April 01, 2016, 12:23:17 pm »

I'm going to dump a lot of information on this thread, as I have been working out a larger Medium post on the topic.

Backblaze is great, and I use them for my SMB IT clients (day job, though self employed).  They're the only safe bet in cloud backup as they're cash flow positive, without any debt, and will be around for a while.  Their StoragePods are amazing, and if you really want one you can either build it or buy one.  This isn't for home users.  The downside is that a pelican case with 5tb-8tb drives moves data faster than any internet connection can.

Large RAID sets are great, but really, only should be used as on online copy of all data.  RAID is not backup, and if you can afford the 2-4 days to rebuild a drive set, then you've got more time than me.  Tape is a love hate relationship, mostly because while the drive is expensive, the tapes are fairly cheap and unlike hard drives, they're designed to be inactive for a long period of time and work fine.  The downside is you have to restore the tape to a destination in order to use the data on it, and even at the fastest speeds, no one wants to wait that long.

JBOD arrays and software RAID are nice, I highly recommend using the external TB JBOD arrays when dealing with lack of expandable disks (thanks Apple...).  Depending on your comfort levels, ZFS makes life a lot better, though everything has trade offs.  I like Synology NAS boxes, mostly due to having done data recovery on one.  A linux live-cd and a motherboard with enough sata ports and I got my clients data back.

Generally speaking, the whole archive/backup solution starts with your workflow.  When you get to a computer, how do you handle files you've created?  In a perfect world, you'd copy from cf/sd/xd/cfast/ssd to 2-3 places - working space, backup space, archive space.  The working space should be a SSD, be it local or on a network share.  The backup space should your machine backup process - in the Mac world this is your Time Machine setup - so that copy happens automatically in the background (you backup your workstation right?).  The archive space is where one copy of all your data lives, but you never overwrite the files there, think of it as a read only share.

For cloud backup, point it at the root working folder of the SSD and let it do it's thing.  They're offsite, and it just runs.  Make sure your online backup company likes your file sizes and media types - I know some companies won't touch video files without a surcharge.

There are two other copies that need to be maintained, and it's a mostly manual process as they're the 'local offline' and 'remote offline' copy.  When you finish a project, or need to free up working room, you'll need to copy the project folder to 3 locations - archive space, local offline and remote offline.  It's easy to grab a drive off the shelf and put it back (the local offline) but getting a disk from somewhere else is a pain.  Some folks would just have a stack of 500gb/1tb disks ready to copy the data to then ship off site.  Folks with a office/studio away from home can do this on their own schedule.

By using 'copies' you're not relying on a backup software package for file access.  I use Chronosync on the Mac, but there are a few different options out there.  The goal is that worst case, you're doing a single drive recovery with the likes of Ontrak or DriveSavers.  If you want to be redundant, do 2 copies of either the local or remote offline, each on a single brand of drives - there's only 2 really, WD and Seagate.

I'll push further into the Medium document, and will think of 4-5 cases where things could be improved.  The key is to have multiple copies, in multiple places and an easy way to get back up and running.  This is even more critical now due to the crypto ransomware making life for PC users hell.  Imagine your entire portfolio encrypted and held for ransom that you pay in bitcoin.  It's happened at hospitals here in the USA, so keep an eye on things.

-Joe

PS - forgot to post a photo - this is photo of 47.5TB of disk, and doesn't include the 30TB I have in my colocation rack, nor the drives in the safe...
« Last Edit: April 01, 2016, 12:28:34 pm by Joe Towner »
Logged
t: @PNWMF

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Archive Backup Solutions?
« Reply #10 on: April 01, 2016, 12:45:01 pm »

And now that I got that brain dump done, I know I didn't answer the question.  It also depends on the client work.  The workflow above is great for a wedding photographer, where as once you deliver the files/prints/etc, you're 98% done with them.  For others, not so much.  I think the big thing I was trying to get across is the idea of "working copy" - "archive copy" - "local offline" - "remote offline".  Toss on local workstation backup, and cloud backup of your "working copy" folder, and most situations will be covered.

Doing NAS to NAS replication is really hard, in that you're still trying to shove a lot of data over a small pipe.  Unless you're in a space that's got lots of upload speed at both locations, think 10mbps minimum, sneaker-net is your answer.

If I had a software engineer, what you're looking for is a NAS that has a plugin that copies all the 'new' data to a SSD that's in a dock on the top of the machine.  You'd eject the SSD and take it home, where you'd plug it into the dock on your home NAS.  It'd read all the 'updates' from the work NAS and copy the data accordingly.  It'd throw a 'done' report back onto the SSD and when you got up in the morning, you'd eject the SSD and take it back to work.  Repeat daily.

-Joe
Logged
t: @PNWMF

andyptak

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 469
Re: Archive Backup Solutions?
« Reply #11 on: April 02, 2016, 05:56:10 pm »

Like most of you I live in fear of losing my (almost) life's work.

I have my own backup and archive solution which works for me, but the one thing that still nags at me is bit rot.

I have four copies of my archive and I could still lose data because it just wastes away and rots when not in use. I understand that just defragging a disk is not enough to avoid the issue. (I'm a Windows heretic here), so I'm wondering if there is a solution that is not onerous and time consuming? Would running CHKDSK every now and then work? Thanks.
Logged

aaronleitz

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 95
Re: Archive Backup Solutions?
« Reply #12 on: April 03, 2016, 02:17:01 am »

My offline storage/archive is all on bare hard drives. A RAID box handles all the stuff I'm working on and that is backed up to (among other things) a group of bare drives that rotate in and out of the office. I use OWC's very well made Thunderbolt docking station (see attached photo). When the rotating drives are full (usually 2 TB or so) they become "archive" drives. I have two identical sets - one set of drives lives in an anti-static foam insert in my office file cabinet (see attached photo) and the other set lives in a TurtleCase:  http://www.turtlecase.com/hard-drive-cases/039-3-5-hard-drive-20      at a top secret but not-too-much-of-a-hassle-to-get-to-location.

I have around 15TB archived this way. A simpletext list tells me what shoots are on what disk. When I come home from a shoot I add the shoot folder name (20160402_clientname_project) to the list. The list is quick to search and can even be printed out and stored with the drives. Earlier this year I had a client request an image from 2009 - took about 30 seconds to look up the project on the list, pull the drive, pop it in the dock, and have the image up on my monitor.
Logged
Pages: [1]   Go Up