Luminous Landscape Forum

Equipment & Techniques => Computers & Peripherals => Topic started by: tonysiciliano1 on July 03, 2018, 02:10:40 pm

Title: Online backup of 6 TB of photos. How?
Post by: tonysiciliano1 on July 03, 2018, 02:10:40 pm
I have 6 TB of photos that (in addition to my backups onto external hard drives) I would like to back up online using something like Backblaze. The problem is my upload max speed is 10-15 mbps so I might die before I get the photos all uploaded. Anybody know of any workarounds for this problem? I'm wondering if you can send your files on a hard drive to any sites like Backblaze. Suggestions?
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 03, 2018, 02:53:08 pm
I have 6 TB of photos that (in addition to my backups onto external hard drives) I would like to back up online using something like Backblaze. The problem is my upload max speed is 10-15 mbps so I might die before I get the photos all uploaded. Anybody know of any workarounds for this problem? I'm wondering if you can send your files on a hard drive to any sites like Backblaze.

You can, at least in the case of BackBlaze:

https://www.backblaze.com/blog/introducing-backblazes-rapid-ingest-service-fireball/

https://www.backblaze.com/blog/cost-data-of-transfer-cloud-storage/

Jim

Title: Re: Online backup of 6 TB of photos. How?
Post by: tonysiciliano1 on July 03, 2018, 11:05:49 pm
Thanks for the info. I checked BackBlaze rapid ingest service fireball service. It costs $550 to rent the device to put your data on and mail back to them. That would make sense if you had a lot of data, but for 6 TB it's too expensive. Oh well, a nice thought....
Title: Re: Online backup of 6 TB of photos. How?
Post by: Farmer on July 04, 2018, 01:52:17 am
Thanks for the info. I checked BackBlaze rapid ingest service fireball service. It costs $550 to rent the device to put your data on and mail back to them. That would make sense if you had a lot of data, but for 6 TB it's too expensive. Oh well, a nice thought....

How much would it cost you if you lost your data?
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jeremy Roussak on July 04, 2018, 03:57:25 am
Thanks for the info. I checked BackBlaze rapid ingest service fireball service. It costs $550 to rent the device to put your data on and mail back to them. That would make sense if you had a lot of data, but for 6 TB it's too expensive. Oh well, a nice thought....

Many of us would think 6TB was quite a lot of data! Anyway, the purpose of backup is ability to restore, not just to provide peace of mind from the though that "it's somewhere else as well". How long would it take you to download 6TB form an online backup if you lost it?

Jeremy
Title: Re: Online backup of 6 TB of photos. How?
Post by: elliot_n on July 04, 2018, 05:49:19 am
I have 6 TB of photos that (in addition to my backups onto external hard drives) I would like to back up online using something like Backblaze. The problem is my upload max speed is 10-15 mbps so I might die before I get the photos all uploaded. Anybody know of any workarounds for this problem? I'm wondering if you can send your files on a hard drive to any sites like Backblaze. Suggestions?

My upload speed is 18mbps and it took me 5 weeks to backup 6TB of photos to Backblaze.

Re. restoring data. Backblaze will send you the data on a hard drive.
Title: Re: Online backup of 6 TB of photos. How?
Post by: Joe Towner on July 04, 2018, 10:26:40 am
Honestly, 6TB is nothing - as I do a 30TB replication between 2 NAS's  ;)  The trick is in Backblaze exclude some folders - as in anything shoot more than 6 months old.  Once it has the most recent stuff uploaded, then start removing the exclusions working from newest to oldest.  That way the most important, valuable shots get uploaded first.

Or head down to the library - or any other place that has faster upload speeds.
https://www.nbcbayarea.com/news/local/San-Francisco-Public-Library-Nations-Fastest-Internet-Access-311586421.html
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 04, 2018, 10:44:00 am
My upload speed is 18mbps and it took me 5 weeks to backup 6TB of photos to Backblaze.

I have somewhat more than that, and it took a couple of weeks to do the upload over a 50 Mb/s ATT IPFlex connection (theoretical time for 6TB is 11.1 days). Backblaze supported that transfer speed: I was getting 6+ MB/s speed reports from GoodSync, and everything else, including HD Netflix streaming, worked just fine during the upload. However, every 24 or 36 hours the Backblaze server would stop responding and I'd have to re-click "Analyze and Synch" in GoodSync.

Jim
Title: Re: Online backup of 6 TB of photos. How?
Post by: PeterAit on July 04, 2018, 12:00:17 pm
Why do you need online? I assume you are worried that a fire or tornado or burglary will happen? Why not back up onto hard disks yourself and make sure the disks are secure? I use our safe, which will survive pretty much anything. Or, a safe deposit box at a bank.
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 04, 2018, 01:40:35 pm
Why do you need online? I assume you are worried that a fire or tornado or burglary will happen? Why not back up onto hard disks yourself and make sure the disks are secure? I use our safe, which will survive pretty much anything. Or, a safe deposit box at a bank.

I do that, too. Can't have too many copies...

Jim
Title: Re: Online backup of 6 TB of photos. How?
Post by: rdonson on July 04, 2018, 07:00:34 pm
Does your bank say that your safe deposit box is waterproof or fireproof?  Is it insured? 

Title: Re: Online backup of 6 TB of photos. How?
Post by: ned on July 04, 2018, 07:03:57 pm
Just keep your computer on and let it run. I uploaded about 2.5tb to black blaze that way.


Sent from my iPhone using Tapatalk
Title: Re: Online backup of 6 TB of photos. How?
Post by: BobShaw on July 04, 2018, 08:08:37 pm
I do that, too. Can't have too many copies...
That is the key. I have a PO Box and just keep one in there also. Safes are good but "fireproof" safes are only fireproof for documents which burn at a much higher temp than hard drives melt.
To me in Australia on line systems are useless. It would take 6 months to upload my data
Title: Re: Online backup of 6 TB of photos. How?
Post by: bassman51 on July 04, 2018, 10:53:42 pm
Just because it will take a “long” is no reason to abandon the idea.    Make a complete backup (or two) on external drives, and store them in two different locations.  Then start backing up new data you create and files you change today and going forward at Backblaze. This will rapidly reach stasis, and you will have all of your data offsite.  Then start adding the older data to Backblaze.   Eventually, it will finish. 
Title: Re: Online backup of 6 TB of photos. How?
Post by: Alan Klein on July 04, 2018, 11:11:01 pm
If you backed up two external drives and kept one in your house and one in your neighbor's or relatives, what would be the odds both houses would burn down at the same time.  Is the cloud any more secure?
Title: Re: Online backup of 6 TB of photos. How?
Post by: fdisilvestro on July 04, 2018, 11:50:34 pm
If you backed up two external drives and kept one in your house and one in your neighbor's or relatives, what would be the odds both houses would burn down at the same time.  Is the cloud any more secure?

Prior to 9/11, there were companies in the WTC that had their backup in the other tower ...
Title: Re: Online backup of 6 TB of photos. How?
Post by: kbolin on July 05, 2018, 12:20:55 am
I created my own Cloud backup using two QNAP NAS's.  I had both onsite for the initial data load and then packed up the second and installed it at my son's home.  Initial cost is higher but I have no monthly costs and I control everything with the hardware and network connections.  Using RAID10 I would need two hard drives in the NAS to fail to lose any data.  Oh, and my son is 10 hour drive away so I'm pretty sure that no fire or flood will hit both homes at the same time. 

I know it's not what you were asking but I contemplated the same before I went this route. 
Title: Re: Online backup of 6 TB of photos. How?
Post by: TommyWeir on July 05, 2018, 03:04:40 am
For offsite, the appeal of Backblaze is that you don't have to think about it or arrange anything.  Set it off and away it goes.   50 bucks a year is hard to beat cost wise.  You haven't had online backup for the past few months, at least you will know over the next few months it will be taken care of.
Title: Re: Online backup of 6 TB of photos. How?
Post by: bassman51 on July 05, 2018, 09:49:57 pm
If you backed up two external drives and kept one in your house and one in your neighbor's or relatives, what would be the odds both houses would burn down at the same time.  Is the cloud any more secure?

Sadly, any backup process which requires manual intervention everyday is likely to fail because we are lazy.  The beauty of a cloud-like solution - whether it’s a commercial cloud or one you set up yourself - is that it runs every day (or hour) with no intervention. 
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 06, 2018, 12:16:37 am
For offsite, the appeal of Backblaze is that you don't have to think about it or arrange anything. 

For any backup system, I recommend periodic trial restores. Set and forget is not a good strategy, IMO.

Jim
Title: Re: Online backup of 6 TB of photos. How?
Post by: TommyWeir on July 06, 2018, 02:19:37 am
I never advocated forget...

And agreed always do trial restores.   There's always World Backup Day (http://www.worldbackupday.com/en/) which we use to test each of ours.
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 06, 2018, 11:22:08 am
I never advocated forget...

And agreed always do trial restores.   There's always World Backup Day (http://www.worldbackupday.com/en/) which we use to test each of ours.

Maybe I got the wrong idea when you wrote: "...don't have to think about."

Jim
Title: Re: Online backup of 6 TB of photos. How?
Post by: TommyWeir on July 06, 2018, 04:33:10 pm
You did.

My comparison was with managing offsite drives in other premises.
Title: Re: Online backup of 6 TB of photos. How?
Post by: davidgp on July 07, 2018, 10:08:46 am
Sadly, any backup process which requires manual intervention everyday is likely to fail because we are lazy.  The beauty of a cloud-like solution - whether it’s a commercial cloud or one you set up yourself - is that it runs every day (or hour) with no intervention.

Yes... if the process relies in a human remembering to do something... it is going to fail.


http://dgpfotografia.com
Title: Re: Online backup of 6 TB of photos. How?
Post by: bassman51 on July 07, 2018, 04:56:01 pm
I never advocated forget...

And agreed always do trial restores.   There's always World Backup Day (http://www.worldbackupday.com/en/) which we use to test each of ours.

Sadly, I’ve restored from Crashplan more than once.  Happily, it succeeded each time.
Title: Re: Online backup of 6 TB of photos. How?
Post by: TommyWeir on July 08, 2018, 02:30:16 am
The one truth... drive failure is inevitable, whether through internal or human failure.  There's simply the question of how will you restore.

I've been there more than once.  It was interesting to me restoring my MacBook Pro last year how much of my data lies in online services now.  Dropbox and iCloud hold all of my documents, my photographs and video and other media all on Backblaze, in addition to local drives.  I had not done a clean install of MacOS for well over a decade, simply upgrading as I went so I took the opportunity to not do a simple Time Machine restore and instead do a clean install and let the files download from iCloud and Dropbox.   The main pain point was dealing with automation which I had to reconstruct.
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 08, 2018, 10:27:58 am
Sadly, I’ve restored from Crashplan more than once.  Happily, it succeeded each time.

My experiences with Crashplan were less satisfactory. Upload speeds were fastish, at about 15 Mb/s, at first, then incredibly slow as the backup continued. After 5 months, about halfway through a 6 TB upload, the app hung and couldn't be restarted. Uploading more data than that in a couple of weeks with Backblaze was a relief.

Jim
Title: Re: Online backup of 6 TB of photos. How?
Post by: elliot_n on July 08, 2018, 10:43:14 am
I also gave up on trying to upload 6TB to Crashplan - it was just too slow. I switched to Backblaze and my upload speeds were ten times faster.
Title: Re: Online backup of 6 TB of photos. How?
Post by: MattBurt on July 19, 2018, 06:18:48 pm
I set up my own offsite "cloud" backup.
I have close to 8TB I want an offsite backup of and what I did was buy a cheap external 8TB drive and used SyncToy (free download from Microsoft (https://www.microsoft.com/en-us/download/details.aspx?id=15155)) to make a copy of all that data right from my PC to the external drive.
Then I took that drive to work and connected to my computer there and set up another free app called Resilio Sync (https://www.resilio.com/platforms/desktop/). Sync uses Bit Torrent technology to sync folders across the Internet. The free version will do the trick but I upgraded for a few nice features like bandwidth throttling.
Since I had copied all my data already, it just had to index everything (took a couple of days) and now it just copies any new files or changes I make at home to the offsite drive at my office. Works great and no monthly fees!

You could also set this up at a friend's or relative's house if you have someone who could host it for you and you don't have an office outside of your home.
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jeremy Roussak on July 20, 2018, 03:59:07 am
Matt, thanks for the link - resilio looks interesting. I have a similar setup, backing up to my mother's iMac nightly using ChronoSync as well as using Dropbox and Time Machine. Ever since I lost three weeks' work when I was an undergraduate, I've been obsessional about backups. My setup provides that within 24 hours (because I have set ChronoSync to run at 0030), anything I have done is stored on three computers' internal drives, three TM backups and in Dropbox. That satisfies even my neuroticism.

Jeremy
Title: Re: Online backup of 6 TB of photos. How?
Post by: Joe Towner on July 20, 2018, 02:12:12 pm
Matt's setup works, as long as folks are aware that any file overwrites & corruption on the source computer are transmitted automatically to the offsite one.  Something that allows versioning on the remote side - like anything overwritten or changed in the previous 14 days is recoverable would be huge.  Also, alerting as to a sync that didn't happen.
Title: Re: Online backup of 6 TB of photos. How?
Post by: MattBurt on July 20, 2018, 04:32:38 pm
Matt's setup works, as long as folks are aware that any file overwrites & corruption on the source computer are transmitted automatically to the offsite one.  Something that allows versioning on the remote side - like anything overwritten or changed in the previous 14 days is recoverable would be huge.  Also, alerting as to a sync that didn't happen.

Like Jeremy I'm also a bit of a backup fanatic. I haven't lost any significant data but working in IT have seen it happen to others many times. I hate telling people their data will need to be restored from backup when I know they probably don't have one.

What I outlined was my offsite backup plan which is one component of my whole approach. I also have a home server running the open source (free) application URBackup. That backs up all the computers in my household on an ongoing basis, keeping multiple versions of changed files as well as full images to quickly recover from a drive failure. The offsite is a second line of defense in case of fire, flood or other catastrophic situation.
 8)
Title: Re: Online backup of 6 TB of photos. How?
Post by: Joe Towner on July 20, 2018, 04:55:32 pm
Like Jeremy I'm also a bit of a backup fanatic. I haven't lost any significant data but working in IT have seen it happen to others many times. I hate telling people their data will need to be restored from backup when I know they probably don't have one.

What I outlined was my offsite backup plan which is one component of my whole approach. I also have a home server running the open source (free) application URBackup. That backs up all the computers in my household on an ongoing basis, keeping multiple versions of changed files as well as full images to quickly recover from a drive failure. The offsite is a second line of defense in case of fire, flood or other catastrophic situation.
 8)

Defense in depth - nicely done!
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jim Kasson on July 20, 2018, 05:23:59 pm
Matt's setup works, as long as folks are aware that any file overwrites & corruption on the source computer are transmitted automatically to the offsite one.  Something that allows versioning on the remote side - like anything overwritten or changed in the previous 14 days is recoverable would be huge.  Also, alerting as to a sync that didn't happen.

And setting the maximum change that will propagate over...

Jim
Title: Re: Online backup of 6 TB of photos. How?
Post by: Jeremy Roussak on July 21, 2018, 04:18:48 am
Matt's setup works, as long as folks are aware that any file overwrites & corruption on the source computer are transmitted automatically to the offsite one.  Something that allows versioning on the remote side - like anything overwritten or changed in the previous 14 days is recoverable would be huge.  Also, alerting as to a sync that didn't happen.

You're right about the utility of versioning, and that's something that ChronoSync doesn't offer (it will move deleted files to a temporary folder for a while, but that's obviously not quite the same thing). Time Machine does versioning, of course, and since there's a TM drive attached to my mother's computer (the remote backup machine) I get versioning that way. It's tricky to use unless I go round to her flat myself, of course, but it's only a mile away.

CS sends me an email after each run, listing files which have been copied or deleted; and it will email me about any sync which hasn't worked. Since I get those emails daily, I notice if they don't appear.

Like Jeremy I'm also a bit of a backup fanatic. I haven't lost any significant data but working in IT have seen it happen to others many times. I hate telling people their data will need to be restored from backup when I know they probably don't have one.

What I outlined was my offsite backup plan which is one component of my whole approach. I also have a home server running the open source (free) application URBackup. That backs up all the computers in my household on an ongoing basis, keeping multiple versions of changed files as well as full images to quickly recover from a drive failure. The offsite is a second line of defense in case of fire, flood or other catastrophic situation.

Thanks for that link, Matt. I'll investigate.

Jeremy
Title: Re: Online backup of 6 TB of photos. How?
Post by: Farmer on July 21, 2018, 05:42:45 pm
If you use Dropbox Plus you can enable versioning for up to 1 year.

For Crashplan users you can set quite detailed versioning, including backup frequency from every 15 minutes to once a day, then versioning allows you to set for one week, after one week, after 90 days, after a year, and deleted files with various timings (or in the case of deletions including "never").