Pages: [1] 2   Go Down

Author Topic: Move Backup Strategy away from Drobos?  (Read 1722 times)

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Move Backup Strategy away from Drobos?
« on: January 01, 2018, 09:21:57 AM »

Here is my current image setup driven by only one computer, a late 2013 15” Retina MBP:
Images split into three categories, each has an associated LR catalog:
  • MASTER: Images that are “keepers”. This is my master catalog ~ 7500 images
  • WIP: Organized by year, i.e. WIP/2016, WIP/2017, etc. This is where images are imported from the field. Each year’s folder contains ~ 5000 images. It is a “temporary” location. Keepers get copied into MASTER. After 2 years the WIP/20XX folder gets moved to BOXED and the catalog gets exported / imported into BOXED.
  • BOXED: Permanent storage for all raw images. ~ 50,000 images.
Main reason for this structure is that I can take the MASTER and WIP images with me on portable SSD drives when I travel (which is about 80% of the time). Those are the working images I would need access to export or work with.

MASTER and WIP images reside on a OWC Mini 4-bay JBOD (RAID 0) with (4) 1TB SSD’s. I have portable copies on (2) 2TB SSD’s that get synced with CarbonCopyCloner. This is where the Drobos come in: MASTER and WIP images are backed up to one drobo, which is then backed up to a second Drobo. Plus, all the BOXED images reside on the first Drobo, backed up on the second Drobo. Each Drobo has (5) 2TB drives. Then about 1/month I copy ALL images onto bare HD’s via a NextStar dock; those drives get stored off-site.

My thinking here is my working images (MASTER/WIP) are on a fast SSD setup, while the BOXED images are only on the Drobos since they represent most of the capacity. I rarely access the BOXED images, so I don’t care if they are slow. The Drobos simply function as back-up. I liked the ability to hot-swap a drive if it fails in the Drobo.

The problem is, my last two failures have not been HD failures but Drobo failures. Last night was the second failure, and that was on the newer Drobo! I need to replace my Drobo-hole, but am wondering about the value of that hot-swap feature. After that long preamble, my three questions:
  • Is there a more reliable option to Drobos that have a hot-swap feature?
  • Does Softraid have an option to configure a JBOD in a similar way so I can hot-swap a failed drive, say 1/4 or 1/5? [Edit: Just realized I have SoftRAIDLight, which only gives me raid 0/1 options. So I could upgrade to SoftRAID and get what I need with RAID 5 or 6 in a 5-bay enclosure. Still not sure if that is any more reliable?]
  • How valuable is the hot-swap feature anyway??

Dave
« Last Edit: January 01, 2018, 09:52:23 AM by dchew »
Logged

degrub

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 748
Re: Move Backup Strategy away from Drobos?
« Reply #1 on: January 01, 2018, 10:15:33 AM »

any Raid level is about availability and recovery from a failure, not backup. You seem to have a reasonable backup strategy and understand the weak points.
Hot swap is there to allow rebuilding of the array without shut down of the system. If you don't need access to the system during the rebuild, then it may not be of operational value. However, shutting down and stopping the disks and then restarting after a disk change out might cause an issue. Low probability, but possible.

Why use the Drobos at all ?  Why not single disks ?
The JBOD volume may be at higher risk of loss of files if the OS does not manage the files across the disks correctly ( ie not distribute the data across multiple spindles).
Logged

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #2 on: January 01, 2018, 10:40:51 AM »

Why use the Drobos at all ?  Why not single disks ?
The JBOD volume may be at higher risk of loss of files if the OS does not manage the files across the disks correctly ( ie not distribute the data across multiple spindles).

That's a really good, obvious option I've ignored. I suppose there is no reason why I couldn't separate them in the same way I separate the off-site bare drives. Is there an enclosure that can house 5 drives with data ports for each, or is it best to just get single housings and stack/daisy chain them? I'd rather not have to hook each one up one-by-one every time I back up. It would probably require 4 or 5 2TB drives for everything.

Dave

Edit: Never mind; I guess I can just put them in a 4-bay enclosure as a JBOD. Thank you. Since I don't have to access this very often it is probably the most sensible thing to do (and cheapest).

« Last Edit: January 01, 2018, 11:01:34 AM by dchew »
Logged

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #3 on: January 01, 2018, 11:26:25 AM »

So the next question: are there any recommendations for 4 or 5-bay enclosures? I don't need the enclosure to control anything like the Drobo did. I just need an enclosure. I can manage the drives with SoftRAID. I just want something that is reliable. It won't even be on all the time for access. It just gets turned on for the backup exercise, and the occasional dig into archived images in the BOXED folders.

Dave
Logged

degrub

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 748
Re: Move Backup Strategy away from Drobos?
« Reply #4 on: January 01, 2018, 11:51:49 AM »

Thunderbolt 2 enclosure ?
i would use that instead of a USB3 enclosure.
Otherwise, what about an old Mac or PC on the network ? just add the drives and set up sharing ?
Logged

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #5 on: January 01, 2018, 12:06:42 PM »

Yes I could go either way, thunderbolt or USB3. I haven't considered network-based since I usually have them turned off and disconnected to extend the life, and I don't have an old spare computer. I might upgrade the MPB this year so that could be an option in the future. I'm out on a limb with one computer anyway.

It's been a while since I investigated HDD's. My head is thinking 2TB drives because that is what I have, but I suppose 8TB drives are pretty common now. I could even get a few bare 8TB drives and just use my docking station. I guess I have a bunch of options. Drobo is not going to be one of them.

Dave
Logged

Joe Towner

  • Sr. Member
  • ****
  • Online Online
  • Posts: 839
Re: Move Backup Strategy away from Drobos?
« Reply #6 on: January 01, 2018, 01:17:13 PM »

Hey Dave,

Yea, I need to add/update my photo storage article, but really you should start moving away from the Drobo's.  8TB external drives were on sale for $150 over the holiday break.  As for bare enclosures for the drives, OWC really has me with their Thunderbay or Mercury enclosures.  Unless you're doing massive amounts of copies or using SSD drives the performance difference between USB3 and TB2 is minimal.

Hot-swap is only for situations where you can't take a reboot - as in a server that's running 24x7 with business critical 'ish.  If you can handle the reboot, or eject/swap disk, it's MUCH cheaper to go that route.

For all intensive purposes, if you're buying disks smaller than 4tb they need to be 2.5" in size.  If you're buying 5tb or more is the only time to think 3.5" drives.  You've got plenty of SSD space, but for what reason?  So you have a 4tb OWC Mini, 2 8tb Drobo's for your two copies of BOXED.  Toss in the 2x 2tb SSD's and you're carrying a lot of files.

Tell me a bit more about your normal workspace.  Since you travel how do you handle backup on the road?  How full are these drives, or is there lots of head room?  What's your home internet connection like?

I MIGHT consider a small NAS, mostly due to your travel schedule.  It would allow full access to everything remotely, though slowly via VPN, even allow for uploads from the road.  I would also consider using it as a point that would have an external usb3 drive or two connected.

-Joe
Logged
t: @PNWMF

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #7 on: January 01, 2018, 01:40:59 PM »

Joe,
Thank you for the response. I kinda ignore backup on the road since trips are short. I carry lots of CF cards so I don’t have to reformat/delete them after import. If I make a change to a legacy file in MSTR or WIP those changes are “backed up”* in the LR database even if the base file is already a tiff. I don’t do much hard editing on the road in PS. For the most part I want access for client response. Exporting a jpeg, that sort of thing.

My problem with a NAS is file size from an IQ3100. Base size is about 150mb, and a converted tiff 2-image pano is 1gb even if flattened. At home I have DSL so upload is painful. No cable down my street :(

I try to keep drives at 50%, but the MSTR portable SSD is now at 75% and growing fast with every tiff exported from C1.

Dave

*I also have the usual TimeMachine backup which covers my LR and C1 catalogs.
« Last Edit: January 01, 2018, 02:32:46 PM by dchew »
Logged

Joe Towner

  • Sr. Member
  • ****
  • Online Online
  • Posts: 839
Re: Move Backup Strategy away from Drobos?
« Reply #8 on: January 02, 2018, 12:40:47 PM »

Dave,
The 100mp files really do take a different method of storage - anything less than SSD is painful.  One thing I'm working towards is the newer Nbase-T networking - 2.5/5/10gbps.  I added the 2x M.2 SSD cache to my Synology and though it's limited to 4x1gbps ports, I'll trunk them to get faster.  The caching helps with the slower SATA disks, but I'm still testing on a new MBP as to the real impact and effectiveness when doing imports and stitching.  Only got 512gb internal space on the new laptop, so the issue will force itself.  I'm planning some RED work with their 8k cameras, so even more giant file fun.

Promise has a TB3-Nbase-T adapter (SanLink3 SL3T1GL).  One other is the QNAP that basically embeds a TB2/3 nic in NAS may work for others.

-Joe
Logged
t: @PNWMF

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2553
Re: Move Backup Strategy away from Drobos?
« Reply #9 on: January 02, 2018, 04:33:18 PM »

Joe - if you want some portable high speed working storage for your MBP you could do a lot worse than this:

http://www.orico.cc/goods.php?id=6451

No external power, very fast - capacity depends on your budget but you won't get much faster in an external and portable (no additional power needed) option.

Frankly, for working space (rather than general mass storage) it's bloody good regardless of being portable or otherwise.
Logged
Phil Brown

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #10 on: January 02, 2018, 10:00:58 PM »

Thanks Joe and Phil,

Wow you guys are a whole other level of fast. I was happy with my Samsung 2TB T3 at a measly 400-ish mb/s! It would be nice to gang a few of those orico's together while on the road, but assuming they don't daisy chain I'm probably limited to (2) 1 TB M.2 cards in one housing. That would probably be enough for the next year or so. I could use that for the current year's WIP image folder, then use my existing Samsung for the MSTR which I don't access as much. I am not sure what the internal limit is of my MBP, but I am sure it is often the bottleneck. Looks like 2018 will be the year of computer upgrades...

Back to the backup failure, sounds like I should just get a standard thunderbolt enclosure and a few 8TB drives.

Dave
« Last Edit: January 02, 2018, 11:49:20 PM by dchew »
Logged

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2553
Re: Move Backup Strategy away from Drobos?
« Reply #11 on: January 02, 2018, 10:09:12 PM »

It would be nice to gang a few of those orico's together while on the road, but assuming they don't daisy chain I'm probably limited to (2) 1 TB M.2 cards in one housing. That would probably be enough for the next year or so. I could use that for the current year's WIP image folder, then use my existing Samsung for the MSTR which I don't access as much.

You can't daisy chain them.  You can get larger storage, but it needs an external power supply and you move away from M.2 format.  That's fine if that's what you need/want, the benefit of these is their extreme portability and speed, with decent capacity (subject to wallet-strain).  Of course, you can dump them to another storage option periodically and just keep what you need to be portable with you plus space for imports.  There are probably other manufacturers of similar products, but I've found Orico to perform well and be reliable.
Logged
Phil Brown

Joe Towner

  • Sr. Member
  • ****
  • Online Online
  • Posts: 839
Re: Move Backup Strategy away from Drobos?
« Reply #12 on: January 02, 2018, 10:16:20 PM »

Damn Phil - you reading peoples minds?  Only request I would have is 4x M.2 slots, not just 2 ;)  Or a TB3 - PCIe Bridge - 4x NVMe devices.

DM2-RC3 is $100 and $300 per 1tb M.2 drive, that's some mighty fast and small storage!
Logged
t: @PNWMF

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #13 on: January 02, 2018, 11:55:28 PM »

I'm a little confused on where to buy these from. I found them on aliexpress, with a 20-38 day delivery time...?

Couldn't find them at the normal places here in the states like B&H or Amazon.

Dave
Logged

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2553
Re: Move Backup Strategy away from Drobos?
« Reply #14 on: January 03, 2018, 12:28:34 AM »

Dave - somewhere like NewEgg maybe?  I'm in Australia, so my knowledge of US vendors is limited.  There does seem to be some Orico stuff on Amazon, but this might be a little new/specialised.  You could email them at [email protected], too - that might be a good start.
Logged
Phil Brown

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2553
Logged
Phil Brown

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #16 on: January 03, 2018, 06:02:42 AM »

Many thanks Phil!

Dave
Logged

dchew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 903
    • Dave Chew Photography
Re: Move Backup Strategy away from Drobos?
« Reply #17 on: January 05, 2018, 08:08:19 PM »

Update:
Ordered a few OWC 2-bay Thunderbolt enclosures. Received them today and put the drives in as a JBOD; fired them up no issues. Good thing I did it right away because I think my other drobo is dying. CCC hangs sometimes and the OS gives me a drive error. Try again and it works. Should have ordered 4 enclosures instead of 2.  :(  I wonder if part of my drobo problem is using SoftRAID(light)? I can't imagine drobos fail as often as I'm seeing. My use pattern is weird too: Turned off for extended periods then on for hard use in a few hours, then off again. Maybe that contributes. No drive failures, just the drobo itself.

Just glad I have a backup strategy that seems to be working, so far...

I also ordered that orico M.2 housing; will be here in a few weeks. I can see what a really fast drive does for me. Thanks again for everyone's help.

Dave
« Last Edit: January 06, 2018, 03:30:58 AM by dchew »
Logged

vjbelle

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 428
Re: Move Backup Strategy away from Drobos?
« Reply #18 on: January 07, 2018, 02:36:32 PM »

Dave, I am a little late to this thread but find the Orico very interesting - but expensive to populate.  I've never read much good about Drobo and have a 15TB Synology NAS set up that has never given me a single hiccup.  I like that I can access anything on the Nas from anywhere in my house as I have lots of computers - PC and MACs. I use my PC as my main Photoshop computer (a very recent build) and find that I really don't need much storage on it.  Once an image has been completed to my liking it goes off the PC and onto the NAS.  The NAS also has to be backed up which is accomplished with another 2 drive Western Digital setup. All of this stuff is fairly slow except for my main computer which is blazing fast..... but that's the only place I need speed.  On the road I use a 1TB T5 Samsung which I has never been lacking for storage space but your needs may differ.  I use a very recent MacBook Pro 13 inch as my travel computer and like it a lot.... its plenty fast and I can easily work on 100MP files if I want to.  But..... I usually downsize those files to 50MP just out of habit from working on a much less powerful MacBook.  That's easily enough size for me to evaluate whether or not something needs attention.

Please add some feedback about the Orico and how you intend to populate it. 

Cheers......

Victor
 
Logged

D White

  • Full Member
  • ***
  • Offline Offline
  • Posts: 108
  • Don White
    • Alamy link
Re: Move Backup Strategy away from Drobos?
« Reply #19 on: January 21, 2018, 11:04:55 AM »

What is not clear to me on this is thread; did the Drobo unit itself fail or the discs in it? Two very different scenarios.
Logged
Dr D White DDS BSc
 https://www.alamy.com
Pages: [1] 2   Go Up