End of RAID?

[John.Murray]

from the OpenStorage Summit:

http://www.zdnet.com/blog/storage/the-end-of-raid/1154?tag=mantle_skin;content

I've experienced recent RAID rebuild times taking longer than productivity lost by simply restoring a recent backup.  Interesting times ahead ...

[Jack Flesher]

I would agree that RAID *5* or even *6* is no longer viable for most photographers as drives are so cheap now that having a RAID 0-1 array --- RAID-0 for speedy reads and writes RAID-1 (mirrored) to another array for redundancy --- is probably the more advisable path.  I recently spoke with a client who had one 2TB drive in a 4x2TB RAID-5 array fail, and after installing the replacement drive it took 8 days to rebuild!  By comparison, copying 4TB of data from one array to a new, empty array over eSATA, Gig LAN, or even FW800 would probably finish in around 24 hours...  

[Schewe]

I've had several less than optimal situations involving proprietary RAID 5/6 setups. In one, the host card manufacturer went out of biz and when 1 port of the card went down the contents of the RAID 5 was no longer secure and I couldn't replace the card without loosing the contents. I've also had issues trying to move RAID/cards from one computer to another.

My current solution is a pair of 6 drive Burly Storage units using the Disk Utility ability to create RAID 0. Thus 6 2TB drives are arrayed and in identical units with Array 1 being backed up to Array 2 nightly. Not satisfied with 2x redundancy, I also backup to a 3rd system, a DroboPro–which is indeed a proprietary array structure (that I'm not "in love" with but find expedient). I'll be the first to admit that 3X redundancy may be sufficient, but I'm still looking for yet a 4th method of backup that can handle the volume that I need...I'm disinclined to use another proprietary solution although I may end up getting a NAS for offsite storage (I can place the unit next door at my house).

[bradleygibson]

Wow.

My 3TB RAID-6 array (5x1TB drives) rebuilt from a failed drive overnight -- ~8hours.  8 days for a rebuild?!  Ouch.  (I should add that the array was also available for reading and writing the whole time, so there was no real downtime, only diminished read/write speeds.)

Not sure about your client, but I recommend hardware RAID--I am using an Areca 1680ix RAID controller.

I would agree that RAID *5* or even *6* is no longer viable for most photographers as drives are so cheap now that having a RAID 0-1 array --- RAID-0 for speedy reads and writes RAID-1 (mirrored) to another array for redundancy --- is probably the more advisable path.  I recently spoke with a client who had one 2TB drive in a 4x2TB RAID-5 array fail, and after installing the replacement drive it took 8 days to rebuild!  By comparison, copying 4TB of data from one array to a new, empty array over eSATA, Gig LAN, or even FW800 would probably finish in around 24 hours...  

[Jack Flesher]

Not sure about your client, but I recommend hardware RAID--I am using an Areca 1680ix RAID controller.

This was a 4-drive hardware RAID-5 internal on a current gen MacPro with RAID controller card.  

I can understand that a RAID-6 should rebuild faster due to the dual redundancy of the error volumes.  My point is that given how inexpensive large drives are now, it seems the 2 or 3 drive RAID-0 being mirrored to an identical onsite RAID-0 remains an operably fast, simple, secure and easy to manage structure for most users.  Even the DROBO R-5/6 is a questionable solution given how long it takes to rebuild itself in the event of an upgraded or replaced drive, as it's a lot simpler to just reformat it and copy everything over fresh from your main storage volume; in which case a JBOD box could serve just as well...

[John.Murray]

@Bradley:

The most recent failure was Dual Channel SAS Controllers connected to a Promise vTrack Cabinet containing representing 15TB of storage capability.  It was much quicker to add in additional drives and restore from the previous night's backup

[ErikKaffehr]

Hi,

I'm clearly pro RAID5. That said what Jack says makes a lot of sense. With RAID5 we also have two additional issues:

- The RAID controller is a single point of failure. If it fails everything is lost.
- Weather 8 hours or 8 days, the array takes significant time to rebuild. It's recommended that identical drives are used, so if the drives are truly identical they may reach end of life about the same time. If another drive fails during rebuild the data are lost

I use RAID5 for those files that I don't want to loose but don't want backup, like panorama work.

Best regards
Erik

This was a 4-drive hardware RAID-5 internal on a current gen MacPro with RAID controller card.  

I can understand that a RAID-6 should rebuild faster due to the dual redundancy of the error volumes.  My point is that given how inexpensive large drives are now, it seems the 2 or 3 drive RAID-0 being mirrored to an identical onsite RAID-0 remains an operably fast, simple, secure and easy to manage structure for most users.  Even the DROBO R-5/6 is a questionable solution given how long it takes to rebuild itself in the event of an upgraded or replaced drive, as it's a lot simpler to just reformat it and copy everything over fresh from your main storage volume; in which case a JBOD box could serve just as well...

[Jack Flesher]

There are lots of storage strategies for sure.  I find as I get older, keeping things simple is easiest. And with drives so inexpensive now, I save everything in my normal file structure: my images are arranged by YEAR/SHOOT or CLIENT NAME/DATE (as required)/, and then I add a subfolder for OUTPUT FINALS which contains all processed files including any blends for Pano, HDR or Focus stacked images with name extensions appropriate to the type.  I back that volume up 100% onsite and another 100% offsite.  Heck, I find drives cheap enough I don't even delete my "junk" images from the shoot folders anymore, and my MF raw files are big --  never know when I might need to steal a patch-piece from one for an adjacent keeper file LOL!   

[Steve Weldon]

I'm in the drives are cheap camp.

I've had hardware RAID's for years (Promise Tech 8 port cards) and they usually worked fine until a hardware failure.  This is when they're supposed to shine, but they always seemed to be more trouble than they're worth to rebuild.

Today.. 2tb drives are so cheap.. I have 8 in my station, 4 I use, 4 clones of the ones I use.  At any time I can use the disk manager to be right back up and working in just minutes.. without even powering the system down or rebooting.

As a third copy they're backed up via Ghost 15 to my NAS array..  This copy would take time to restore, but not as much rebuilding a RAID array.

My archives are stepped from year, location, date for personal stuff.. or year, client, date for work stuff.  Of course they're keyworded.

Right now I'm torn.  My original plans were to shop for a sale of 2.5 inch laptop drives, put each years work on each drive, and store them in a fireproof safe as a last resort.  I was all set to do this when I started using Amazon S3 storage to relieve the storage on my website.  It's so fast and so cheap.. I'll probably choose to go this route.  It will take a few weeks to get the initial uploads completed, and then I'll either use Ghost in network mode or what I'd really like is a network capable clone tool so the files are individual stand alone files and not back up files.  I'll probably continue to use Amazon S3 for a few more months and if I remain impressed I'll go that route.  I recently copied 500gb from S3 to my local workstation as a test and it maintained the full speed my bandwidth allowed which I wasn't using.. it averaged 12-20mbps.   Pretty zippy..

Of course I could do the 2.5 inch laptop drives as a off-site fireproof safe last resort AND Amazon S3.. 4 copies is almost prudent if you've suffered the losses I have through the years.  If it was possible to screw up and/or have things go wrong, I did each and every one of them..

Still, I can't help but feel cloud storage is the future with bandwidth increasing every year.. 50mbps is common (I can even get that here in Bangkok) in the states and they're talking 100mbps being the norm in a years time.  That's faster than my current 2tb drives in my local case!

[John.Murray]

  I think the paradigm itself is beginning to crumble.  In my recent example, the customer insisted on all data (medical imaging) to be contained on a single volume, even though, being database driven, it was completely possible to spread the dataset across multiple volumes - inertia, and nothing else drove the original choices.....

  For my storage needs, i've settled on a single Lightroom Catalog pointing to a number of disks containing the images.  Strangely enough I came to this conclusion after viewing Michael and Seth's "Where the &^%& are My Pictures" series.

  This eliminates the need for that "one big honking drive".  It does complicate my backup strategies for sure, but on the other hand, the complexities of dual-channel, battery backed up, RAID setups are eliminated; a failure of a single drive is just that - it no longer puts *all* my data at risk.

[tived]

End of RAID ? What the F***?! Not yet!

We are now just raiding SSD drives, faster and faster, next if not now, is raid-0's of PCIe Storage.

We use raid for other things then storage, to increase bandwidth in area such as your scratch disk
but for storage I would still use raid but I would move away from your RAID 3,5 and 6 in their various incarnations and
stick to the Stripe and Mirror (RAID 0-1) as I think Jack is pointing out.

There is no doubt that when the shit hits the fan with a raid system it really does sucks, but if done as above stripe and mirror
you are still running when one of the stripes fails.

Something to think about when building raid arrays, in particular the 0-1's is two use two sets of similar but different disks, to avid
wholesale damage to a series of disks, where more then one disk dies, as has been the case in the past where a while family of disks
need to be replace due to faulty firmware.

It is of course as mentioned here a really good idea to have an external backup.

Henrik