Pages: [1]   Go Down

Author Topic: Another RAID question  (Read 1440 times)

Eric Brody

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 487
    • http://www.ericbrodyphoto.com
Another RAID question
« on: April 01, 2019, 12:01:16 pm »

I already partially asked this question but as I continue to try to think it through, I'd like to rephrase and re-ask it if my LULA colleagues will permit.

The four 8 TB drives in my Thunderbay IV (Thunderbolt 2) are getting close to full. It’s attached to my 2013 Mac Pro (64GB OWC RAM, 1TB SSD).
In the Thunderbay IV, I have a “Photos” drive, two backups of it and a backup of my SSD. All backups are done each night with Carbon Copy Cloner. I have other off-site backups of the photos and the boot SSD.
I have some older drives purchased as I gradually worked myself up to the 8's. I like to have all my photos in one place, with multiple backups of course.

Would a RAID (eg 3 6TB drives using RAID 5 [Soft Raid] giving a 12TB capacity) be faster than a single drive or should I bite the bullet and get some larger drives, eg 10 or 12 TB? My other option, as I spend endless hours thinking this through, is just a plain RAID 0, I know this provides no protection against disk failure, but my current system doesn't either.
At this point I’d prefer to do whatever will be faster for Photoshop saves. I’ve done a bit of web searching and it is not clear to me whether RAID 5, or any RAID configuration, would be significantly faster than a single disk for PS saves. I could get a new Thunderbay 4 (Thunderbolt 3) with Soft Raid and keep all my backups in the boxes even if I used one of the Thunderbays for the RAID. This would be forwardly compatible with whatever my next computer would be (iMac 2019, or new modular Mac Pro if it’s affordable).

With RAID 0, should my "Photos" disk fail, I'd be left with my backups the oldest of which would be from last night. A RAID 0 should be faster than my current system which writes my Photoshop files to the "Photos" disk. I do not earn money with my photos, I'm an avid amateur, a retired bum who puts almost as much energy into my photo passion as I did to my previous profession. The loss of even a full day's work would be frustrating but not tragic.

Thanks to all.

Eric
Logged

MattBurt

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3912
  • Looking for that other shot
    • Matt Burt Photography
Re: Another RAID question
« Reply #1 on: April 01, 2019, 01:22:27 pm »

RAID 5 does not give any kind of performance increase over a lone drive, it is purely for redundancy.
Hardware RAID is generally faster than software so if you want to increase performance while maintaining redundancy, adding a dedicated card to handle it might help but it may not be enough to make it all worth it.

My solution to a similar issue is to just keep the RAW files on a lone drive and back that up nightly to my own server as well as mirroring to an offsite backup server for an offsite backup too.
My OS, apps, and Lightroom previews are all on my SSD which keeps things pretty quick once the previews are generated (which I do on import).
« Last Edit: April 01, 2019, 01:58:02 pm by MattBurt »
Logged
-MattB

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Another RAID question
« Reply #2 on: April 01, 2019, 01:39:50 pm »

Hey Eric,

A small RAID5 set will outperform a single hard drive.  It's about the mechanics of it - the mechanical arm and head inside the drive can only perform so fast. When multiple drives are working together, you see a performance increase.  The catch is in the writes - as in for RAID1 or RAID10, you have to make 2x the writes (N+N) compared to RAID4/5 which are N+1, while RAID6 is N+2.  RAID10 was huge when the RAID chips were the bottleneck calculating parity info.

https://www.softraid.com/pages/features/raid_levels.html#raid_speeds_chart

Nothing will hold a candle to the speed of SSDs, which is why I tend to recommend folks look to those as a 'working' drive, while platter drives are great 'storage'.  If absolute speed is the goal, I'd look at the Thunder6 enclosure, use the M.2 slot on the bottom for an 'active' space, and then do a RAID set on a set of drives in it.  If you're building it with 6tb drives & RAID5, I'd actually go a step further and say do 4 of them, as you've already paid for the parity, that extra disk is all free space.  The catch is how CCC would 'backup' a 16tb volume to 3 8tb drives (or do you have to setup a second RAID5 volume and then backup to it).

If you go the route of the RAID0, you could increase your protection by changing the frequency from daily to hourly or 15 minutes.  The amount of data that would be lost would drop, though it could impact system performance during the backups.

Keep asking questions, there's lots of knowledge and every situation is different.
Logged
t: @PNWMF

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Another RAID question
« Reply #3 on: April 01, 2019, 01:51:52 pm »

RAID 5 does not give any kind of performance increase over a lone drive, it is purely for redundancy.
Hardware RAID is generally faster than software so if you want to increase performance while maintaining redundancy, adding a dedicated card to handle it might help but it may not be enough to make it all worth it.

Sorry Matt, both of these are false.  RAID5 adds performance compared to single drives, purely on mechanics. A hard drive on it's own has but one arm, and can only read/write at one location.  Software RAID beats hardware RAID - been this way for over 15 years.  I'll dig out the Youtube talk about how they figured out hardware RAID really was holding them back since the kernel could only talk to 1 device, and that device was I/O bound by it's interface.  No one uses hardware RAID in the big storage arrays, it's all software RAID.
Logged
t: @PNWMF

faberryman

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4851
Re: Another RAID question
« Reply #4 on: April 01, 2019, 01:57:15 pm »

8TB of keepers. Wow! I can store my keepers on an SD card.
Logged

elliot_n

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1219
Re: Another RAID question
« Reply #5 on: April 01, 2019, 02:00:15 pm »

Hi Eric. I keep all my photos on a RAID5 (in an ancient Promise Pegasus, Thunderbolt 1, with 5 x 2TB drives configured in RAID5, and one hot spare). Save speeds in Photoshop seem very fast - certainly much faster than saving to a single drive. Blackmagic Disk Speed Test gives the write speed as 400 MB/s.

Like you, I'm getting close to my current max capacity of 8TB. I'm considering replacing the 2TB drives in the Promise Pegasus with 4TB drives, to give me a total capacity of 16TB.

(I don't really understand this stuff, so I'm interested in other replies.)
Logged

MattBurt

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3912
  • Looking for that other shot
    • Matt Burt Photography
Re: Another RAID question
« Reply #6 on: April 01, 2019, 02:16:03 pm »

Sorry Matt, both of these are false.  RAID5 adds performance compared to single drives, purely on mechanics. A hard drive on it's own has but one arm, and can only read/write at one location.  Software RAID beats hardware RAID - been this way for over 15 years.  I'll dig out the Youtube talk about how they figured out hardware RAID really was holding them back since the kernel could only talk to 1 device, and that device was I/O bound by it's interface.  No one uses hardware RAID in the big storage arrays, it's all software RAID.

I guess my IT info is a bit out of date! I thought that the overhead of writing parity info was greater than the savings of multiple mechanisms. Maybe that used to be the case when I was more connected to that stuff too.
I've been out of an IT role and in application development for over 15 years.
Logged
-MattB

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Another RAID question
« Reply #7 on: April 01, 2019, 02:42:56 pm »

I guess my IT info is a bit out of date! I thought that the overhead of writing parity info was greater than the savings of multiple mechanisms. Maybe that used to be the case when I was more connected to that stuff too.
I've been out of an IT role and in application development for over 15 years.
The tech sure has changed, I envy your transition having 15 years experience.  Trust me, I remember spending thousands on 80-pin SCSI drives and PERC, Adaptec and LSI cards because they were A LOT faster back then.  As the tech has changed, the connectivity has changed.  Hardware RAID is used where you can't put software RAID - boot volumes, VMware ESXi local volumes, things like that.  Rather than do a giant local RAID, it's faster to do iSCSI over 10gb to a device that's presenting a volume and doing software RAID locally.  Then you get things like SSD caching & data tiering - things a hardware controller couldn't manage.
Logged
t: @PNWMF

DP

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 727
Re: Another RAID question
« Reply #8 on: April 01, 2019, 07:27:19 pm »

Sorry Matt, both of these are false.  RAID5 adds performance compared to single drives, purely on mechanics. A hard drive on it's own has but one arm, and can only read/write at one location.  Software RAID beats hardware RAID - been this way for over 15 years.  I'll dig out the Youtube talk about how they figured out hardware RAID really was holding them back since the kernel could only talk to 1 device, and that device was I/O bound by it's interface.  No one uses hardware RAID in the big storage arrays, it's all software RAID.

with a fine print that any hardware raid is a software running on a dedicated (for that task) hardware ... hence big storage arrays are hardware RAID are too  ;D in that sense... As for RAID 5 beating a single drive in writing (not reading) of a single addressable sector of data in a simple case of identical disks and absolutely no extra resources dedicated to perform RAID operations vs a single disk ? show me an example (w/o such extra resources)  ;D ...
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Another RAID question
« Reply #9 on: April 02, 2019, 10:03:39 am »

It's all about trade offs - little computing power here gives you much faster drive access there :D  Lots of reading for someone who wants to spend the time.

https://macperformanceguide.com/idx-mpg.html
Logged
t: @PNWMF

Eric Brody

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 487
    • http://www.ericbrodyphoto.com
Re: Another RAID question
« Reply #10 on: April 02, 2019, 01:36:58 pm »

Hi Faberryman, I get it. I am certain I too could store my real keepers in a much smaller space but it's cheaper to store these files than to spend the the time to select out the best. Disk space is still relatively inexpensive.
Logged

Eric Brody

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 487
    • http://www.ericbrodyphoto.com
Re: Another RAID question
« Reply #11 on: April 20, 2019, 07:13:18 pm »

After all this, I set up a RAID 5 with three 6TB drives giving me 12TB of usable space. I got a new OWC Thunderbay 4 with no drives and with Softraid software. Setup was easy, even for me, and it seems to be humming along just fine. Speed with my Mac Pro 2013 with a 1TB SSD and 64GB of OWC RAM seems fine. It's hard to tell whether it's significantly faster. The slowest thing I do is save Photoshop TIFF's that are in the 1GB range. That hasn't changed.

Thanks for all the advice and help. This is a great forum with friendly helpful people.
Logged
Pages: [1]   Go Up