Pages: [1]   Go Down

Author Topic: New build and disk/RAID configuration  (Read 1097 times)

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
New build and disk/RAID configuration
« on: June 26, 2020, 06:06:12 pm »

Just completed a new build after six years. I have a NVMe boot disk with LR catalog partition. Also have 2 Western Digital Red SATA drives (6 TB each) for which I need advice as to how I might configure them. I was thinking I might use RAID 0 to "stripe" them and improve performance along with a nightly backup to mirrored external USB drives. Obviously, I could just use them conventionally and backup similarly, but would I miss out on the better performance of the RAID 0? Any of this make sense or is there a better way?
Thanks in advance.

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: New build and disk/RAID configuration
« Reply #1 on: June 27, 2020, 01:13:19 am »

Hi David,

can i suggest that you add another NVME for your Lightroom catalogue, none of this partition stuff!

i would divide my storage up in 3 parts (at least)

disk 1 - NVME (as fast as you can get it) at least 256GB - OS and applications (you can run a mirror backup of this on a second disk if you like so you can get online should something happen)
disk 2 - NVME (as fast as you can get it) this can be a small disk -  this is for your LR catalogue, PS scratch disk
disk 3 - NVME (as fast as you can get it) at least 1TB - current Working files

Redundancy (not a backup) - just large accessible storage
disk 4 - DISK HDD/NAS - live copy of current and present work (ideally you will have a setup with some redundancy built in  8 disk NAS RAID-6 or Synology SHR2
 - here you may want to add fast ethernet and SSD cache (see BudJames setup $$$)

Backup
disk 5 - DISK HDD - Last but not least - offline and off site backup, some people will use duplicate copy on different brand disks in case you run into a bad batch

good luck

Henrik

 
Logged

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: New build and disk/RAID configuration
« Reply #2 on: June 27, 2020, 01:15:17 am »

obviously you can add RAID to them all, but you still need the fundamental data structure where you have a backup!

btw, i run a RAID0 of two NVME M.2 Corsair MP600 PCIe Gen4 - I am not getting 2x speed improvement, more like 1.5x

Henrik
Logged

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #3 on: June 27, 2020, 08:40:47 am »

Thanks for the advice, Henrik.
So, if you backup RAID0 drives to an external USB or offline, do you get the original files or is the data on the backup drive still striped?
BTW, I do have two NVME (Sabrient) a 2TB that I will use for OS/programs/caches/catalog and a 4TB for data files. The 12TB of HDD will be used for redundant storage, original NEF files, and archives including master files, but I'd like that storage to be as snappy as possible.
PS Motherboard does have three NVMe.2 slots, but I thought I'd wait and see what comes down the road in a year or so.
« Last Edit: June 27, 2020, 08:51:13 am by David Eckels »
Logged

BFD

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 85
    • Dana Neibert
Re: New build and disk/RAID configuration
« Reply #4 on: June 29, 2020, 06:06:23 pm »

Thanks for the advice, Henrik.
So, if you backup RAID0 drives to an external USB or offline, do you get the original files or is the data on the backup drive still striped?
BTW, I do have two NVME (Sabrient) a 2TB that I will use for OS/programs/caches/catalog and a 4TB for data files. The 12TB of HDD will be used for redundant storage, original NEF files, and archives including master files, but I'd like that storage to be as snappy as possible.
PS Motherboard does have three NVMe.2 slots, but I thought I'd wait and see what comes down the road in a year or so.
You'll want to go RAID 0 for the speed. But, FWIW, spinning hard drives are fairly inexpensive nowadays. RAID 0 will leave you highly vulnerable to a hard drive crash unless you are also backing that stuff up somewhere else. I would suggest a RAID 10 with 4,6, 8 or more drives—obviously the more drives you use the faster the system will be. Also, more drives will allow you to use smaller drives (usually even cheaper) while still ending up with a large storage volume. RAID 10 will be making 2 copies of everything.

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: New build and disk/RAID configuration
« Reply #5 on: June 29, 2020, 10:21:32 pm »

How much of the 6/12TB will be used immediately?  How much data do you add per year?  Honestly, if you wanted to do a backup setup, I'd do the 2 Red's as independent drives.  6TB(1) would be the archive drive, backing up to the USB disk.  6TB(2) would be the backup for the 2 NVMe drives.
Logged
t: @PNWMF

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #6 on: July 01, 2020, 04:33:38 pm »

Did some tests with Windows software striping vs just a regular SATA HDD (6TB) and the regular write was about a third faster than writing to striped drives. Software slowness? Anyway, I decided to use the two drives extended to ~12TB with backup to a high capacity USB 3.1, which can be written to somewhere north of 125 MB/s using File History overnight. Should take care of my needs for the next eight years.
Thanks for the help, all!

geneo

  • Full Member
  • ***
  • Offline Offline
  • Posts: 152
Re: New build and disk/RAID configuration
« Reply #7 on: July 01, 2020, 06:07:58 pm »

Did some tests with Windows software striping vs just a regular SATA HDD (6TB) and the regular write was about a third faster than writing to striped drives. Software slowness? Anyway, I decided to use the two drives extended to ~12TB with backup to a high capacity USB 3.1, which can be written to somewhere north of 125 MB/s using File History overnight. Should take care of my needs for the next eight years.
Thanks for the help, all!

Hi,

Just to make sure I understand you, striping RAID0 resulted in 30% slower performance?
What was the test?
« Last Edit: July 01, 2020, 06:13:13 pm by geneo »
Logged
Walk softly and carry a big lens

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #8 on: July 01, 2020, 07:38:57 pm »

Thanks for calling me on this geneo, but I was questioning my own conclusions anyway and found I'd managed to bolux it up somehow ::) Redid a write test (drag and drop file copy). Folders containing NEF and PSD files. Copying from 4 TB Sabrient NVMe to either spanned or striped WD Red 6 TB SATA drives. Interesting results:

At 5 GB data (two folders): avg 1.5 GB/s for spanned; steady 1.9 GB/s striped (same folders)
At 25 GB data (5 folders): 1.9 GB/s decreasing to 150 MB/s spanned; 1.9 GB/s decreasing to 800 MB/s striped (definitely faster)
At 291 GB data (11 folders): didn't test the spanned drives; 1.9 GB/s decreased to a steady 325 MB/s before I stopped the transfer

My conclusion is that smaller transfers (5-10 GB) move really fast onto the striped drives as one might expect. For some reason, bigger transfers (longer time?) slow down to about double that for a spanned volume. But 25 Gigs go lickety split with the striped configuration! I could have timed these runs, but the "RAID 0" is definitely faster and I will go with that and do File History backups to an external USB 3.1. Should balance plenty of speed, capacity, and redundancy. As to why the rate of transfer drops, maybe it has something to do with buffers; curious.

degrub

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1947
Re: New build and disk/RAID configuration
« Reply #9 on: July 01, 2020, 09:18:51 pm »

Almost sounds like cache size is initially playing a role - both the disk controller cache abd possibly the write to drive.
Logged

mcbroomf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1534
    • Mike Broomfield
Re: New build and disk/RAID configuration
« Reply #10 on: July 02, 2020, 04:43:04 am »

If you decide not to use stripped/raid 0 then I would not use them spanned.  Could get messy recovering if one drive fails although you will have a separate backup.  Just use them as 2 individual drives, no speed advantage either way and only a small amount of extra file management on your part as one drive starts to fill and you have to cascade over.
Logged

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: New build and disk/RAID configuration
« Reply #11 on: July 02, 2020, 05:12:52 am »

Hi David

Thanks for sharing your stats

I think there is always a risk regardless of you using one or many disks in your setup.

Have a closer look at your stats and see which setup performance better with small or large files and then set it up accordingly where you see you get the best performance

Henrik
Logged

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #12 on: July 02, 2020, 03:19:31 pm »

degrub: That's what I meant by buffers (me no expert!), but I assume those are "hardwired" and since I am getting acceptable performance, I'll leave it lie.

Mike: That thought occurred to me, too. Simplicity can be a good thing!

Henrik: The tests were with my Photoshop files so I think they are representative of what I can expect; a mixture of ~50MB raw files and larger >1+ GB tif and psd files. For reasons beyond my level of experience, on my old build, I was getting transfer speeds in the <20 MB/s; probably plugged in to the wrong hole! ::) I'll take a two log improvement any day!

All: Can't thank you enough, but it's worth a try: THANKS!

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #13 on: July 04, 2020, 10:46:02 am »

Just as an afterthought on this thread, I bought a couple of Seagate Backup Plus USB drives. Couldn't figure out how to mirror them easily and from what I read, RAID 1 is not backup anyway. So I set one of them to backup (using Win10) my photo files and for the other I set File History to back up the same folders. A kludge, I know, but couldn't figure a better way. At least I will have three copies of everything!

geneo

  • Full Member
  • ***
  • Offline Offline
  • Posts: 152
Re: New build and disk/RAID configuration
« Reply #14 on: July 04, 2020, 11:34:15 am »

You really can't raid two separate USB drives.

It is always good to have multiple backups. I have several large USB drives and I have a copy of my data, including photos, on all three. For this I use Microsoft Synctoy for syncing the replication (system backups using macrium reflect). I keep one of those USB drives disconnected from the computer and power and hook it up only to sync.
Logged
Walk softly and carry a big lens

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #15 on: July 04, 2020, 12:37:13 pm »

For this I use Microsoft Synctoy for syncing the replication (system backups using macrium reflect).
Do you think Synctoy would work with MS File History or Backup images?

geneo

  • Full Member
  • ***
  • Offline Offline
  • Posts: 152
Re: New build and disk/RAID configuration
« Reply #16 on: July 04, 2020, 06:15:52 pm »

Do you think Synctoy would work with MS File History or Backup images?

Sorry, don't know. I don't use those.
Logged
Walk softly and carry a big lens

David Eckels

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3526
  • It's just a camera.
    • Website
Re: New build and disk/RAID configuration
« Reply #17 on: July 05, 2020, 01:41:03 pm »

Looks like Synctoy works just fine with File History  8)
Pages: [1]   Go Up