Pages: [1]   Go Down

Author Topic: A question about using two Thunderbolt RAID arrays in a multi-stage workflow  (Read 3071 times)

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

Starting with raw images stored in a library on a Thunderbolt connected RAID array and ending up with the post-processed final images also stored on that array,  would using a second Thunderbolt RAID array (set up as RAID 0 for speed) as an intermediate "holding pen" for the high resolution (50mp and up) images while they are being being post-processed, significantly enough speed up the workflow to justify the cost of that second (much smaller capacity) array?
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365

It depends on a few things, but you're better working with SSDs as the core part of the structure.  How many bays/drives are you looking for?  Generally speaking how many cards do you dump at once?  I take it you have a USB3 reader(s)?  I'm using this kind of workflow:

Import/copy files from cards to SSD
Copy imported files from SSD to platter drives
Work against SSD files for editing, if dealing with large photo stitching, a second SSD for the Photoshop swap may be added
Save edited psd/jpeg to SSD
Copy/move updated files from SSD to platter drives as projects complete or more import space is needed

SSDs are way faster than any RAID0 setup you can throw at it.  A single SSD will work fine for a 'holding pen' and be much much faster.  Before you get into buying a TB RAID array, what size do you need, and what are you looking at?  I say this as most external RAID setups are bad for users as there isn't a recovery method that doesn't involve finding another one of them.
Logged
t: @PNWMF

Manoli

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2299

... would using a second Thunderbolt RAID array (set up as RAID 0 for speed) as an intermediate "holding pen" for the high resolution (50mp and up) images while they are being being post-processed, significantly enough speed up the workflow to justify the cost of that second (much smaller capacity) array?

In a word - no.
There is no speed benefit using a 7200RPM Thunderbolt RAID-0 drive over a USB3 SSD.
See this post regarding gotchas and speed comparisons
http://forum.luminous-landscape.com/index.php?topic=108021.msg889862#msg889862
« Last Edit: March 01, 2016, 01:45:09 am by Manoli »
Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

Thanks! I am already using an external SSD (USB 3.0 connected) for my Lightroom catalog so using another one as an holding pen for the TIFFs makes sense.

And yes I am stitching panoramas - 48 frames per 360-degree rotation and shooting with a 5DS. The blended + layers .psb panoramas tend to be around 9GB @ 8 BPC or 18GB @ 16 BPC.
A 240 or 480GB SSD makes the most sense.
Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

based on the comments by Joe and Manoli As opposed to a RAID 0 solution,  I am now looking at either the OWC 480GB Envoy Pro Mini USB3 Thumbdrive vs. the OWC Mercury Envoy Pro EX 480GB USB3 SSD.  I have looked at the reviews on amazon but paid more attention to the reviews on  macperformanceguide.com. Unfortunately Mr. Chambers seems to have used different test protocols for the two so using his information I cannot make an apples to apples comparison. , but both seem to be very fast indeed.

My baseline question is: will adding either to my workflow be fast enough to justify the expense? In other words will I save minutes or seconds?

Logged

dwswager

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1375

Starting with raw images stored in a library on a Thunderbolt connected RAID array and ending up with the post-processed final images also stored on that array,  would using a second Thunderbolt RAID array (set up as RAID 0 for speed) as an intermediate "holding pen" for the high resolution (50mp and up) images while they are being being post-processed, significantly enough speed up the workflow to justify the cost of that second (much smaller capacity) array?

Accounting for the much higher reliability of today's SSDs and even Hard Drives, I would say a 2nd RAID array would not be beneficial from a cost/speed perspective.

Assuming you have already created a backup of the original images offline, I recommend a mirror setup for the intermediaries.  I lost a junk of work back when to a HD failure and always use a mirror setup for intermediaries because that is where a great chunk of time and effort exists.  As much as loosing originals is tragic, loosing all that intermediary work is depressing.

As to seconds or minutes gained, some operations will be measured in seconds, but cumulative operations might be minutes.  SSDs crush hard drives for random reads.  Bottomline, sometimes even seconds feel like minutes and can be the difference between getting done before your attention moves to something else and it waiting till you can get back to it.  For that reason, I always consider the an SSD as the better alternative for working data.
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365

http://eshop.macsales.com/shop/Thunderbolt/

I'd do 4 drives - 2 SSDs (single drives for PS Swap and intermediaries), and 2 platter drives (setup as a RAID1).  Or go nuts, 4 SSDs, 1 for swap, RAID1 for Intermediaries, 1 for source files.  Either the mini or the full size unit will work.  The big thing is to take the additional I/O off the USB3 bus, and put it on the TB bus.  The full size units need a drive carrier for the SSDs to fit, but allow platter drives up to 8tb each. 
Logged
t: @PNWMF

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995

And yes I am stitching panoramas - 48 frames per 360-degree rotation and shooting with a 5DS. The blended + layers .psb panoramas tend to be around 9GB @ 8 BPC or 18GB @ 16 BPC.

a nearby barebone desktop computer sporting 64-128GB RAM with __RAM__ disk... 10Gpbs Ethernet.

Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995

SSDs crush hard drives for random reads. 
I use RAM disk, it crushes PCIe RAID0 SSD by a magnitude...
Logged

dwswager

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1375

I use RAM disk, it crushes PCIe RAID0 SSD by a magnitude...

From a speed perspective, absolutely.  But volatile memory is not the smartest for "storing" work, even temporarily.  Where RAM disks are good is if you can script and have the file loaded to the RAM disk and then execute whatever operations are necessary and then have the output written back to non volatile memory.  Essentially using the RAM as the intermediary internal to the operations.  That is how I used to do Nuclear Reactor Kinetics computations.

I have also used RAM disks for Photoshop scratch disks in the past.
Logged
Pages: [1]   Go Up