Topic: Internal RAID setup for Photographers (Read 11850 times)

AgencyDigital · « **on:** March 21, 2009, 06:00:54 pm »

So many options available. Can we discuss ideas on fastest / safest internal RAID configurations? I know the two don't necessarily go hand in hand, but I am curious about some of the best compromises. For instance, If i installed 4 VelociRaptors as two striped sets that mirrored each other would i get large gains in speed over setting up drives as (4) individual task related drives (boot drive; backup drive; cache drive; app drive)?

Anyway, I need a system whereby if & when i am out on location and a drive fails (which will happen), it's only a tiny hurdle to jump. I'm not interested in talking about external RAIDs because i already have one. I am curious what serious working professionals are thinking about with the new line-up of Mac Pros. We shoot (on average) 20 - 40 GBs a day & are constantly moving from place to place.

Anyone have any interesting configurations that they are loving & want to share?

DarkPenguin · « **Reply #1 on:** March 21, 2009, 09:18:27 pm »

You might try the Medium format forum just down from this one. (Or the computers and peripherals forum just down from that.) More pros there.

Woodcorner · « **Reply #2 on:** March 22, 2009, 07:32:41 am »

AgencyDigital,

I would recommend checking out Lloyd Chambers ecellent website for photographers, especially the 'Macintosh Performance Guide' dealing with RAID drives at
http://macperformanceguide.com/Storage-RAID.html

Cheers,

Andrew

james_elliot · « **Reply #3 on:** March 22, 2009, 09:00:29 am »

Quote from: AgencyDigital

So many options available. Can we discuss ideas on fastest / safest internal RAID configurations? I know the two don't necessarily go hand in hand, but I am curious about some of the best compromises. For instance, If i installed 4 VelociRaptors as two striped sets that mirrored each other would i get large gains in speed over setting up drives as (4) individual task related drives (boot drive; backup drive; cache drive; app drive)?

Anyway, I need a system whereby if & when i am out on location and a drive fails (which will happen), it's only a tiny hurdle to jump. I'm not interested in talking about external RAIDs because i already have one. I am curious what serious working professionals are thinking about with the new line-up of Mac Pros. We shoot (on average) 20 - 40 GBs a day & are constantly moving from place to place.

Anyone have any interesting configurations that they are loving & want to share?

I could not answer the question regarding Mac (in my job we are only using Linux machines), but I could give you some advice about raid, as we have many machines reading and writing Terabytes of data.

If you have more than 2 drives, RAID5 is the way to go. It will give you both redundancy, faster reading speed and a minimal loss of space. It is slightly slower (with software RAID) regarding writing speed (parity computation). With 4x1 Tb drives, you will have a 3 Tb capacity, and the system can support one drive failure (rebuilding the array when replacing a disk is however long: a few hours, but the system is still available while rebuilding).

I don't know if MacPros can support RAID5 in software. Buying a hardware controller is always a benefit regarding performance, however it is safer to have a spare controller because data stored on the disks can only be recovered with an identical controller if the controller fails.

When we have to go to a distant site to collect data, we usually use a machine with 4 drives assembled in a RAID5 array, with 3 drives in the RAID and the fourth used as a spare (it would automatically replace a failing drive in the array, without having to disassemble the machine to replace the failing drive).

PS: the solution you are suggesting (two stripped drives mirrored) is know as RAID 0+1 or RAID01. It is indeed fast, but rebuilding the array is quite slow and I don't think it is available in software. It is not used very much.

PPS: Be aware however that RAID5 arrays have to be checked periodically. One of the very unpleasant accident with RAID5 is when one sector fails on one disk and remains unnoticed. Then, if another drive fails, rebuiding the array is impossible.

AgencyDigital · « **Reply #4 on:** March 22, 2009, 11:19:31 am »

Quote from: james_elliot

One of the very unpleasant accident with RAID5 is when one sector fails on one disk and remains unnoticed. Then, if another drive fails, rebuiding the array is impossible.

wow, i had no idea. How would a failure go unnoticed?

AgencyDigital · « **Reply #5 on:** March 22, 2009, 11:22:02 am »

Quote from: Woodcorner

AgencyDigital,

I would recommend checking out Lloyd Chambers ecellent website for photographers, especially the 'Macintosh Performance Guide' dealing with RAID drives at
http://macperformanceguide.com/Storage-RAID.html

Cheers,

Andrew

Funny, i was at this site yesterday. I have been using RAID 5 for a while. I am curious what kind of other setups photographers have come up with specific for digital capture & Photoshop.

james_elliot · « **Reply #6 on:** March 22, 2009, 12:50:00 pm »

Quote from: AgencyDigital

wow, i had no idea. How would a failure go unnoticed?

Yes that's something many people do not know, and when it happens all you have left is your eyes to cry, believe me... (and we were lucky because we were also making incremental backups on high capacity tapes each night)...

This is how it happens: your files are stored on sectors, but a sector is only checked when you are trying to read the data. If you don't try to access the file, the corresponding sector might die, and that would remain unnoticed.
Then when another disk fails, you have to access all the sectors on all the drives to rebuild the faulty disk. That's when you are going to try to access your (first) faulty sector, and realize that your data is lost for good.
A good practice is to check your disks regurlarly, at night for example.

DarkPenguin · « **Reply #7 on:** March 22, 2009, 01:01:28 pm »

This shouldn't be an issue. If you've recently accessed the file in question all of its sectors should be checked and recovered. If you have not recently accessed the file it should be in your backup.

AgencyDigital · « **Reply #8 on:** March 25, 2009, 09:40:47 am »

Still, does anyone have any knowledge about getting the most from Photoshop via internal hard drive configurations?

Craig Murphy · « **Reply #9 on:** March 25, 2009, 01:10:44 pm »

I would also go over and visit the http://thedambook.com/smf/index.php

AgencyDigital · « **Reply #10 on:** March 25, 2009, 06:26:44 pm »

Quote from: Woodcorner

AgencyDigital,

I would recommend checking out Lloyd Chambers ecellent website for photographers, especially the 'Macintosh Performance Guide' dealing with RAID drives at
http://macperformanceguide.com/Storage-RAID.html

Cheers,

Andrew

Great site, thanks. It's all the info i have been looking for.
Cheers.

JDClements · « **Reply #11 on:** March 25, 2009, 06:53:00 pm »

The purpose of RAID is to eliminate downtime in case of drive failure. (E.g. drive fails, application continues to run off other drive, failed drive is replaced, rebuilds itself, and no one accessing what that drive is offering even notices. No downtime due to failed hard drives.)

Why does a photographer need that? Why go to all that trouble, instead of just having regular drives for backup?

AgencyDigital · « **Reply #12 on:** March 26, 2009, 10:36:49 am »

Quote from: JDClements

The purpose of RAID is to eliminate downtime in case of drive failure. (E.g. drive fails, application continues to run off other drive, failed drive is replaced, rebuilds itself, and no one accessing what that drive is offering even notices. No downtime due to failed hard drives.)

Why does a photographer need that? Why go to all that trouble, instead of just having regular drives for backup?

Really, for obvious reasons. The above statement is only a half truism. We work on massive national campaigns with huge name photographers. In these scenarios, once a model, celebrity, etc. is on set things move at lightning speed. There is no margin for error at this point. So what happens if a boot drive fails, or even worse a data drive?

My current configuration is this: Wiebetech RTX800 SCSI (RAID 5) & (2) Dual Quad G5's. C1 Pro shoots (45MP raw images) directly to the Wiebetech, bypassing the boot drive on the G5, leaving it free for system & application (Adobe Photoshop) usage. Drives fail & life goes on. It's super fast for a plethora of reasons: 8 drive RAID 5; SCSI; puts no pressure on the G5 drive, which is then free for Photoshop.

It's a fine setup for what we do. Maybe their is better, but for me now this is perfect... fast as i need it to be, & very safe.

JDClements · « **Reply #13 on:** March 26, 2009, 05:52:19 pm »

Quote from: AgencyDigital

We work on massive national campaigns with huge name photographers. In these scenarios, once a model, celebrity, etc. is on set things move at lightning speed. There is no margin for error at this point. So what happens if a boot drive fails, or even worse a data drive?

Exactly what RAID is for, and pretty much what I described. (Not to sure which 'half' of what I said is not true.)

The point I am trying to get across is that RAID is NOT a backup solution. Redundancy and backup are two different things, but from what I read in various threads I get the impression that many people confuse the two and think a RAID setup is a backup solution.

(I do realize that I inserted this into your thread and it doesn't really apply to you, so I apologize for that, but I just couldn't help myself.)

ziocan · « **Reply #14 on:** March 28, 2009, 04:56:19 pm »

Quote from: AgencyDigital

Really, for obvious reasons. The above statement is only a half truism. We work on massive national campaigns with huge name photographers. In these scenarios, once a model, celebrity, etc. is on set things move at lightning speed. There is no margin for error at this point. So what happens if a boot drive fails, or even worse a data drive?

My current configuration is this: Wiebetech RTX800 SCSI (RAID 5) & (2) Dual Quad G5's. C1 Pro shoots (45MP raw images) directly to the Wiebetech, bypassing the boot drive on the G5, leaving it free for system & application (Adobe Photoshop) usage. Drives fail & life goes on. It's super fast for a plethora of reasons: 8 drive RAID 5; SCSI; puts no pressure on the G5 drive, which is then free for Photoshop.

It's a fine setup for what we do. Maybe their is better, but for me now this is perfect... fast as i need it to be, & very safe.

Just a question. doesn't using a G5 grade Mac affect performance?
Since you were talking about things moving at "light speed", I was wondering how does it feels waiting for 45mp previews on C1 on a dual G5. Especially while having to deal with celebrities on set and more than that, their managers and "queens" paranoid AD involved on that kind of shoots.
I would be more concerned about that need for speed rather than hard drive, or back up speed.
just wondering, since I may be missing something.

AgencyDigital · « **Reply #15 on:** March 29, 2009, 01:17:26 pm »

Quote from: ziocan

Just a question. doesn't using a G5 grade Mac affect performance?
Since you were talking about things moving at "light speed", I was wondering how does it feels waiting for 45mp previews on C1 on a dual G5. Especially while having to deal with celebrities on set and more than that, their managers and "queens" paranoid AD involved on that kind of shoots.
I would be more concerned about that need for speed rather than hard drive, or back up speed.
just wondering, since I may be missing something.

To be honest, our capture speed is faster than our lights recycle which is all we really need. Faster than that is unnecessary.

AgencyDigital · « **Reply #16 on:** March 29, 2009, 01:18:10 pm »

Quote from: JDClements

Why does a photographer need that? Why go to all that trouble, instead of just having regular drives for backup?

The point I am trying to get across is that RAID is NOT a backup solution. Redundancy and backup are two different things, but from what I read in various threads I get the impression that many people confuse the two and think a RAID setup is a backup solution.

I completely hear what you are saying. RAID is not a single backup solution, but redundancy has to be a integral part of the backup process.

ziocan · « **Reply #17 on:** March 29, 2009, 03:23:03 pm »

Quote from: AgencyDigital

To be honest, our capture speed is faster than our lights recycle which is all we really need. Faster than that is unnecessary.

I can hardly believe that, unless you plug 3 heads on a Prophoto at full power, but even then, the recycle time of the flash would be plenty faster than "unfuzzing" a 39mp preview on C1.
anyway....

fike · « **Reply #18 on:** March 30, 2009, 02:16:28 pm »

RAID is indeed a great solution, but remember that if you go for RAID 0+1 using 4 drives, you have increased your chance of hard drive failure by 4X. I say hard drive failure, not data loss, but this means you will spend more time managing your equipment and messing with RAID rebuilding and configurations than using them.

KISS is always a good principle with anything computer-related.

gunnar1 · « **Reply #19 on:** March 31, 2009, 12:21:53 am »

Must agree with fike on the k.i.s.s. principle. I used to run a raid 5 internal array on a Windows machine when I had a some sort of a bios corruption that literally caused the entire array to overwrite before my eyes. I could do nothing but watch as it happened. It was at that time that I decided to go Mac and never looked back. Luckily I am paranoid and had dual external data backup drives but it was still a huge hassle that would have been disastrous had I not taken the precaution to back up the Raid.

This isn't a comment related to the o.p.'s original question, just a comment on computer simplicity in general.

Author Topic: Internal RAID setup for Photographers (Read 11850 times)

DarkPenguin

DarkPenguin