Luminous Landscape Forum
Equipment & Techniques => Digital Cameras & Shooting Techniques => Topic started by: AgencyDigital on March 21, 2009, 06:00:54 pm
-
So many options available. Can we discuss ideas on fastest / safest internal RAID configurations? I know the two don't necessarily go hand in hand, but I am curious about some of the best compromises. For instance, If i installed 4 VelociRaptors as two striped sets that mirrored each other would i get large gains in speed over setting up drives as (4) individual task related drives (boot drive; backup drive; cache drive; app drive)?
Anyway, I need a system whereby if & when i am out on location and a drive fails (which will happen), it's only a tiny hurdle to jump. I'm not interested in talking about external RAIDs because i already have one. I am curious what serious working professionals are thinking about with the new line-up of Mac Pros. We shoot (on average) 20 - 40 GBs a day & are constantly moving from place to place.
Anyone have any interesting configurations that they are loving & want to share?
-
You might try the Medium format forum just down from this one. (Or the computers and peripherals forum just down from that.) More pros there.
-
AgencyDigital,
I would recommend checking out Lloyd Chambers ecellent website for photographers, especially the 'Macintosh Performance Guide' dealing with RAID drives at
http://macperformanceguide.com/Storage-RAID.html (http://macperformanceguide.com/Storage-RAID.html)
Cheers,
Andrew
-
So many options available. Can we discuss ideas on fastest / safest internal RAID configurations? I know the two don't necessarily go hand in hand, but I am curious about some of the best compromises. For instance, If i installed 4 VelociRaptors as two striped sets that mirrored each other would i get large gains in speed over setting up drives as (4) individual task related drives (boot drive; backup drive; cache drive; app drive)?
Anyway, I need a system whereby if & when i am out on location and a drive fails (which will happen), it's only a tiny hurdle to jump. I'm not interested in talking about external RAIDs because i already have one. I am curious what serious working professionals are thinking about with the new line-up of Mac Pros. We shoot (on average) 20 - 40 GBs a day & are constantly moving from place to place.
Anyone have any interesting configurations that they are loving & want to share?
I could not answer the question regarding Mac (in my job we are only using Linux machines), but I could give you some advice about raid, as we have many machines reading and writing Terabytes of data.
If you have more than 2 drives, RAID5 is the way to go. It will give you both redundancy, faster reading speed and a minimal loss of space. It is slightly slower (with software RAID) regarding writing speed (parity computation). With 4x1 Tb drives, you will have a 3 Tb capacity, and the system can support one drive failure (rebuilding the array when replacing a disk is however long: a few hours, but the system is still available while rebuilding).
I don't know if MacPros can support RAID5 in software. Buying a hardware controller is always a benefit regarding performance, however it is safer to have a spare controller because data stored on the disks can only be recovered with an identical controller if the controller fails.
When we have to go to a distant site to collect data, we usually use a machine with 4 drives assembled in a RAID5 array, with 3 drives in the RAID and the fourth used as a spare (it would automatically replace a failing drive in the array, without having to disassemble the machine to replace the failing drive).
PS: the solution you are suggesting (two stripped drives mirrored) is know as RAID 0+1 or RAID01. It is indeed fast, but rebuilding the array is quite slow and I don't think it is available in software. It is not used very much.
PPS: Be aware however that RAID5 arrays have to be checked periodically. One of the very unpleasant accident with RAID5 is when one sector fails on one disk and remains unnoticed. Then, if another drive fails, rebuiding the array is impossible.
-
One of the very unpleasant accident with RAID5 is when one sector fails on one disk and remains unnoticed. Then, if another drive fails, rebuiding the array is impossible.
wow, i had no idea. How would a failure go unnoticed?
-
AgencyDigital,
I would recommend checking out Lloyd Chambers ecellent website for photographers, especially the 'Macintosh Performance Guide' dealing with RAID drives at
http://macperformanceguide.com/Storage-RAID.html (http://macperformanceguide.com/Storage-RAID.html)
Cheers,
Andrew
Funny, i was at this site yesterday. I have been using RAID 5 for a while. I am curious what kind of other setups photographers have come up with specific for digital capture & Photoshop.
-
wow, i had no idea. How would a failure go unnoticed?
Yes that's something many people do not know, and when it happens all you have left is your eyes to cry, believe me... (and we were lucky because we were also making incremental backups on high capacity tapes each night)...
This is how it happens: your files are stored on sectors, but a sector is only checked when you are trying to read the data. If you don't try to access the file, the corresponding sector might die, and that would remain unnoticed.
Then when another disk fails, you have to access all the sectors on all the drives to rebuild the faulty disk. That's when you are going to try to access your (first) faulty sector, and realize that your data is lost for good.
A good practice is to check your disks regurlarly, at night for example.
-
This shouldn't be an issue. If you've recently accessed the file in question all of its sectors should be checked and recovered. If you have not recently accessed the file it should be in your backup.
-
Still, does anyone have any knowledge about getting the most from Photoshop via internal hard drive configurations?
-
I would also go over and visit the http://thedambook.com/smf/index.php (http://thedambook.com/smf/index.php)
-
AgencyDigital,
I would recommend checking out Lloyd Chambers ecellent website for photographers, especially the 'Macintosh Performance Guide' dealing with RAID drives at
http://macperformanceguide.com/Storage-RAID.html (http://macperformanceguide.com/Storage-RAID.html)
Cheers,
Andrew
Great site, thanks. It's all the info i have been looking for.
Cheers.
-
The purpose of RAID is to eliminate downtime in case of drive failure. (E.g. drive fails, application continues to run off other drive, failed drive is replaced, rebuilds itself, and no one accessing what that drive is offering even notices. No downtime due to failed hard drives.)
Why does a photographer need that? Why go to all that trouble, instead of just having regular drives for backup?
-
The purpose of RAID is to eliminate downtime in case of drive failure. (E.g. drive fails, application continues to run off other drive, failed drive is replaced, rebuilds itself, and no one accessing what that drive is offering even notices. No downtime due to failed hard drives.)
Why does a photographer need that? Why go to all that trouble, instead of just having regular drives for backup?
Really, for obvious reasons. The above statement is only a half truism. We work on massive national campaigns with huge name photographers. In these scenarios, once a model, celebrity, etc. is on set things move at lightning speed. There is no margin for error at this point. So what happens if a boot drive fails, or even worse a data drive?
My current configuration is this: Wiebetech RTX800 SCSI (RAID 5) & (2) Dual Quad G5's. C1 Pro shoots (45MP raw images) directly to the Wiebetech, bypassing the boot drive on the G5, leaving it free for system & application (Adobe Photoshop) usage. Drives fail & life goes on. It's super fast for a plethora of reasons: 8 drive RAID 5; SCSI; puts no pressure on the G5 drive, which is then free for Photoshop.
It's a fine setup for what we do. Maybe their is better, but for me now this is perfect... fast as i need it to be, & very safe.
-
We work on massive national campaigns with huge name photographers. In these scenarios, once a model, celebrity, etc. is on set things move at lightning speed. There is no margin for error at this point. So what happens if a boot drive fails, or even worse a data drive?
Exactly what RAID is for, and pretty much what I described. (Not to sure which 'half' of what I said is not true.)
The point I am trying to get across is that RAID is NOT a backup solution. Redundancy and backup are two different things, but from what I read in various threads I get the impression that many people confuse the two and think a RAID setup is a backup solution.
(I do realize that I inserted this into your thread and it doesn't really apply to you, so I apologize for that, but I just couldn't help myself.)
-
Really, for obvious reasons. The above statement is only a half truism. We work on massive national campaigns with huge name photographers. In these scenarios, once a model, celebrity, etc. is on set things move at lightning speed. There is no margin for error at this point. So what happens if a boot drive fails, or even worse a data drive?
My current configuration is this: Wiebetech RTX800 SCSI (RAID 5) & (2) Dual Quad G5's. C1 Pro shoots (45MP raw images) directly to the Wiebetech, bypassing the boot drive on the G5, leaving it free for system & application (Adobe Photoshop) usage. Drives fail & life goes on. It's super fast for a plethora of reasons: 8 drive RAID 5; SCSI; puts no pressure on the G5 drive, which is then free for Photoshop.
It's a fine setup for what we do. Maybe their is better, but for me now this is perfect... fast as i need it to be, & very safe.
Just a question. doesn't using a G5 grade Mac affect performance?
Since you were talking about things moving at "light speed", I was wondering how does it feels waiting for 45mp previews on C1 on a dual G5. Especially while having to deal with celebrities on set and more than that, their managers and "queens" paranoid AD involved on that kind of shoots.
I would be more concerned about that need for speed rather than hard drive, or back up speed.
just wondering, since I may be missing something.
-
Just a question. doesn't using a G5 grade Mac affect performance?
Since you were talking about things moving at "light speed", I was wondering how does it feels waiting for 45mp previews on C1 on a dual G5. Especially while having to deal with celebrities on set and more than that, their managers and "queens" paranoid AD involved on that kind of shoots.
I would be more concerned about that need for speed rather than hard drive, or back up speed.
just wondering, since I may be missing something.
To be honest, our capture speed is faster than our lights recycle which is all we really need. Faster than that is unnecessary.
-
Why does a photographer need that? Why go to all that trouble, instead of just having regular drives for backup?
The point I am trying to get across is that RAID is NOT a backup solution. Redundancy and backup are two different things, but from what I read in various threads I get the impression that many people confuse the two and think a RAID setup is a backup solution.
I completely hear what you are saying. RAID is not a single backup solution, but redundancy has to be a integral part of the backup process.
-
To be honest, our capture speed is faster than our lights recycle which is all we really need. Faster than that is unnecessary.
I can hardly believe that, unless you plug 3 heads on a Prophoto at full power, but even then, the recycle time of the flash would be plenty faster than "unfuzzing" a 39mp preview on C1.
anyway....
-
RAID is indeed a great solution, but remember that if you go for RAID 0+1 using 4 drives, you have increased your chance of hard drive failure by 4X. I say hard drive failure, not data loss, but this means you will spend more time managing your equipment and messing with RAID rebuilding and configurations than using them.
KISS is always a good principle with anything computer-related.
-
Must agree with fike on the k.i.s.s. principle. I used to run a raid 5 internal array on a Windows machine when I had a some sort of a bios corruption that literally caused the entire array to overwrite before my eyes. I could do nothing but watch as it happened. It was at that time that I decided to go Mac and never looked back. Luckily I am paranoid and had dual external data backup drives but it was still a huge hassle that would have been disastrous had I not taken the precaution to back up the Raid.
This isn't a comment related to the o.p.'s original question, just a comment on computer simplicity in general.
-
Must agree with fike on the k.i.s.s. principle. I used to run a raid 5 internal array on a Windows machine when I had a some sort of a bios corruption that literally caused the entire array to overwrite before my eyes. I could do nothing but watch as it happened. It was at that time that I decided to go Mac and never looked back. Luckily I am paranoid and had dual external data backup drives but it was still a huge hassle that would have been disastrous had I not taken the precaution to back up the Raid.
This isn't a comment related to the o.p.'s original question, just a comment on computer simplicity in general.
Raid is not a substitute for a backup.
-
I can hardly believe that, unless you plug 3 heads on a Prophoto at full power, but even then, the recycle time of the flash would be plenty faster than "unfuzzing" a 39mp preview on C1.
anyway....
It's absolutely true. We shoot Profoto 7A's, one head per pack, usually with one or more heads at full power.
-
Here's my set-up from a thread in the computer section of this site:
http://luminous-landscape.com/forum/index....st&p=268590 (http://luminous-landscape.com/forum/index.php?s=&showtopic=32914&view=findpost&p=268590)
Cheers,
-
I can hardly believe that, unless you plug 3 heads on a Prophoto at full power, but even then, the recycle time of the flash would be plenty faster than "unfuzzing" a 39mp preview on C1.
anyway....
Also, i'm talking about burst speeds. "Unfuzzing" is neither here or there. What i'm talking about is capture speed over light recycle. In a system using a P45 (MP), the speed is fast, very fast considering that we use IQRAW large format (Phase One).
-
Raid is not a substitute for a backup.
I will politely disagree DarkPenguin; it depends on how you configure the array. I would agree that a RAID 0 cannot be used as a backup as it is really only for increasing write speed. Using a RAID 1+mirror or even 1+0 can be and easy way to build in a back up at the time of writing especially a mirror. As I noted in my post, I have always maintained separate data backup drives, however if I use LR to create the backup on a separate drive or allow a RAID 1+mirror to do the same thing, what really is the difference?
-
I will politely disagree DarkPenguin; it depends on how you configure the array. I would agree that a RAID 0 cannot be used as a backup as it is really only for increasing write speed. Using a RAID 1+mirror or even 1+0 can be and easy way to build in a back up at the time of writing especially a mirror. As I noted in my post, I have always maintained separate data backup drives, however if I use LR to create the backup on a separate drive or allow a RAID 1+mirror to do the same thing, what really is the difference?
When you delete a file on a raid system the file is gone. When you do the same with an actual backup you can go the backup and recover it.
-
I will politely disagree DarkPenguin; it depends on how you configure the array. I would agree that a RAID 0 cannot be used as a backup as it is really only for increasing write speed. Using a RAID 1+mirror or even 1+0 can be and easy way to build in a back up at the time of writing especially a mirror. As I noted in my post, I have always maintained separate data backup drives, however if I use LR to create the backup on a separate drive or allow a RAID 1+mirror to do the same thing, what really is the difference?
Probably the biggest misconception related to hard drives is that Redundancy = Backup. It is not. Any one who has a RAID set up and thinks they have a back up solution is mistaken. Like DarkPenguin says, with RAID if you delete a file, it is gone on ALL drives. If a file gets corrupted by a software application, it is corrupted on ALL drives. You have NO backup!
-
Exactly. RAID is about uptime or speed.
-
Exactly. RAID is about uptime or speed.
Agreed. However one RAID array *backed up* (read, not mirrored) to a second RAID array is in fact a back up, and if that second array is RAID with redundancy (like R5), so much the better...
Cheers,
-
Agreed. However one RAID array *backed up* (read, not mirrored) to a second RAID array is in fact a back up, and if that second array is RAID with redundancy (like R5), so much the better...
Cheers,
Yes. But again the RAID aspect is being used for uptime or speed. This time keeping your backup online. And that is a great way to use a raid.
Another way to use a raid as a backup is to occasionally break the raid, store that disk, put a new disk into the raid and let it rebuild. This is even better if you can mirror across 3 disks.
-
Still, RAID 1 is recommended. A 160GB raid specific drive from WD (RE2 series) is only $60. Two are $120 and work great as a main OS drive. My RAID 1 array crashed three days ago. I rebuilt it in 2tow hours while doing my normal business. Nearly painless recovery from a bad FAT that would have bricked a normal drive and required data recovery.
Worth every penny.
-
Thanks for the education; I stand corrected.
-
Thanks for the education; I stand corrected.
Interesting conversation. If you have ever heard the statement that " One is none, two is one, three is two." then you will be somewhat familiar with RAID.
Raid can also speed up the performance of a PC or Mac running PhotoShop, for a great online resource see digilloyd as previously stated.
As far as back up goes the above statement should apply, that and keep the back ups at separate locations.
Raid is a tool, that has proven very effective for digital photographers.
-
Okay, so for me this configuration will win:
Mac Pro 2.66 Dual Quad
12GB RAM
Velociraptor RAID 0 Boot
Barracuda RAID 0 Scratch
eSATA RAID 5 Media
Firewire 800 Backup drive.
Maybe not the fastest, but very fast & secure.
-
I have just started using an all digital work flow and my set up is
Mac Pro 2x2.8 Quad 12 GB Ram
Drive 1 System Disc (750GB)
Drives 2+3 Striped (2x 1TB) Aperture Library and H3D ScratchPads
Drive 4 Backup (1TB) inc Aperture Vault + H3D Archive + PPArchive (Active Archive)
External Backups 1TB FW 800's A-Vault + H3D Archive + PPArchive , stored off site ( Active and Preservation Archives)
To take advantage of lower cost SATA drives I am considering buying a WiebeTech SilverSATA 4 and 4x1TB (or 2TB?) drives with a eSATA controller has any one experience of this unit?
I have just seen Jack Flesher's solution of using the lower optical drive bay for two SATA drives must investigate
http://luminous-landscape.com/forum/index....mp;#entry268590 (http://luminous-landscape.com/forum/index.php?showtopic=32914&st=0&p=268590&#entry268590)