Pages: [1]   Go Down

Author Topic: How Large Are Your Drives + Raid?  (Read 12208 times)

HSakols

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1239
    • Hugh Sakols Photography
How Large Are Your Drives + Raid?
« on: December 17, 2013, 05:17:25 pm »

Right now I have just under 2 TB of images  which are on two 3 TB drives.  I do have another backup but it is missing months of work.  I imagine many of you have rather large backup requirement.  Also, it is nice to have all images on one drive so there is never confusion of what is on what drive.  So how big are your drives.  I would think there are photographers out there with say over 16 TB of files.

Second, do all of you use a raid system where one drive is mirrored to another? Or do any of you just manually copy to a different drive?
Logged

Phil Indeblanc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2017
Re: How Large Are Your Drives + Raid?
« Reply #1 on: December 17, 2013, 08:12:46 pm »

As you mention its good to know where they are.  On a parity RAID4+1, you can have a array of 3 drives that act as one large drive and 1 as the "backup". that way they are all in one drive letter. Slower than mirror.
Mirror uses 2 drives for data and 2 drives to mirror the data as backup.

Expect about 50-60MB a second sustained to be a healthy speed for a SATA3 network drive parity
Logged
If you buy a camera, you're a photographer...

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: How Large Are Your Drives + Raid?
« Reply #2 on: December 27, 2013, 04:00:45 pm »

Hi,

4TB data disk, mirrored each night. Time Machine backup to a 6 TB RAID 5, off site backup on a 4 TB external drive. Video and derived stuff on a 4 TByte RAID 1 (internal), with no extra backup. Off site backup updated far to seldom.

Best regards
Erik

Right now I have just under 2 TB of images  which are on two 3 TB drives.  I do have another backup but it is missing months of work.  I imagine many of you have rather large backup requirement.  Also, it is nice to have all images on one drive so there is never confusion of what is on what drive.  So how big are your drives.  I would think there are photographers out there with say over 16 TB of files.

Second, do all of you use a raid system where one drive is mirrored to another? Or do any of you just manually copy to a different drive?
Logged
Erik Kaffehr
 

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: How Large Are Your Drives + Raid?
« Reply #3 on: December 27, 2013, 08:43:23 pm »

8 TB of data to deal with. 3x on-site NAS boxes, RAID 5 and 6, for local backup. Copy to several 4TB disks for off-site (sneakernet to safe deposit box) backup. Want at least five copies, and two off-site.

Am now backing away from RAID5, because of excessive restore times -- long enough for a reasonable chance of another error during the restoration -- now that disks are so big. Replacing RAID5 with RAID1 (mirroring) which is OK now that disks are so cheap.

Jim

phcorrigan

  • Newbie
  • *
  • Offline Offline
  • Posts: 36
Re: How Large Are Your Drives + Raid?
« Reply #4 on: December 31, 2013, 05:11:11 pm »

Second, do all of you use a raid system where one drive is mirrored to another? Or do any of you just manually copy to a different drive?

RAID is not backup. RAID is designed to prevent downtime, increase volume size beyond a single disk, and, in some cases, improve performance. Even if you have a RAID system you still need separate backups.
Logged
Patrick Corrigan
Author, Data Protection for Photographers, now available (http://rockynook.com/book/0/259/data-protection-for-photographers.html)
Website/blog: http://dpworkflow.com

Phil Indeblanc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2017
Re: How Large Are Your Drives + Raid?
« Reply #5 on: December 31, 2013, 06:27:43 pm »

RAID is not backup. RAID is designed to prevent downtime, increase volume size beyond a single disk, and, in some cases, improve performance. Even if you have a RAID system you still need separate backups.

This may apply for some situations, but I have heard of some repairs taking days.
I am starting to lean towards Jim's direction of doing Mirror, as I think I remember testing it, and it I am almost certain that the write/read on it is faster.
Logged
If you buy a camera, you're a photographer...

phcorrigan

  • Newbie
  • *
  • Offline Offline
  • Posts: 36
Re: How Large Are Your Drives + Raid?
« Reply #6 on: December 31, 2013, 06:57:10 pm »

> I am starting to lean towards Jim's direction of doing Mirror, as I think I remember testing it, and it I am almost certain that the write/read on it is faster.

Mirroring is RAID 1 and still not backup. It is designed to prevent downtime. If you lose one disk in the pair the other will continue functioning. However, if you delete data from one disk you delete it from the other. If you overwrite data on one disk you overwrite it on the other. RAID 1 is great for preventing downtime, but you still need backup in addition.
Logged
Patrick Corrigan
Author, Data Protection for Photographers, now available (http://rockynook.com/book/0/259/data-protection-for-photographers.html)
Website/blog: http://dpworkflow.com

bradleygibson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 828
    • http://GibsonPhotographic.com
Re: How Large Are Your Drives + Raid?
« Reply #7 on: January 01, 2014, 10:57:50 pm »

+1 to Patrick's advice.

Please consider what would happen in the event of accidental overwrite or deletion with a RAID mirror.  Both copies would be deleted.
Also, your array is susceptible to natural disaster, theft, fire, et al.  So it really is important to remember that RAID is not backup.

With that being said, my RAID solution is a 7x 3TB WD SATA3 in RAIDZ2 (similar to a growable RAID-6) NAS, connected via 10GbE for high throughput.

My backup solution is CrashPlan.
« Last Edit: January 01, 2014, 10:59:55 pm by bradleygibson »
Logged
-Brad
 [url=http://GibsonPhotographic.com

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Re: How Large Are Your Drives + Raid?
« Reply #8 on: January 02, 2014, 01:29:58 am »

+1 to Patrick's advice.

Please consider what would happen in the event of accidental overwrite or deletion with a RAID mirror.  Both copies would be deleted.
Also, your array is susceptible to natural disaster, theft, fire, et al.  So it really is important to remember that RAID is not backup.

With that being said, my RAID solution is a 7x 3TB WD SATA3 in RAIDZ2 (similar to a growable RAID-6) NAS, connected via 10GbE for high throughput.

My backup solution is CrashPlan.


1.  RAID is a drive just like any other drive with some special features.  No drive by itself should be considered adequate backup.  But any drive can be considered as a backup drive if so designated.  The real problem with most RAID's is people buy inexpensive ones that take forever to restore, are highly unreliable, and more trouble than their worth.  A quality RAID on the other hand, properly utilized, is a valuable tool.. much more valuable than a single disk solution.

2.  Crashplan and other services like them, are just another drive.  They are not backup solutions.  They should still be considered as one copy of your data stored in one place and at risk from the same issues any other drive is.   It's one basket to hold one set of eggs.  Eggs are fragile.

3.  To be considered "backed up" you need a minimum of 3 baskets to hold three sets of eggs.  Remember, eggs are fragile.   

4.  How you set up each drive is just as important as the quality of your drives.  For instance, a 8 bay RAID/NAS device might be capable of 5-6 different RAID schemes.  Which one is right for you?  Is your device protected by a quality UPS?  If it's not, it's not part of your solution.  Rule that device out, at best its' temporary.  If you set up a mirror, does it mirror in real time or perhaps at the end of each work day?  What is right for you?  It probably depends on how you r other devices are set up.  What quality of drives, the quality of the enclosure, the quality of the software that runs the enclosure, and the power protection for the enclosure.  A lot of boxes to tick before it's called backup.

5.  My personal system that works for me:  High quality SSD drives for work drives.  This is backed up at the end of each day to a RAID1.  When the job of finished it's archived to the same RAID1 (2x3tb WD blacks) to my active archives which are roughly 2tb.   An additional standalone 3tb WD Black is :"mirrored" at the end of each workday to the RAID1.   So.. in my box there are 3 separate sets of my work, possibly 4 if I'm still working on it.  This means my box needs to be able to hold 3 3.5 inch drives and 3 2.5 inch SSD's.  If I had a new Mac Pro I'd need to change this.  The box is protected by a quality server class UPS.

Then, at the end of each workday my RAID1 is backed up to my Synology 1813+ 8 bay NAS/RAID which I can transfer data to at about the same rate as my WD Blacks inside my box.  It rebuilds one if it's 8 3tb drives in roughly 24 hours.  It reads and writes both Mac and PC files.  It's located in another room, on another AC circuit, and protected by a quality server class UPS.  This is big enough to run a personal cloud, FTP, store my personal media, and it's separated from my active and inactive archives through designation.

Both my active and inactive archives (roughly 12tb) on my NAS/RAID are also backed up in real time via a 150mbps line to my Amazon S3 account.  Amazon has one of the best server farms anywhere, and they own t heir own.  It's not some startup company we just heard of yesterday who's renting servers hoping their business model pays off.  Still, I'm thinking of buying a second Synology 1813+ and server class UPS to be located off site  to take the places (and relieve the expense) of the S3.  I'm currently working on getting permission to run a cable to my climate controlled storage facility where I'll then set up a second 1813+.. off site, the protections of RAID, all admin controls can be remotely accessed, and I'll put it in a dust proof safe secured t a shelf at about 10 feet just in case of that 100 year flood.  All I need is the permission for the cable, but this is proving difficult as people sometimes try and live in these places so they're careful what they allow.  But once it's done, I'll run it for a few months and then cut Amazon lose for backup purposes.

And finally:  I watch for sales of 2.5 inch SATA HDD's.  Any speed is fine, but they must be of a good quality.  Each year they are labelled with that year and then that years data is copied to that 2.5 inch drive using a ESATA cable.  This drive is then stored in a small fire/water proof safe rated for 60 minutes. 

All that and I'm still not sure it's enough.  The safe is setup as a faraday cage just in case..  Everything is automated so I only need to check if things are going as they should, except for the year end backup.  My images are in other places such as on my website which is also backed up three times.. and on websites where I publish there are web size copies..

2.
Logged
----------------------------------------------
http://www.BangkokImages.com

fotogen

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: How Large Are Your Drives + Raid?
« Reply #9 on: January 05, 2015, 04:47:38 pm »

1TB SSD for Work, 4TB raid 1 intern for storage everything in work under the running year. 12 TB NAS as  Time Machine (covering SSD internal and external Raids). 4 TB external Raid 1 storage for valuable rated pics in Archives (one archive each year). One external HDD per anno for all pics of the year.

Archies and Work in Lightroom (RAW).

Second Archiv in Aperture (JPG) for all valuable Pics from all Years since 1979 (Dia scans included) on the internal Raid. Yearly Backup on sperate HDD. Pics provided via iCloud to all mobil etc. devices. JPG as multi-platform and time independ file format.
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: How Large Are Your Drives + Raid?
« Reply #10 on: January 06, 2015, 02:34:19 am »

Hi,

Right now I have an internal 4TB drive containing my images and personal stuff, duplicated each night to another internal 4TB drive. To that I also have a 6TB RAID 5 (4x2TB) volume used as "TimeMachine" backup plus a single 4TB drive for off-site backup. Another 2 TB RAID1 holds derived work, like video projects.

The next generation of storage I would buy 2 6TB drives and use with a RAID1 configuration, and move the existing 4TB drives in a 12TB RAID5.

But we have 8TB drives now, so a pair of 8TB drives may also be an option.

Best regards
Erik

Logged
Erik Kaffehr
 

Hans Kruse

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2106
    • Hans Kruse Photography
Re: How Large Are Your Drives + Raid?
« Reply #11 on: January 06, 2015, 06:55:36 am »

Right now I have just under 2 TB of images  which are on two 3 TB drives.  I do have another backup but it is missing months of work.  I imagine many of you have rather large backup requirement.  Also, it is nice to have all images on one drive so there is never confusion of what is on what drive.  So how big are your drives.  I would think there are photographers out there with say over 16 TB of files.

Second, do all of you use a raid system where one drive is mirrored to another? Or do any of you just manually copy to a different drive?

My setup is simple and as follows:

Main and only computer MacBook Pro 15" retina with 1TB SSD. On the SSD are typically the most recent pictures from trips (mostly my workshops) and private shoots. Older pictures are stored in a RAID-1 LaCie thunderbolt connected drive. Storage space is 3TB (2x3TB drives). I always do a Time Machine backup on all backup drives before I leave on a trip. Should I loose everything on the trip (computer and backups) I have one backup in the office. The read and write speed on the SSD is 700-900 MB/sec. The write speed on the LaCie is about 120-170MB/sec and read speed is up to 180MB/sec according to disk speed test by Black Magic Design.

I backup the MBP separately on 3 separate 2TB drives using OSX Time Machine. One 3.5" drive is in my office and 2x2TB 2.5" USB-3 drives are taken with me on my trips. I do daily backups. The LaCie storage is backed up on drives in my office and this is done when changes occur. I also have a 2.5" drive that a synchronize from the LaCie drive and I take this one on my trips as well. I use ChronoSync to synchronize the folders from the LaCie storage to the backups. The LaCie makes noise and I typically will have it disconnected except when needed. The MBP when not doing backups is completely quiet and one reason I keep all active folders on the MBP.

My photo folders are almost entirely RAW files. Some TIFF and PSD files from Photoshop and other pixel editors. I typically prune my folders some months after a shoot to cut down on pictures that I will not keep. The total space from 2014 was about 300GB. On the MBP I have all the folders that I'm actively editing. Usually one year back. I edit 95%+ of my pictures in Lightroom so this dramatically cuts down on storage space needed.

I also backup online using https://www.backblaze.com/ both my MBP and the photo folders on the LaCie storage system. The total backed up is 2.2TB. The online backup is only as a last resort backup should I loose all HD backups.
Pages: [1]   Go Up