Pages: [1] 2 3   Go Down

Author Topic: Check in with how you are archiving and backing up images  (Read 14923 times)

budjames

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 956
    • http://www.budjamesphotography.com
Check in with how you are archiving and backing up images
« on: December 26, 2008, 02:29:07 pm »

I'm using a MacPro 8-core as my main image station. I have 4TB of HD space which includes 2x1TB in RAID 0, 1TB boot drive and workings documents and a 1TB drive w/2 partitions: 100GB for PS scratch and the rest to save SuperDuper clones of my boot drive. I also use a new aluminum MacBookPro 15" 2.8ghz as my road warrior machine.

For over a year, I've been using the ReadyNAS NV+ on my gigabit network to archive photos, video and other documents. But with image files growing larger with my new 1DsMkIII plus even larger video files, the network drive is too pokey for anything but deep archives.

As we now have 4 Macs in our household, I also use Apple's Airport Extreme with a 1.5TB USB drive attached for Time Machine backups of user files and data on for each Mac.

I'm considering purchasing an external eSata 5-bay enclosure to use with my MacPro for local backups. I plan on using 2 bays in RAID 0 and the other 3 for JBOD so that I can rotate these drives offset for redundant fire-proof backups. I'm looking at the SansDigital Tower RAID TR5M1 that I can populate with Seagate 1TB drives.

I keep back up drives in synch using Chronosync and I just purchased Synchronize X Pro for faster performance with backups to my NAS.

I am curious how others are dealing with keeping backups with their growing collection of images.

Thanks for any sharing of ideas.

Bud James
North Wales, PA
Logged
Bud James
North Wales, PA [url=http://ww

Chris_Brown

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 974
  • Smile dammit!
    • Chris Brown Photography
Check in with how you are archiving and backing up images
« Reply #1 on: December 26, 2008, 06:05:11 pm »

This issue has been coming for a while and I hope there's good discussion on it.

I currently use an external 6-drive unit configured into RAID 1 for a total of 3 TBs. It appears on the studio network as an external set of drives hosted by one of the computers. It also hosts our studio music (all tunes saved in original AIFF format) and the host computer runs iTunes over wireless for music throughout the day. The most important images, and there are hundreds, also get saved to gold optical media (archival DVDs).

Since this drive bay is almost full, we're considering buying an old dual G5 and have it run Xserve, and buying a rack system to load up with redundant drives. Although more expensive than a drive tower, this system is only limited by the size of the rack and its drive expandability is easy. Plus if we choose to go optical (as in $$$) the transfer of image data will be lickity-split.  
Logged
~ CB

Josh-H

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2079
    • Wild Nature Photo Travel
Check in with how you are archiving and backing up images
« Reply #2 on: December 26, 2008, 06:38:11 pm »

Quote from: budjames
I'm using a MacPro 8-core as my main image station. I have 4TB of HD space which includes 2x1TB in RAID 0, 1TB boot drive and workings documents and a 1TB drive w/2 partitions: 100GB for PS scratch and the rest to save SuperDuper clones of my boot drive. I also use a new aluminum MacBookPro 15" 2.8ghz as my road warrior machine.

For over a year, I've been using the ReadyNAS NV+ on my gigabit network to archive photos, video and other documents. But with image files growing larger with my new 1DsMkIII plus even larger video files, the network drive is too pokey for anything but deep archives.

As we now have 4 Macs in our household, I also use Apple's Airport Extreme with a 1.5TB USB drive attached for Time Machine backups of user files and data on for each Mac.

I'm considering purchasing an external eSata 5-bay enclosure to use with my MacPro for local backups. I plan on using 2 bays in RAID 0 and the other 3 for JBOD so that I can rotate these drives offset for redundant fire-proof backups. I'm looking at the SansDigital Tower RAID TR5M1 that I can populate with Seagate 1TB drives.

I keep back up drives in synch using Chronosync and I just purchased Synchronize X Pro for faster performance with backups to my NAS.

I am curious how others are dealing with keeping backups with their growing collection of images.

Thanks for any sharing of ideas.

Bud James
North Wales, PA


Hi Bud,

I always follow your posts with more than a casual interest as you utilize a lot of the same equipment I do in my day-to-day photography - 1DSMK3, a lot of the same glass, mac pro etc.. So I hope my feedback is of assistance.

I have an 8-Core MAc Pro 2.8Ghz w/ 10gb RAM and with 4 x 500 Gig Enterprise grade drives in RAID 5 - this is my main workhorse for photo editing - not dissimilar to your own. Connected to this I have 2 x LACEY 2 big Triple RAID NAS drives - each one has 2 x 1TB drives in RAID 1 configuration - the two NAS's mirror each other. Thus I have effectivley 3 x RAID systems all with the same data. I regularly disconnect one of the NAS drives and store it offsite.

This may seem like overkill - but I learnt my lesson earlier this year when I had 3 Maxtor RAID drives go bang within a week and take more than a terrabyte of data with them...

This works very well for my set-up as I only store my own personal work long term on the hard drives - client work I back up to 2 x DVD's and store in a safe.

Keep us posted on what you decide to do as I would like to know - especially on how you find the 5-Bay Esata.
« Last Edit: December 26, 2008, 06:39:50 pm by Josh-H »
Logged
Wild Nature Photo Travel

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2848
Check in with how you are archiving and backing up images
« Reply #3 on: December 26, 2008, 09:29:31 pm »

I use a Thecus NAS RAID device for my primary working set of data - it's convenient and redundant (RAID 5) and has dual Gigabit networking so the speed is decent.

Backups are to two single eSATA drives for speed and for universal connectivity.  Using RAID for backup means you need to ensure that the device you're using to run the RAID doesn't fail or you can obtain another or you can find a derivative that's going to accept your HDDs and make sure you don't connect them in the wrong order and cause problems.

One of the backup drives is stored offsite and rotated with the onsite one.

With single drives of up to 1.5TB available and any number of software options to split large contents and keep them up to date without having to copy the entire contents, it's cheap, effective, reliable and efficient.

Whatever solutions people use, I strongly recommend that you have a working copy, a local backup and an offsite backup and that your backups are universal (ie do not use proprietary system/software/hardware to run and access them).  I also strongly recommend eSATA due to the huge speed advantage over USB or Firewire.
Logged
Phil Brown

joergen geerds

  • Newbie
  • *
  • Offline Offline
  • Posts: 46
    • http://joergengeerds.com
Check in with how you are archiving and backing up images
« Reply #4 on: December 27, 2008, 10:42:30 am »

Whatever you do, a RAID0 is not for backup, it is not a safe way to _store_ data... use a raid5 or better to store data... in addition, copy the contents of your main data backup onto 1TB disks, and bring them to another place every 4-8 weeks, and do a complete copy over every 12 months on those externally stored disks, to make sure that the data doesn't degrade.


jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #5 on: December 27, 2008, 08:57:31 pm »

Quote from: joergen geerds
Whatever you do, a RAID0 is not for backup, it is not a safe way to _store_ data... use a raid5 or better to store data... in addition, copy the contents of your main data backup onto 1TB disks, and bring them to another place every 4-8 weeks, and do a complete copy over every 12 months on those externally stored disks, to make sure that the data doesn't degrade.
Ahem.

The "original" data may have degraded, too.

Hard disks have, typically, a non-recoverable read error rate of 1 per 10^14 bits read.

1.5 terabytes = 1.2 * 10^13 bits

This means that you on average should expect a non-recoverable read error for every 12 terabytes read. That used to be a lot, but it's not anymore. Similarly, there may be non-recoverable write errors.

In addition, there may be write errors that aren't discovered by the RAID controller (if there is one).

NetApp (with WAFL) and SUN (with ZFS) have created specific solutions to these problems by calculating checksums on writing and reading of data from disk.

For a more home grown solution than "enterprise" storage solutions, you'll have to use either backup software that creates such checksums (CRC, MD5, whatever they call it), or you have to create those yourself, in order to feel reasonably certain that your data remains uncorrupted/undegraded.
Logged
Jan

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #6 on: December 27, 2008, 11:48:43 pm »

Quote from: jani
Ahem.

The "original" data may have degraded, too.

Hard disks have, typically, a non-recoverable read error rate of 1 per 10^14 bits read.

1.5 terabytes = 1.2 * 10^13 bits

This means that you on average should expect a non-recoverable read error for every 12 terabytes read. That used to be a lot, but it's not anymore. Similarly, there may be non-recoverable write errors.

In addition, there may be write errors that aren't discovered by the RAID controller (if there is one).

NetApp (with WAFL) and SUN (with ZFS) have created specific solutions to these problems by calculating checksums on writing and reading of data from disk.

For a more home grown solution than "enterprise" storage solutions, you'll have to use either backup software that creates such checksums (CRC, MD5, whatever they call it), or you have to create those yourself, in order to feel reasonably certain that your data remains uncorrupted/undegraded.
For some reason, I get no regular errors every 12 terabytes or so, and I read that much data within a couple of months.  Where I do get errors is every now and then, about once a year on average, a segment of one or more files may be overwritten with garbage, which I can nearly always trace to a preventable cause.  The main reason I know I don't get random errors, other than the aforementioned corruption, is because I do content comparisons of all of my current working files (about 30,000 files) on the 10 or so backup systems I use, an average of once a week each.  The batch files that do that work are automated, but when a file doesn't compare, the program stops and the bell rings.  I would guess that most quality media have enough redundancy to get past read errors without the user having to be alerted or concerned.
Logged

budjames

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 956
    • http://www.budjamesphotography.com
Check in with how you are archiving and backing up images
« Reply #7 on: January 01, 2009, 11:00:43 am »

Hi all and Happy New Year!

Thanks for the collective input on my original post. Based on feedback here and my research on other forums, including Apple's, and discussion my my "guy" at Other World Computing, I have decided on the following setup for my MacPro 8-core:

Inside the MP - I'm leaving my existing setup of all Seagate Barracuda 1TB drives: Bay1 - Boot Mac OS, apps and normal user files (iPhoto, documents, etc) and a 100GB partition for Bootcamp WinXP Pro for PC-only games (all work and no play thing); Bay 2 100GB PS Scratch and the rest for SuperDuper clones of boot drive; Bays 3 and 4 are RAID 0 1.8TB for photos and video working files.

For external storage, I ordered a SansDigital 5 bay eSata tower and HighPoint PCIe eSata host/port multiplier card. The tower has easy slide out trays that are hot swappable. I'll use 2 x 1TB drives to backup my internal RAID. The remaining 3 bays will be JBOD for backups.

This ought to keep me going for a while.

Cheers.
Bud James
Logged
Bud James
North Wales, PA [url=http://ww

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2848
Check in with how you are archiving and backing up images
« Reply #8 on: January 01, 2009, 03:14:52 pm »

Sounds like a very nice setup :-)  Only query I have is what are you doing in terms of offsite backup?  I'd strongly recommend that you include some mechanism to get data offsite regularly.
Logged
Phil Brown

feppe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2906
  • Oh this shows up in here!
    • Harri Jahkola Photography
Check in with how you are archiving and backing up images
« Reply #9 on: January 01, 2009, 03:51:29 pm »

Repeat after me: "RAID is not for backup." It is designed for uptime, speed and/or space depending on the version. You don't want your critical backups being reliant on two things: the controller card and software functioning properly, and hard drive failure. The fewer points of failure there are in the system, the more likely it is to function well.

I go the KISS way: three external HDDs. I have two Hyperdrive SPACEs which I fill with the backups. And I have a 1TB external HDD which I store at work for offsite storage, in case of fire or theft at my home. I fully verify the entire backup every time I run the program. And I don't use any proprietary archiving methods, but plain CR2 files. I never ever delete anything from the backup drives to avoid user error, and I don't keep them plugged in the computer to minimize the same.

Offsite is something which many people seem to forget. Theft, fire, flood, leaking pipes, etc. has destroyed numerous people's pictures. A colleague lost the entire collection of childhood photos of his daughter that way. I'm thinking of getting a fireproof safe, but that's mainly for paperwork, and doesn't protect from theft.

The main thing I'm worried about with this setup is viruses.

David Sutton

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1345
    • David Sutton Photography
Check in with how you are archiving and backing up images
« Reply #10 on: January 01, 2009, 04:29:15 pm »

On the principle that some back up is better than none at all, and as off site storage would not work for me as I'd never get around to doing it, I have a fire proof (read: resistant) safe in the laundry (the place I estimate would have the lowest temperature in case of a fire). I use syncing software to back up to external hard drives. BTW, the only problem I've had so far with hard drives has been an external back up drive. The case shorted internally and blew the fuses in the house. The drive survived fine though. David
Logged

budjames

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 956
    • http://www.budjamesphotography.com
Check in with how you are archiving and backing up images
« Reply #11 on: January 01, 2009, 04:31:11 pm »

Quote from: Farmer
Sounds like a very nice setup :-)  Only query I have is what are you doing in terms of offsite backup?  I'd strongly recommend that you include some mechanism to get data offsite regularly.

I forgot to mention that my previous external drives (2 x 1TB and 3 x 750GB Seagate drives in single drive OWC FW400/800 USB enclosures) are used to back up photos, videos and my Lightroom database for off site storage that I keep in my bank safe deposit box.

I use Synchronize! Pro X to mirror my local drive contents to these external drives. Depending upon how many new images I create (I'm an amateur), I usually rotate the drives through the safe deposit box about every 1 to 2 weeks.

On my network, I have a ReadyNAS NV+ on with 4 x 1TB drives in RAID-X configuration. I back up to every 3 days. I have it set to automatic backups.

For my financial planning practice (it's how I can afford expense cameras and computer gear) I back up my MacProLap to my ReadyNAS NV+ and I mirror the drive using SuperDuper to 2 OWC portable FW800 drives. These drives are rotated through my bank safe deposit box weekly.

Lastly, I have a 1TB drive connected to the USB port on my Apple Airport Extreme for automatic Time Machine backups of my MacPro and MacBook Pro User folders which contains iPhoto and iTunes files along with documents and financial files.

Cheers.
Bud James
Logged
Bud James
North Wales, PA [url=http://ww

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #12 on: January 01, 2009, 05:21:48 pm »

Quote from: dalethorn
For some reason, I get no regular errors every 12 terabytes or so, and I read that much data within a couple of months.
"On average" does not mean that you're guaranteed to have errors.

Single, anecdotal evidence does not count, I'm afraid.

Quote
I would guess that most quality media have enough redundancy to get past read errors without the user having to be alerted or concerned.
Yes, those are the recoverable read errors.

The non-recoverable ones are just that: non-recoverable (by the hard drive's firmware).

If they'd had been much better, this wouldn't have been a large enough issue that NetApp, SUN and other storage solution providers would have to worry about it ten years ago.
Logged
Jan

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #13 on: January 01, 2009, 05:36:55 pm »

Quote from: feppe
Repeat after me: "RAID is not for backup." It is designed for uptime, speed and/or space depending on the version. You don't want your critical backups being reliant on two things: the controller card and software functioning properly, and hard drive failure. The fewer points of failure there are in the system, the more likely it is to function well.

I go the KISS way: three external HDDs. I have two Hyperdrive SPACEs which I fill with the backups. And I have a 1TB external HDD which I store at work for offsite storage, in case of fire or theft at my home. I fully verify the entire backup every time I run the program. And I don't use any proprietary archiving methods, but plain CR2 files. I never ever delete anything from the backup drives to avoid user error, and I don't keep them plugged in the computer to minimize the same.

Offsite is something which many people seem to forget. Theft, fire, flood, leaking pipes, etc. has destroyed numerous people's pictures. A colleague lost the entire collection of childhood photos of his daughter that way. I'm thinking of getting a fireproof safe, but that's mainly for paperwork, and doesn't protect from theft.
To expand on what you write:

I think it's important to distinguish between the following:

  • Redundancy
  • Backup
  • Archive
  • Disaster contingencies and risk reduction
You'd use RAID for redundancy, near-line copies for backup, and off-site copies for archives and disaster risk reduction.

The point of a backup solution is that your data should be recoverable easily and quickly to a state not too far in the past.

The lack of provably long-term reliable large-scale storage is a challenge for archival use. (Okay, I use DVDs, but only because I take incremental backups immediately. I'd hate to restore from hundreds of DVDs.)

That being said, a backup system may very well be a disk-based RAID. I have no qualms about that, not for personal use, nor in professional use; at work we have around 44 TB of online RAID-6 backup systems. I know it's not a lot, but those who do have a lot of data they need to backup use e.g. EMC's or NetApp's VTL (virtual tape library), or similar solutions from competitors.

I'd say the risk of damaging your data by jostling that hard drive you transport back and forth is greater than losing data to a RAID controller.
Logged
Jan

budjames

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 956
    • http://www.budjamesphotography.com
Check in with how you are archiving and backing up images
« Reply #14 on: January 01, 2009, 08:52:40 pm »

Quote from: jani
To expand on what you write:

I think it's important to distinguish between the following:

  • Redundancy
  • Backup
  • Archive
  • Disaster contingencies and risk reduction
You'd use RAID for redundancy, near-line copies for backup, and off-site copies for archives and disaster risk reduction.

The point of a backup solution is that your data should be recoverable easily and quickly to a state not too far in the past.

The lack of provably long-term reliable large-scale storage is a challenge for archival use. (Okay, I use DVDs, but only because I take incremental backups immediately. I'd hate to restore from hundreds of DVDs.)

That being said, a backup system may very well be a disk-based RAID. I have no qualms about that, not for personal use, nor in professional use; at work we have around 44 TB of online RAID-6 backup systems. I know it's not a lot, but those who do have a lot of data they need to backup use e.g. EMC's or NetApp's VTL (virtual tape library), or similar solutions from competitors.

I'd say the risk of damaging your data by jostling that hard drive you transport back and forth is greater than losing data to a RAID controller.

I agree with your last sentence in particular. If I have redundant and recent duplicates of my working files. If they are on multiple RAID set ups, the chances of all RAID systems failing at the same time are pretty remote. Using a local redundant RAID array for repeatable backups is fine as long as I keep recent back ups off-site too.

In my case, my MacPro RAID, the eSata tower RAID connected to it and the ReadyNAS NV+ RAID-X network box would all have to fail at the same time AND my bank that has my safe deposit box storing my off-site single drive back ups would have to burn down to the ground or get swallowed by an earthquake (not likely in s.e. Pennsylvania) in order for my back up strategy to fail completely. I think that I'll take the odds that this will not occur.

Cheers.
Bud James
Logged
Bud James
North Wales, PA [url=http://ww

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #15 on: January 01, 2009, 09:10:36 pm »

Quote from: jani
Single, anecdotal evidence does not count, I'm afraid.
You should be afraid. In my 27-1/2 years of not only expert backup, but *creating* backup tools, I've gained a great deal more than anecdotal evidence.
Logged

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2848
Check in with how you are archiving and backing up images
« Reply #16 on: January 02, 2009, 01:40:35 am »

Quote from: dalethorn
You should be afraid. In my 27-1/2 years of not only expert backup, but *creating* backup tools, I've gained a great deal more than anecdotal evidence.

Which is a qualification that your original statement did not carry.  I've been using computers since 1981 - but no one will know that until I tell them.  Your  original statement was solitary and runs contrary to the general experience reported.
Logged
Phil Brown

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #17 on: January 02, 2009, 07:47:15 am »

Quote from: Farmer
Which is a qualification that your original statement did not carry.  I've been using computers since 1981 - but no one will know that until I tell them.  Your  original statement was solitary and runs contrary to the general experience reported.
The problem with the post I replied to was not its skepticism, it was its smug dismissiveness.
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #18 on: January 02, 2009, 10:10:50 am »

Quote from: dalethorn
The problem with the post I replied to was not its skepticism, it was its smug dismissiveness.
I don't think you should be one to complain about "smug dismissiveness".

Even if your experience is 27 years and running, it's still anecdotal evidence until that you can show some research in the field of hard drive reliability, or you can show that there is research by other people supporting your claims.

This is particularly important when your claims run contrary to well-established industry assumptions and published research results.

It's also possible that you may not have been measuring what you think you've been measuring.

For the benefit of others, who may not have your experience, here's a nice table about read reliability, and then some links to documents with more information:


(from the first document mentioned below)


There were a couple of interesting talks in last year's IDEMA symposium, but I haven't found any web version of those. As I didn't go there myself, I don't know what was said, either.
Logged
Jan

budjames

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 956
    • http://www.budjamesphotography.com
Check in with how you are archiving and backing up images
« Reply #19 on: January 02, 2009, 10:25:56 am »

Hi guys. I think that the latest posts are off my original topic a bit. To the casual observer, it would now appear to be a "___ssing contest". I suggest that we cool it for the benefit of others or take your banter off line.

Happy New Year.
Bud
Logged
Bud James
North Wales, PA [url=http://ww
Pages: [1] 2 3   Go Up