Luminous Landscape Forum
Raw & Post Processing, Printing => Digital Image Processing => Topic started by: budjames on December 26, 2008, 02:29:07 pm
-
I'm using a MacPro 8-core as my main image station. I have 4TB of HD space which includes 2x1TB in RAID 0, 1TB boot drive and workings documents and a 1TB drive w/2 partitions: 100GB for PS scratch and the rest to save SuperDuper clones of my boot drive. I also use a new aluminum MacBookPro 15" 2.8ghz as my road warrior machine.
For over a year, I've been using the ReadyNAS NV+ on my gigabit network to archive photos, video and other documents. But with image files growing larger with my new 1DsMkIII plus even larger video files, the network drive is too pokey for anything but deep archives.
As we now have 4 Macs in our household, I also use Apple's Airport Extreme with a 1.5TB USB drive attached for Time Machine backups of user files and data on for each Mac.
I'm considering purchasing an external eSata 5-bay enclosure to use with my MacPro for local backups. I plan on using 2 bays in RAID 0 and the other 3 for JBOD so that I can rotate these drives offset for redundant fire-proof backups. I'm looking at the SansDigital Tower RAID TR5M1 that I can populate with Seagate 1TB drives.
I keep back up drives in synch using Chronosync and I just purchased Synchronize X Pro for faster performance with backups to my NAS.
I am curious how others are dealing with keeping backups with their growing collection of images.
Thanks for any sharing of ideas.
Bud James
North Wales, PA
-
This issue has been coming for a while and I hope there's good discussion on it.
I currently use an external 6-drive unit configured into RAID 1 for a total of 3 TBs. It appears on the studio network as an external set of drives hosted by one of the computers. It also hosts our studio music (all tunes saved in original AIFF format) and the host computer runs iTunes over wireless for music throughout the day. The most important images, and there are hundreds, also get saved to gold optical media (archival DVDs).
Since this drive bay is almost full, we're considering buying an old dual G5 and have it run Xserve, and buying a rack system to load up with redundant drives. Although more expensive than a drive tower, this system is only limited by the size of the rack and its drive expandability is easy. Plus if we choose to go optical (as in $$$) the transfer of image data will be lickity-split.
-
I'm using a MacPro 8-core as my main image station. I have 4TB of HD space which includes 2x1TB in RAID 0, 1TB boot drive and workings documents and a 1TB drive w/2 partitions: 100GB for PS scratch and the rest to save SuperDuper clones of my boot drive. I also use a new aluminum MacBookPro 15" 2.8ghz as my road warrior machine.
For over a year, I've been using the ReadyNAS NV+ on my gigabit network to archive photos, video and other documents. But with image files growing larger with my new 1DsMkIII plus even larger video files, the network drive is too pokey for anything but deep archives.
As we now have 4 Macs in our household, I also use Apple's Airport Extreme with a 1.5TB USB drive attached for Time Machine backups of user files and data on for each Mac.
I'm considering purchasing an external eSata 5-bay enclosure to use with my MacPro for local backups. I plan on using 2 bays in RAID 0 and the other 3 for JBOD so that I can rotate these drives offset for redundant fire-proof backups. I'm looking at the SansDigital Tower RAID TR5M1 that I can populate with Seagate 1TB drives.
I keep back up drives in synch using Chronosync and I just purchased Synchronize X Pro for faster performance with backups to my NAS.
I am curious how others are dealing with keeping backups with their growing collection of images.
Thanks for any sharing of ideas.
Bud James
North Wales, PA
Hi Bud,
I always follow your posts with more than a casual interest as you utilize a lot of the same equipment I do in my day-to-day photography - 1DSMK3, a lot of the same glass, mac pro etc.. So I hope my feedback is of assistance.
I have an 8-Core MAc Pro 2.8Ghz w/ 10gb RAM and with 4 x 500 Gig Enterprise grade drives in RAID 5 - this is my main workhorse for photo editing - not dissimilar to your own. Connected to this I have 2 x LACEY 2 big Triple RAID NAS drives - each one has 2 x 1TB drives in RAID 1 configuration - the two NAS's mirror each other. Thus I have effectivley 3 x RAID systems all with the same data. I regularly disconnect one of the NAS drives and store it offsite.
This may seem like overkill - but I learnt my lesson earlier this year when I had 3 Maxtor RAID drives go bang within a week and take more than a terrabyte of data with them...
This works very well for my set-up as I only store my own personal work long term on the hard drives - client work I back up to 2 x DVD's and store in a safe.
Keep us posted on what you decide to do as I would like to know - especially on how you find the 5-Bay Esata.
-
I use a Thecus NAS RAID device for my primary working set of data - it's convenient and redundant (RAID 5) and has dual Gigabit networking so the speed is decent.
Backups are to two single eSATA drives for speed and for universal connectivity. Using RAID for backup means you need to ensure that the device you're using to run the RAID doesn't fail or you can obtain another or you can find a derivative that's going to accept your HDDs and make sure you don't connect them in the wrong order and cause problems.
One of the backup drives is stored offsite and rotated with the onsite one.
With single drives of up to 1.5TB available and any number of software options to split large contents and keep them up to date without having to copy the entire contents, it's cheap, effective, reliable and efficient.
Whatever solutions people use, I strongly recommend that you have a working copy, a local backup and an offsite backup and that your backups are universal (ie do not use proprietary system/software/hardware to run and access them). I also strongly recommend eSATA due to the huge speed advantage over USB or Firewire.
-
Whatever you do, a RAID0 is not for backup, it is not a safe way to _store_ data... use a raid5 or better to store data... in addition, copy the contents of your main data backup onto 1TB disks, and bring them to another place every 4-8 weeks, and do a complete copy over every 12 months on those externally stored disks, to make sure that the data doesn't degrade.
-
Whatever you do, a RAID0 is not for backup, it is not a safe way to _store_ data... use a raid5 or better to store data... in addition, copy the contents of your main data backup onto 1TB disks, and bring them to another place every 4-8 weeks, and do a complete copy over every 12 months on those externally stored disks, to make sure that the data doesn't degrade.
Ahem.
The "original" data may have degraded, too.
Hard disks have, typically, a non-recoverable read error rate of 1 per 10^14 bits read.
1.5 terabytes = 1.2 * 10^13 bits
This means that you on average should expect a non-recoverable read error for every 12 terabytes read. That used to be a lot, but it's not anymore. Similarly, there may be non-recoverable write errors.
In addition, there may be write errors that aren't discovered by the RAID controller (if there is one).
NetApp (with WAFL) and SUN (with ZFS) have created specific solutions to these problems by calculating checksums on writing and reading of data from disk.
For a more home grown solution than "enterprise" storage solutions, you'll have to use either backup software that creates such checksums (CRC, MD5, whatever they call it), or you have to create those yourself, in order to feel reasonably certain that your data remains uncorrupted/undegraded.
-
Ahem.
The "original" data may have degraded, too.
Hard disks have, typically, a non-recoverable read error rate of 1 per 10^14 bits read.
1.5 terabytes = 1.2 * 10^13 bits
This means that you on average should expect a non-recoverable read error for every 12 terabytes read. That used to be a lot, but it's not anymore. Similarly, there may be non-recoverable write errors.
In addition, there may be write errors that aren't discovered by the RAID controller (if there is one).
NetApp (with WAFL) and SUN (with ZFS) have created specific solutions to these problems by calculating checksums on writing and reading of data from disk.
For a more home grown solution than "enterprise" storage solutions, you'll have to use either backup software that creates such checksums (CRC, MD5, whatever they call it), or you have to create those yourself, in order to feel reasonably certain that your data remains uncorrupted/undegraded.
For some reason, I get no regular errors every 12 terabytes or so, and I read that much data within a couple of months. Where I do get errors is every now and then, about once a year on average, a segment of one or more files may be overwritten with garbage, which I can nearly always trace to a preventable cause. The main reason I know I don't get random errors, other than the aforementioned corruption, is because I do content comparisons of all of my current working files (about 30,000 files) on the 10 or so backup systems I use, an average of once a week each. The batch files that do that work are automated, but when a file doesn't compare, the program stops and the bell rings. I would guess that most quality media have enough redundancy to get past read errors without the user having to be alerted or concerned.
-
Hi all and Happy New Year!
Thanks for the collective input on my original post. Based on feedback here and my research on other forums, including Apple's, and discussion my my "guy" at Other World Computing, I have decided on the following setup for my MacPro 8-core:
Inside the MP - I'm leaving my existing setup of all Seagate Barracuda 1TB drives: Bay1 - Boot Mac OS, apps and normal user files (iPhoto, documents, etc) and a 100GB partition for Bootcamp WinXP Pro for PC-only games (all work and no play thing); Bay 2 100GB PS Scratch and the rest for SuperDuper clones of boot drive; Bays 3 and 4 are RAID 0 1.8TB for photos and video working files.
For external storage, I ordered a SansDigital 5 bay eSata tower and HighPoint PCIe eSata host/port multiplier card. The tower has easy slide out trays that are hot swappable. I'll use 2 x 1TB drives to backup my internal RAID. The remaining 3 bays will be JBOD for backups.
This ought to keep me going for a while.
Cheers.
Bud James
-
Sounds like a very nice setup :-) Only query I have is what are you doing in terms of offsite backup? I'd strongly recommend that you include some mechanism to get data offsite regularly.
-
Repeat after me: "RAID is not for backup." It is designed for uptime, speed and/or space depending on the version. You don't want your critical backups being reliant on two things: the controller card and software functioning properly, and hard drive failure. The fewer points of failure there are in the system, the more likely it is to function well.
I go the KISS way: three external HDDs. I have two Hyperdrive SPACEs which I fill with the backups. And I have a 1TB external HDD which I store at work for offsite storage, in case of fire or theft at my home. I fully verify the entire backup every time I run the program. And I don't use any proprietary archiving methods, but plain CR2 files. I never ever delete anything from the backup drives to avoid user error, and I don't keep them plugged in the computer to minimize the same.
Offsite is something which many people seem to forget. Theft, fire, flood, leaking pipes, etc. has destroyed numerous people's pictures. A colleague lost the entire collection of childhood photos of his daughter that way. I'm thinking of getting a fireproof safe, but that's mainly for paperwork, and doesn't protect from theft.
The main thing I'm worried about with this setup is viruses.
-
On the principle that some back up is better than none at all, and as off site storage would not work for me as I'd never get around to doing it, I have a fire proof (read: resistant) safe in the laundry (the place I estimate would have the lowest temperature in case of a fire). I use syncing software to back up to external hard drives. BTW, the only problem I've had so far with hard drives has been an external back up drive. The case shorted internally and blew the fuses in the house. The drive survived fine though. David
-
Sounds like a very nice setup :-) Only query I have is what are you doing in terms of offsite backup? I'd strongly recommend that you include some mechanism to get data offsite regularly.
I forgot to mention that my previous external drives (2 x 1TB and 3 x 750GB Seagate drives in single drive OWC FW400/800 USB enclosures) are used to back up photos, videos and my Lightroom database for off site storage that I keep in my bank safe deposit box.
I use Synchronize! Pro X to mirror my local drive contents to these external drives. Depending upon how many new images I create (I'm an amateur), I usually rotate the drives through the safe deposit box about every 1 to 2 weeks.
On my network, I have a ReadyNAS NV+ on with 4 x 1TB drives in RAID-X configuration. I back up to every 3 days. I have it set to automatic backups.
For my financial planning practice (it's how I can afford expense cameras and computer gear) I back up my MacProLap to my ReadyNAS NV+ and I mirror the drive using SuperDuper to 2 OWC portable FW800 drives. These drives are rotated through my bank safe deposit box weekly.
Lastly, I have a 1TB drive connected to the USB port on my Apple Airport Extreme for automatic Time Machine backups of my MacPro and MacBook Pro User folders which contains iPhoto and iTunes files along with documents and financial files.
Cheers.
Bud James
-
For some reason, I get no regular errors every 12 terabytes or so, and I read that much data within a couple of months.
"On average" does not mean that you're guaranteed to have errors.
Single, anecdotal evidence does not count, I'm afraid.
I would guess that most quality media have enough redundancy to get past read errors without the user having to be alerted or concerned.
Yes, those are the recoverable read errors.
The non-recoverable ones are just that: non-recoverable (by the hard drive's firmware).
If they'd had been much better, this wouldn't have been a large enough issue that NetApp, SUN and other storage solution providers would have to worry about it ten years ago.
-
Repeat after me: "RAID is not for backup." It is designed for uptime, speed and/or space depending on the version. You don't want your critical backups being reliant on two things: the controller card and software functioning properly, and hard drive failure. The fewer points of failure there are in the system, the more likely it is to function well.
I go the KISS way: three external HDDs. I have two Hyperdrive SPACEs which I fill with the backups. And I have a 1TB external HDD which I store at work for offsite storage, in case of fire or theft at my home. I fully verify the entire backup every time I run the program. And I don't use any proprietary archiving methods, but plain CR2 files. I never ever delete anything from the backup drives to avoid user error, and I don't keep them plugged in the computer to minimize the same.
Offsite is something which many people seem to forget. Theft, fire, flood, leaking pipes, etc. has destroyed numerous people's pictures. A colleague lost the entire collection of childhood photos of his daughter that way. I'm thinking of getting a fireproof safe, but that's mainly for paperwork, and doesn't protect from theft.
To expand on what you write:
I think it's important to distinguish between the following:
- Redundancy
- Backup
- Archive
- Disaster contingencies and risk reduction
You'd use RAID for redundancy, near-line copies for backup, and off-site copies for archives and disaster risk reduction.
The point of a backup solution is that your data should be recoverable easily and quickly to a state not too far in the past.
The lack of provably long-term reliable large-scale storage is a challenge for archival use. (Okay, I use DVDs, but only because I take incremental backups immediately. I'd hate to restore from hundreds of DVDs.)
That being said, a backup system may very well be a disk-based RAID. I have no qualms about that, not for personal use, nor in professional use; at work we have around 44 TB of online RAID-6 backup systems. I know it's not a lot, but those who do have a lot of data they need to backup use e.g. EMC's or NetApp's VTL (virtual tape library), or similar solutions from competitors.
I'd say the risk of damaging your data by jostling that hard drive you transport back and forth is greater than losing data to a RAID controller.
-
To expand on what you write:
I think it's important to distinguish between the following:
- Redundancy
- Backup
- Archive
- Disaster contingencies and risk reduction
You'd use RAID for redundancy, near-line copies for backup, and off-site copies for archives and disaster risk reduction.
The point of a backup solution is that your data should be recoverable easily and quickly to a state not too far in the past.
The lack of provably long-term reliable large-scale storage is a challenge for archival use. (Okay, I use DVDs, but only because I take incremental backups immediately. I'd hate to restore from hundreds of DVDs.)
That being said, a backup system may very well be a disk-based RAID. I have no qualms about that, not for personal use, nor in professional use; at work we have around 44 TB of online RAID-6 backup systems. I know it's not a lot, but those who do have a lot of data they need to backup use e.g. EMC's or NetApp's VTL (virtual tape library), or similar solutions from competitors.
I'd say the risk of damaging your data by jostling that hard drive you transport back and forth is greater than losing data to a RAID controller.
I agree with your last sentence in particular. If I have redundant and recent duplicates of my working files. If they are on multiple RAID set ups, the chances of all RAID systems failing at the same time are pretty remote. Using a local redundant RAID array for repeatable backups is fine as long as I keep recent back ups off-site too.
In my case, my MacPro RAID, the eSata tower RAID connected to it and the ReadyNAS NV+ RAID-X network box would all have to fail at the same time AND my bank that has my safe deposit box storing my off-site single drive back ups would have to burn down to the ground or get swallowed by an earthquake (not likely in s.e. Pennsylvania) in order for my back up strategy to fail completely. I think that I'll take the odds that this will not occur.
Cheers.
Bud James
-
Single, anecdotal evidence does not count, I'm afraid.
You should be afraid. In my 27-1/2 years of not only expert backup, but *creating* backup tools, I've gained a great deal more than anecdotal evidence.
-
You should be afraid. In my 27-1/2 years of not only expert backup, but *creating* backup tools, I've gained a great deal more than anecdotal evidence.
Which is a qualification that your original statement did not carry. I've been using computers since 1981 - but no one will know that until I tell them. Your original statement was solitary and runs contrary to the general experience reported.
-
Which is a qualification that your original statement did not carry. I've been using computers since 1981 - but no one will know that until I tell them. Your original statement was solitary and runs contrary to the general experience reported.
The problem with the post I replied to was not its skepticism, it was its smug dismissiveness.
-
The problem with the post I replied to was not its skepticism, it was its smug dismissiveness.
I don't think you should be one to complain about "smug dismissiveness".
Even if your experience is 27 years and running, it's still anecdotal evidence until that you can show some research in the field of hard drive reliability, or you can show that there is research by other people supporting your claims.
This is particularly important when your claims run contrary to well-established industry assumptions and published research results.
It's also possible that you may not have been measuring what you think you've been measuring.
For the benefit of others, who may not have your experience, here's a nice table about read reliability, and then some links to documents with more information:
(http://deliveryimages.acm.org/10.1145/1320000/1317403/table1.jpg)
(from the first document mentioned below)
- Hard disk drives: the good, the bad and the ugly! (http://queue.acm.org/detail.cfm?id=1317403) (Jon Elerath, Network Appliance, 2007)
- Reliability analysis of disk drive failure mechanisms (http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?tp=&arnumber=1408366&isnumber=30517) (S. Shah, J.G. Elerath, Reliability and Maintainability Symposium, 2005. Proceedings. ISBN: 0-7803-8824-0)
There were a couple of interesting talks in last year's IDEMA symposium, but I haven't found any web version of those. As I didn't go there myself, I don't know what was said, either.
-
Hi guys. I think that the latest posts are off my original topic a bit. To the casual observer, it would now appear to be a "___ssing contest". I suggest that we cool it for the benefit of others or take your banter off line.
Happy New Year.
Bud
-
I don't think you should be one to complain about "smug dismissiveness".
Even if your experience is 27 years and running, it's still anecdotal evidence until that you can show some research in the field of hard drive reliability, or you can show that there is research by other people supporting your claims.
I know exactly what you're saying:
Admit nothing.
Deny everything.
Demand proof, then refuse to accept it.
Put differently, I don't have any obligation to prove anything. If someone wants more info regarding what I've posted, let them have the burden of asking.
-
Whatever you do, a RAID0 is not for backup, it is not a safe way to _store_ data... use a raid5 or better to store data... in addition, copy the contents of your main data backup onto 1TB disks, and bring them to another place every 4-8 weeks, and do a complete copy over every 12 months on those externally stored disks, to make sure that the data doesn't degrade.
I have a RAID system, but really don't trust it. I make CD backups, but I also have a backup hard drive which is kept in a fireproof container. I've been reading about the MOZY backup system and may give it a try. It sounds like a great idea.
-
Hi guys. I think that the latest posts are off my original topic a bit. To the casual observer, it would now appear to be a "___ssing contest". I suggest that we cool it for the benefit of others or take your banter off line.
Happy New Year.
Bud
I expected better from you. My comments were totally logical. I reject the elitist notion that I'm required to prove my points, and I reject your claim that I'm not supposed to say same, here and now.
-
I reject the elitist notion that I'm required to prove my points..
Elitist??
If a point is debatable, commonsense requires some form of evidence/rational from either side.
If not when did sensible become elitist?
-
Elitist??
If a point is debatable, commonsense requires some form of evidence/rational from either side.
If not when did sensible become elitist?
Since when did you get to decide there was a debate? I provided information only, not debate points. And when did you become the arbiter of common sense? Not by my vote. No chance of that.
-
Backup/Archiving is such a pain, simply as there is no real easy solution.
All media has issues of longevity whether on tape, optical, hard disc and copying from media to media over time is liable to issues, mainly the human problem of doing it correctly and often enough.
Plus every storage media that I've ever used has failed at some time.
Online storage would be a great solution if the storage of the data is in many redundant locations. Sadly the upload speeds are simply not even close to fast enough for most of us who produce a lot of data, particularly professionals. The amount of data I create is way faster that I can upload. Despite having a 8M download, upload is a pathetic 384k.
-
Since when did you get to decide there was a debate? I provided information only, not debate points. And when did you become the arbiter of common sense? Not by my vote. No chance of that.
Someone appears to have got out of the wrong side of bed and repeatedly smacked head against wall!
Not only that, you don't seem to understand the concept of public forums. I also offered an opinion, not a manifesto for my political ascendency, so your vote is not actually required. And your 'information' was in fact debatable, hence the discussion that followed.
-
Someone appears to have got out of the wrong side of bed and repeatedly smacked head against wall!
Not only that, you don't seem to understand the concept of public forums. I also offered an opinion, not a manifesto for my political ascendency, so your vote is not actually required. And your 'information' was in fact debatable, hence the discussion that followed.
Your use of the several emoticons does indicate your emotional problem, which as I've said, you need to seek help for, elsewhere.
-
Backup/Archiving is such a pain, simply as there is no real easy solution.
All media has issues of longevity whether on tape, optical, hard disc and copying from media to media over time is liable to issues, mainly the human problem of doing it correctly and often enough.
Plus every storage media that I've ever used has failed at some time.
Online storage would be a great solution if the storage of the data is in many redundant locations. Sadly the upload speeds are simply not even close to fast enough for most of us who produce a lot of data, particularly professionals. The amount of data I create is way faster that I can upload. Despite having a 8M download, upload is a pathetic 384k.
Actually backup and archiving is not such a problem, but it can be expensive and time consuming. Researching and then buying reliable equipment is a big factor in reducing your pain. Another important step is getting the right software to do what you want. Another big factor is the planning you do for how you store things - folder and file naming for example - the hierarchy of your system. If you get way down the road and then have to change the way you name and store things, you should have a plan in advance for that possibility, otherwise it's a lot of extra work. And file and folder naming isn't much different than what people did with the old-fashioned file cabinets. But there is one difference - those who know how to create scripts and parsers etc. can do most of the restructuring chores automatically when that becomes necessary. And when it comes to media failure, I haven't had a hard disk fail ever, even those I've used for as many as seven years running daily. I just buy reliable brands. Lastly, if you run software that generates so much data that you do have trouble keeping track of it, you need either a better filing plan, or a way to generate less information and still accomplish what you need. BTW, any file that I back up is backed up in at least half a dozen places, except those files that are permanently archived, probably not to be seen again. Those may be stored in only two or three places, depending on my confidence in the storage media (very high or very, very high).
-
Hi guys. I think that the latest posts are off my original topic a bit. To the casual observer, it would now appear to be a "___ssing contest". I suggest that we cool it for the benefit of others or take your banter off line.
Happy New Year.
I apologize for my participation in that decline of usefulness of this thread, that was not my intention, I suppose I was bit by the "someone was wrong on the Internet" syndrome (and that's no excuse, merely an explanation).
Happy new year to you, too!
To get back to the case in question:
I agree with your last sentence in particular. If I have redundant and recent duplicates of my working files. If they are on multiple RAID set ups, the chances of all RAID systems failing at the same time are pretty remote. Using a local redundant RAID array for repeatable backups is fine as long as I keep recent back ups off-site too.
I would also make and immediately check cryptographic checksums of all files at the time of creation or modification, and keep and copy this metadata along with the data.
This makes it easier to detect anomalies that happen after writing the original files, and verify whether all copies (including the original) are correct.
From what I can read of the online features and specs for Chronosync and Synchronize X Pro, these products don't seem to be creating checksums. There are some vague phrases in Synchronize X Pro about checking which file has changed that may have something to do with this. However, this usually just means that the software checks the filesystem's metadata for changes, which will be irrelevant in the cases of corruption that I'm mentioning.
If neither of the products perform such checksums and checks automatically, and if you know a little bit about the Unix shell, PM me if you're interested in details on how to go about this.
I personally also make copies of all images and checksums to two identical DVDs with data verification after burning, which means that I have my data both on magnetic and optical media, which are all physically separated of course.
For really technically minded people, I'm tempted to recommend using Solaris with ZFS for storage management, also for the disk-based copies, and e.g. BackupPC for handling the actual copying (though Synchronize X Pro seems to have a very similar feature set for that purpose). Note: even I don't use Solaris + ZFS personally, because I'm loath to administer Solaris at home; I've decided to take the risks I get by not doing so, and I certainly can't afford the relevant products from NetApp or EMC.
Finally, I'd like to say that the risks you're taking aren't great. Your level of paranoia seems sufficient to protect you from all but unlikely cases of catastrophic loss of many images. And that may be enough. Perhaps occasionally copying important files to DVD or CD is a prudent, additional measure.
-
Anyone archiving using Blu-ray disks? I like to keep back-up copies of all my stuff (incl. all my .flac files from my CD collection) in TWO different off-site locations, so this is something I'm looking at. I know HDs are dirt cheap right now, but don't optical disks have some advantages for archiving?
-
All of the suggestions about checksums and other complexities sound OK until you try them and get bogged down with the work and can't get anything else done. I simply connect two drives containing the same backup files and then execute a single text 'batch' file that compares the contents of all files on drive 'x with those on drive 'y'. A batch file contains one line for each folder like so:
diff %1:\folder1 %2:\folder1
diff %1:\folder2 %2:\folder2
diff %1:\folder3 %2:\folder3
..........
This simple automation will eliminate the need for checksums etc. for simple backup chores, no matter how large. The 'diff' program compares the contents of all files in folder 'n' on drive %1 with the same folder on drive %2.
The use of DVD's etc. is OK when you don't ever need to update a backup with a new file or changed file. Once you get into that, hard disks are the only viable solution.
-
Anyone archiving using Blu-ray disks? I like to keep back-up copies of all my stuff (incl. all my .flac files from my CD collection) in TWO different off-site locations, so this is something I'm looking at. I know HDs are dirt cheap right now, but don't optical disks have some advantages for archiving?
I haven't even considered using Blu-ray for storage yet, simply because the cost of the media is so high that its benefits compared to DVD+R are negligible.
There are several schools of thought regarding media reliability. Here is a brief (and therefore incomplete and inaccurate) list of pros and cons when used for backup copies:
Harddisks:
+ high to very high capacity
+ fast
+ cheap
+ good at random access
- moving parts
- sensitive to electromagnetic fields
- sensitive to moisture and dust
Flash-based media (current solid state disks, CF cards, etc):
+ medium to high capacity
+ fast to very fast
+ no moving parts
+ some media are exceptionally robust against moisture and dust
+ very good at random access
- sensitive to electromagnetic fields
- unproven in terms of long-term reliability
- expensive
Tapes:
+ medium to high capacity
+ proven in terms of reliability (some technologies are bad, though)
- moving parts
- sensitive to electromagnetic fields
- medium expensive
- bad for random access
Optical disks (CD, DVD, Blu-ray):
+ resistant to electromagnetic fields
+ potentially very good reliability and long term properties
+ no moving parts (but spins when in the reader/writer)
- very low or low capacity
- may be sensitive to air pollutants
- some choices of chemical dyes lead to media degradation
- sensitive to light
- sensitive to scratches
I've posted quite a bit about backup strategies before, and I still think that keeping data on different kinds of media, as well as migrating to new copies every few years, is a good strategy for securing the data.
-
All of the suggestions about checksums and other complexities sound OK until you try them and get bogged down with the work and can't get anything else done. I simply connect two drives containing the same backup files and then execute a single text 'batch' file that compares the contents of all files on drive 'x with those on drive 'y'. A batch file contains one line for each folder like so:
diff %1:\folder1 %2:\folder1
diff %1:\folder2 %2:\folder2
diff %1:\folder3 %2:\folder3
..........
This simple automation will eliminate the need for checksums etc. for simple backup chores, no matter how large. The 'diff' program compares the contents of all files in folder 'n' on drive %1 with the same folder on drive %2.
That's fine, but it won't tell you which of the copy and the "original" is the file that's corrupted. Cryptographic checksums will. So your solution does not replace checksums. I'm beginning to think that you don't know what a cryptographic checksum is. You could, of course, keep several backups (e.g. four) and decide which is the corrupt copy by a simple quorum. But then it's beginning to look like you're going to "get bogged down with the work".
Also, your solution is Windows specific, completely non-portable, and relies on a program that's not included in regular Windows installations up to and including Windows XP.
Ensuring your data integrity is a bit of work - that is, unless you use a backup system that does it for you.
BackupPC (http://backuppc.sourceforge.net/) seems to do it, it's cross-platform (Windows, MacOS X, Linux, ...), and it's free (both as in beer and speech). I've been reluctant to tout this particular feature of the backup software, because I don't have any particular experience with it myself.
-
Actually backup and archiving is not such a problem, but it can be expensive and time consuming.
And is exactly why it can be such a problem, duh!
And when it comes to media failure, I haven't had a hard disk fail ever, even those I've used for as many as seven years running daily. I just buy reliable brands.
Aren't you blessed. I've tried them all and Samsung is the only brand of HD that as of yet haven't failed on me. No such thing as a reliable HD, just one that hasn't yet died.
Buying IBM used to be sensible as they were supposedly so reliable, until they made a dodgy batch and stupidly denied it. I had several Deathstars die on me.
Both Lexar and Sandisk cards have failed on me and they are supposedly the high end of cards.
Lastly, if you run software that generates so much data that you do have trouble keeping track of it, you need either a better filing plan, or a way to generate less information and still accomplish what you need.
How I file has no bearing on the space used, keeping track is a completely separate issue. Not taking photographs or using a 2.1mp camera, reduces my HD needs, but both are pretty stupid solutions.
-
And is exactly why it can be such a problem, duh!
Aren't you blessed. I've tried them all and Samsung is the only brand of HD that as of yet haven't failed on me. No such thing as a reliable HD, just one that hasn't yet died.
Buying IBM used to be sensible as they were supposedly so reliable, until they made a dodgy batch and stupidly denied it. I had several Deathstars die on me.
Both Lexar and Sandisk cards have failed on me and they are supposedly the high end of cards.
How I file has no bearing on the space used, keeping track is a completely separate issue. Not taking photographs or using a 2.1mp camera, reduces my HD needs, but both are pretty stupid solutions.
This guy promised to ignore, and now he's back already. I guess we can add "big liar" and "stalker" to the distinctions he's already accrued.
BTW, none of the claims 'jjj' makes have any merit whatsoever. What would you expect of such a bad liar?
-
Edit...... double post
-
This guy promised to ignore, and now he's back already. I guess we can add "big liar" and "stalker" to the distinctions he's already accrued.
BTW, none of the claims 'jjj' makes have any merit whatsoever. What would you expect of such a bad liar?
Uh I was simply replying to a post which I didn't realise it was you at first, as some of it made sense.
And having started to reply, I then simply posted it. What an evil, mendacious and dissembling person I must be for doing such a thing!!
Are you the forum drunk? Maybe you are the the first on this forum, not come across one for a while. They can be entertaining in their flailing around and at least you can't smell the fetid breath or vomit on their shirt front when online.
I see you still avoid answering people's posts, when your daft points are queried.
-
Uh I was simply replying to a post which I didn't realise it was you at first, as some of it made sense.
And having started to reply, I then simply posted it. What an evil, mendacious and dissembling person I must be for doing such a thing!!
Are you the forum drunk? Maybe you are the the first on this forum, not come across one for a while. They can be entertaining in their flailing around and at least you can't smell the fetid breath or vomit on their shirt front when online.
I see you still avoid answering people's posts, when your daft points are queried.
I hate it when I'm right about sick people. But stalking is getting pretty serious. Are you sure you want to do that?
-
Meanwhile, back at the ranch...
As a hobbyist, I don't have the load that quite a few of you have to deal with, but there may be some who find merit in my method.
I'm not a fan of the turnkey automated systems, such as the Drobo and dedicated backup firewire drives, as most utilize proprietary hardware and software that if fails, leaves you twisting in the wind. I put together an OWC Mercury Elite Pro dual drive enclosure with two 500GB Seagate 7200RPM drives. This connects to both FW400 and FW800 and I recently picked up one of the early '08 MBP's so the backup is pretty zippy at 800. Each drive was partitioned with one for a bootable backup of my 'puter and the other as an image archive. I use SuperDuper to manually backup the system to both drives independently as needed and the enclosure is turned off when not in use. I also have a BlacX box, a standalone drive interface that bare drives plug into, with two more of the Seagates that I swap between home and a safe deposit box.
This means there are four bootable copies of my system and image files at all times, one of which is sitting in a safe deposit box. If something happens to that one, I won't be around to care anyhoo. And than there's the Mini, used for 'puter backup if the laptop fails. Overkill maybe, but after one disaster that resulted in lost files there aint no such thing.
I had started out with DVD's, but even with dual layer discs the stack was building faster than spent ink carts from an Epson.
-
BTW, none of the claims 'jjj' makes have any merit whatsoever. What would you expect of such a bad liar?
This "merit" is, of course, Newspeak for "fault".