Pages: 1 [2] 3   Go Down

Author Topic: Check in with how you are archiving and backing up images  (Read 14922 times)

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #20 on: January 02, 2009, 01:33:27 pm »

Quote from: jani
I don't think you should be one to complain about "smug dismissiveness".
Even if your experience is 27 years and running, it's still anecdotal evidence until that you can show some research in the field of hard drive reliability, or you can show that there is research by other people supporting your claims.
I know exactly what you're saying:
Admit nothing.
Deny everything.
Demand proof, then refuse to accept it.

Put differently, I don't have any obligation to prove anything. If someone wants more info regarding what I've posted, let them have the burden of asking.
Logged

JohnBrew

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 868
    • http://www.johnbrewton.zenfolio.com
Check in with how you are archiving and backing up images
« Reply #21 on: January 02, 2009, 02:26:56 pm »

Quote from: joergen geerds
Whatever you do, a RAID0 is not for backup, it is not a safe way to _store_ data... use a raid5 or better to store data... in addition, copy the contents of your main data backup onto 1TB disks, and bring them to another place every 4-8 weeks, and do a complete copy over every 12 months on those externally stored disks, to make sure that the data doesn't degrade.

I have a RAID system, but really don't trust it. I make CD backups, but I also have a backup hard drive which is kept in a fireproof container. I've been reading about the MOZY backup system and may give it a try. It sounds like a great idea.

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #22 on: January 02, 2009, 02:52:28 pm »

Quote from: budjames
Hi guys. I think that the latest posts are off my original topic a bit. To the casual observer, it would now appear to be a "___ssing contest". I suggest that we cool it for the benefit of others or take your banter off line.

Happy New Year.
Bud
I expected better from you. My comments were totally logical. I reject the elitist notion that I'm required to prove my points, and I reject your claim that I'm not supposed to say same, here and now.
Logged

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4728
    • http://www.futtfuttfuttphotography.com
Check in with how you are archiving and backing up images
« Reply #23 on: January 02, 2009, 03:37:50 pm »

Quote from: dalethorn
I reject the elitist notion that I'm required to prove my points..
   Elitist??
If a point is debatable, commonsense requires some form of evidence/rational from either side.  
If not when did sensible become elitist?
Logged
Tradition is the Backbone of the Spinele

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #24 on: January 02, 2009, 03:49:34 pm »

Quote from: jjj
 Elitist??
If a point is debatable, commonsense requires some form of evidence/rational from either side.  
If not when did sensible become elitist?
Since when did you get to decide there was a debate? I provided information only, not debate points. And when did you become the arbiter of common sense? Not by my vote. No chance of that.
Logged

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4728
    • http://www.futtfuttfuttphotography.com
Check in with how you are archiving and backing up images
« Reply #25 on: January 02, 2009, 03:59:46 pm »

Backup/Archiving is such a pain, simply as there is no real easy solution.
All media has issues of longevity whether on tape, optical, hard disc and copying from media to media over time is liable to issues, mainly the human problem of doing it correctly and often enough.
Plus every storage media that I've ever used has failed at some time.


Online storage would be a great solution if the storage of the data is in many redundant locations. Sadly the upload speeds are simply not even close to fast enough for most of us who produce a lot of data, particularly professionals. The amount of data I create is way faster that I can upload. Despite having a 8M download, upload is a pathetic 384k.
Logged
Tradition is the Backbone of the Spinele

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4728
    • http://www.futtfuttfuttphotography.com
Check in with how you are archiving and backing up images
« Reply #26 on: January 02, 2009, 04:05:53 pm »

Quote from: dalethorn
Since when did you get to decide there was a debate? I provided information only, not debate points. And when did you become the arbiter of common sense? Not by my vote. No chance of that.
Someone appears to have got out of the wrong side of bed and repeatedly smacked head against wall!  
Not only that, you don't seem to understand the concept of public forums. I also offered an opinion, not a manifesto for my political ascendency, so your vote is not actually required.    And your 'information' was in fact debatable, hence the discussion that followed.
Logged
Tradition is the Backbone of the Spinele

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #27 on: January 02, 2009, 04:57:44 pm »

Quote from: jjj
Someone appears to have got out of the wrong side of bed and repeatedly smacked head against wall!  
Not only that, you don't seem to understand the concept of public forums. I also offered an opinion, not a manifesto for my political ascendency, so your vote is not actually required.    And your 'information' was in fact debatable, hence the discussion that followed.
Your use of the several emoticons does indicate your emotional problem, which as I've said, you need to seek help for, elsewhere.
Logged

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #28 on: January 02, 2009, 05:12:20 pm »

Quote from: jjj
Backup/Archiving is such a pain, simply as there is no real easy solution.
All media has issues of longevity whether on tape, optical, hard disc and copying from media to media over time is liable to issues, mainly the human problem of doing it correctly and often enough.
Plus every storage media that I've ever used has failed at some time.


Online storage would be a great solution if the storage of the data is in many redundant locations. Sadly the upload speeds are simply not even close to fast enough for most of us who produce a lot of data, particularly professionals. The amount of data I create is way faster that I can upload. Despite having a 8M download, upload is a pathetic 384k.
Actually backup and archiving is not such a problem, but it can be expensive and time consuming.  Researching and then buying reliable equipment is a big factor in reducing your pain.  Another important step is getting the right software to do what you want.  Another big factor is the planning you do for how you store things - folder and file naming for example - the hierarchy of your system.  If you get way down the road and then have to change the way you name and store things, you should have a plan in advance for that possibility, otherwise it's a lot of extra work.  And file and folder naming isn't much different than what people did with the old-fashioned file cabinets.  But there is one difference - those who know how to create scripts and parsers etc. can do most of the restructuring chores automatically when that becomes necessary.  And when it comes to media failure, I haven't had a hard disk fail ever, even those I've used for as many as seven years running daily.  I just buy reliable brands.  Lastly, if you run software that generates so much data that you do have trouble keeping track of it, you need either a better filing plan, or a way to generate less information and still accomplish what you need.  BTW, any file that I back up is backed up in at least half a dozen places, except those files that are permanently archived, probably not to be seen again.  Those may be stored in only two or three places, depending on my confidence in the storage media (very high or very, very high).
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #29 on: January 02, 2009, 11:42:34 pm »

Quote from: budjames
Hi guys. I think that the latest posts are off my original topic a bit. To the casual observer, it would now appear to be a "___ssing contest". I suggest that we cool it for the benefit of others or take your banter off line.

Happy New Year.
I apologize for my participation in that decline of usefulness of this thread, that was not my intention, I suppose I was bit by the "someone was wrong on the Internet" syndrome (and that's no excuse, merely an explanation).

Happy new year to you, too!

To get back to the case in question:
Quote from: budjames
I agree with your last sentence in particular. If I have redundant and recent duplicates of my working files. If they are on multiple RAID set ups, the chances of all RAID systems failing at the same time are pretty remote. Using a local redundant RAID array for repeatable backups is fine as long as I keep recent back ups off-site too.
I would also make and immediately check cryptographic checksums of all files at the time of creation or modification, and keep and copy this metadata along with the data.

This makes it easier to detect anomalies that happen after writing the original files, and verify whether all copies (including the original) are correct.

From what I can read of the online features and specs for Chronosync and Synchronize X Pro, these products don't seem to be creating checksums. There are some vague phrases in Synchronize X Pro about checking which file has changed that may have something to do with this. However, this usually just means that the software checks the filesystem's metadata for changes, which will be irrelevant in the cases of corruption that I'm mentioning.

If neither of the products perform such checksums and checks automatically, and if you know a little bit about the Unix shell, PM me if you're interested in details on how to go about this.

I personally also make copies of all images and checksums to two identical DVDs with data verification after burning, which means that I have my data both on magnetic and optical media, which are all physically separated of course.

For really technically minded people, I'm tempted to recommend using Solaris with ZFS for storage management, also for the disk-based copies, and e.g. BackupPC for handling the actual copying (though Synchronize X Pro seems to have a very similar feature set for that purpose). Note: even I don't use Solaris + ZFS personally, because I'm loath to administer Solaris at home; I've decided to take the risks I get by not doing so, and I certainly can't afford the relevant products from NetApp or EMC.

Finally, I'd like to say that the risks you're taking aren't great. Your level of paranoia seems sufficient to protect you from all but unlikely cases of catastrophic loss of many images. And that may be enough. Perhaps occasionally copying important files to DVD or CD is a prudent, additional measure.
Logged
Jan

PSA DC-9-30

  • Full Member
  • ***
  • Offline Offline
  • Posts: 207
Check in with how you are archiving and backing up images
« Reply #30 on: January 03, 2009, 11:19:21 pm »

Anyone archiving using Blu-ray disks? I like to keep back-up copies of all my stuff (incl. all my .flac files from my CD collection) in TWO different off-site locations, so this is something I'm looking at. I know HDs are dirt cheap right now, but don't optical disks have some advantages for archiving?
Logged

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #31 on: January 04, 2009, 08:02:14 am »

All of the suggestions about checksums and other complexities sound OK until you try them and get bogged down with the work and can't get anything else done. I simply connect two drives containing the same backup files and then execute a single text 'batch' file that compares the contents of all files on drive 'x with those on drive 'y'.  A batch file contains one line for each folder like so:
diff %1:\folder1 %2:\folder1
diff %1:\folder2 %2:\folder2
diff %1:\folder3 %2:\folder3
..........
This simple automation will eliminate the need for checksums etc. for simple backup chores, no matter how large. The 'diff' program compares the contents of all files in folder 'n' on drive %1 with the same folder on drive %2.

The use of DVD's etc. is OK when you don't ever need to update a backup with a new file or changed file. Once you get into that, hard disks are the only viable solution.
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #32 on: January 04, 2009, 09:21:41 am »

Quote from: PSA DC-9-30
Anyone archiving using Blu-ray disks? I like to keep back-up copies of all my stuff (incl. all my .flac files from my CD collection) in TWO different off-site locations, so this is something I'm looking at. I know HDs are dirt cheap right now, but don't optical disks have some advantages for archiving?
I haven't even considered using Blu-ray for storage yet, simply because the cost of the media is so high that its benefits compared to DVD+R are negligible.

There are several schools of thought regarding media reliability. Here is a brief (and therefore incomplete and inaccurate) list of pros and cons when used for backup copies:

Harddisks:

 + high to very high capacity
 + fast
 + cheap
 + good at random access
 - moving parts
 - sensitive to electromagnetic fields
 - sensitive to moisture and dust

Flash-based media (current solid state disks, CF cards, etc):

 + medium to high capacity
 + fast to very fast
 + no moving parts
 + some media are exceptionally robust against moisture and dust
 + very good at random access
 - sensitive to electromagnetic fields
 - unproven in terms of long-term reliability
 - expensive

Tapes:

 + medium to high capacity
 + proven in terms of reliability (some technologies are bad, though)
 - moving parts
 - sensitive to electromagnetic fields
 - medium expensive
 - bad for random access

Optical disks (CD, DVD, Blu-ray):

 + resistant to electromagnetic fields
 + potentially very good reliability and long term properties
 + no moving parts (but spins when in the reader/writer)
 - very low or low capacity
 - may be sensitive to air pollutants
 - some choices of chemical dyes lead to media degradation
 - sensitive to light
 - sensitive to scratches

I've posted quite a bit about backup strategies before, and I still think that keeping data on different kinds of media, as well as migrating to new copies every few years, is a good strategy for securing the data.
Logged
Jan

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Check in with how you are archiving and backing up images
« Reply #33 on: January 04, 2009, 03:48:00 pm »

Quote from: dalethorn
All of the suggestions about checksums and other complexities sound OK until you try them and get bogged down with the work and can't get anything else done. I simply connect two drives containing the same backup files and then execute a single text 'batch' file that compares the contents of all files on drive 'x with those on drive 'y'.  A batch file contains one line for each folder like so:
diff %1:\folder1 %2:\folder1
diff %1:\folder2 %2:\folder2
diff %1:\folder3 %2:\folder3
..........
This simple automation will eliminate the need for checksums etc. for simple backup chores, no matter how large. The 'diff' program compares the contents of all files in folder 'n' on drive %1 with the same folder on drive %2.
That's fine, but it won't tell you which of the copy and the "original" is the file that's corrupted. Cryptographic checksums will. So your solution does not replace checksums. I'm beginning to think that you don't know what a cryptographic checksum is. You could, of course, keep several backups (e.g. four) and decide which is the corrupt copy by a simple quorum. But then it's beginning to look like you're going to "get bogged down with the work".

Also, your solution is Windows specific, completely non-portable, and relies on a program that's not included in regular Windows installations up to and including Windows XP.

Ensuring your data integrity is a bit of work - that is, unless you use a backup system that does it for you.

BackupPC seems to do it, it's cross-platform (Windows, MacOS X, Linux, ...), and it's free (both as in beer and speech). I've been reluctant to tout this particular feature of the backup software, because I don't have any particular experience with it myself.
Logged
Jan

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4728
    • http://www.futtfuttfuttphotography.com
Check in with how you are archiving and backing up images
« Reply #34 on: January 04, 2009, 04:30:52 pm »

Quote from: dalethorn
Actually backup and archiving is not such a problem, but it can be expensive and time consuming.
And is exactly why it can be such a problem, duh!

Quote
And when it comes to media failure, I haven't had a hard disk fail ever, even those I've used for as many as seven years running daily.  I just buy reliable brands.
Aren't you blessed. I've tried them all and Samsung is the only brand of HD that as of yet haven't failed on me. No such thing as a reliable HD, just one that hasn't yet died.
Buying IBM used to be sensible as they were supposedly so reliable, until they made a dodgy batch and stupidly denied it. I had several Deathstars die on me.
Both Lexar and Sandisk cards have failed on me and they are supposedly the high end of cards.


 
Quote
Lastly, if you run software that generates so much data that you do have trouble keeping track of it, you need either a better filing plan, or a way to generate less information and still accomplish what you need.
How I file has no bearing on the space used, keeping track is a completely separate issue. Not taking photographs or using a 2.1mp camera, reduces my HD needs, but both are pretty stupid solutions.
Logged
Tradition is the Backbone of the Spinele

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #35 on: January 04, 2009, 06:00:18 pm »

Quote from: jjj
And is exactly why it can be such a problem, duh!

 Aren't you blessed. I've tried them all and Samsung is the only brand of HD that as of yet haven't failed on me. No such thing as a reliable HD, just one that hasn't yet died.
Buying IBM used to be sensible as they were supposedly so reliable, until they made a dodgy batch and stupidly denied it. I had several Deathstars die on me.
Both Lexar and Sandisk cards have failed on me and they are supposedly the high end of cards.


 How I file has no bearing on the space used, keeping track is a completely separate issue. Not taking photographs or using a 2.1mp camera, reduces my HD needs, but both are pretty stupid solutions.
This guy promised to ignore, and now he's back already.  I guess we can add "big liar" and "stalker" to the distinctions he's already accrued.

BTW, none of the claims 'jjj' makes have any merit whatsoever.  What would you expect of such a bad liar?
« Last Edit: January 04, 2009, 06:04:25 pm by dalethorn »
Logged

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4728
    • http://www.futtfuttfuttphotography.com
Check in with how you are archiving and backing up images
« Reply #36 on: January 04, 2009, 06:29:23 pm »

Edit...... double post
« Last Edit: January 04, 2009, 06:31:16 pm by jjj »
Logged
Tradition is the Backbone of the Spinele

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4728
    • http://www.futtfuttfuttphotography.com
Check in with how you are archiving and backing up images
« Reply #37 on: January 04, 2009, 06:30:35 pm »

Quote from: dalethorn
This guy promised to ignore, and now he's back already.  I guess we can add "big liar" and "stalker" to the distinctions he's already accrued.

BTW, none of the claims 'jjj' makes have any merit whatsoever.  What would you expect of such a bad liar?
Uh I was simply replying to a post which I didn't realise it was you at first, as some of it made sense.
And having started to reply, I then simply posted it. What an evil, mendacious and dissembling person I must be for doing such a thing!!

Are you the forum drunk? Maybe you are the the first on this forum, not come across one for a while. They can be entertaining in their flailing around and at least you can't smell the fetid breath or vomit on their shirt front when online.    
 

I see you still avoid answering people's posts, when your daft points are queried.
Logged
Tradition is the Backbone of the Spinele

dalethorn

  • Guest
Check in with how you are archiving and backing up images
« Reply #38 on: January 04, 2009, 07:55:59 pm »

Quote from: jjj
Uh I was simply replying to a post which I didn't realise it was you at first, as some of it made sense.
And having started to reply, I then simply posted it. What an evil, mendacious and dissembling person I must be for doing such a thing!!

Are you the forum drunk? Maybe you are the the first on this forum, not come across one for a while. They can be entertaining in their flailing around and at least you can't smell the fetid breath or vomit on their shirt front when online.    
 

I see you still avoid answering people's posts, when your daft points are queried.
I hate it when I'm right about sick people.  But stalking is getting pretty serious.  Are you sure you want to do that?
Logged

schrodingerscat

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 374
Check in with how you are archiving and backing up images
« Reply #39 on: January 08, 2009, 12:01:06 am »

Meanwhile, back at the ranch...

As a hobbyist, I don't have the load that quite a few of you have to deal with, but there may be some who find merit in my method.

I'm not a fan of the turnkey automated systems, such as the Drobo and dedicated backup firewire drives, as most utilize proprietary hardware and software that if fails, leaves you twisting in the wind. I put together an OWC Mercury Elite Pro dual drive enclosure with two 500GB Seagate 7200RPM drives. This connects to both FW400 and FW800 and I recently picked up one of the early '08 MBP's so the backup is pretty zippy at 800. Each drive was partitioned with one for a bootable backup of my 'puter  and the other as an image archive. I use SuperDuper to manually backup the system to both drives independently as needed and the enclosure is turned off when not in use. I also have a BlacX box, a standalone drive interface that bare drives plug into, with two more of the Seagates that I swap between home and a safe deposit box.

This means there are four bootable copies of my system and image files at all times, one of which is sitting in a safe deposit box. If something happens to that one, I won't be around to care anyhoo. And than there's the Mini, used for 'puter backup if the laptop fails. Overkill maybe, but after one disaster that resulted in lost files there aint no such thing.

I had started out with DVD's, but even with dual layer discs the stack was building faster than spent ink carts from an Epson.
Logged
Pages: 1 [2] 3   Go Up