Pages: 1 [2] 3   Go Down

Author Topic: Next step for archive storage  (Read 35986 times)

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4726
    • http://www.futtfuttfuttphotography.com
Next step for archive storage
« Reply #20 on: January 31, 2008, 02:21:58 pm »

Quote
jjj --- on the corruptions, you need to verify the integrity regularly of of any disks you intend to back-up, whether you are writing to a second HD or optical disk....
[a href=\"index.php?act=findpost&pid=171281\"][{POST_SNAPBACK}][/a]
I'm not sure they didn't corrupt when drive  filled up or when copying, as the drives are OK.
Logged
Tradition is the Backbone of the Spinele

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #21 on: February 01, 2008, 05:31:35 am »

The bottom line for me is, I've never lost data that I've recorded to optical disc but I have lost heaps of data recorded on hard drives, but fortunately no permanent loss of RAW images because all my RAW images are on DVD.

On the matter of cost, Blu-ray is in its infancy and very much more expensive per gigabyte than the most economical hard drives, but apparently not more expensive than pocket drives. However, for travelling, pocket drives have the advantage of being lighter and less bulky than current single-sided Blu-ray.

As far as I see from prices on the internet, DVDs which are now a mature technology are the cheapest method of storage, but clearly not the most efficient in terms of bulk, weight and speed of recording and retrieval of data.

It will be interesting to see if the eventuall drop in price of recordable Blu-ray, as the technology matures, results in it becoming the cheapest method of storage.

Bear in mind also that one double-sided Blu-ray disc has the potential to store 200GB of data as more layers are added. I believe the specs allow for 4 layers per side.
Logged

Jack Flesher

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2592
    • www.getdpi.com
Next step for archive storage
« Reply #22 on: February 01, 2008, 11:24:59 am »

Quote
The bottom line for me is, I've never lost data that I've recorded to optical disc but I have lost heaps of data recorded on hard drives,
[a href=\"index.php?act=findpost&pid=171463\"][{POST_SNAPBACK}][/a]

I'd be interesting to hear the details on the drives...  Regardless, your back-up plan worked for you and that's what counts

I've had single hard drives fail, but having had all my data redundantly backed up, it was not disastrous.  in my experience the only folks I know who have permanently lost data from a hard drive failure either did not have proper redundant data back-up, or they relied on a single RAID5 array to provide it...

Cheers,
« Last Edit: February 01, 2008, 11:26:05 am by Jack Flesher »
Logged
Jack
[url=http://forum.getdpi.com/forum/

Farkled

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 97
Next step for archive storage
« Reply #23 on: February 01, 2008, 12:19:34 pm »

Quote
I'd be interesting to hear the details on the drives...  Regardless, your back-up plan worked for you and that's what counts

I've had single hard drives fail, but having had all my data redundantly backed up, it was not disastrous.  in my experience the only folks I know who have permanently lost data from a hard drive failure either did not have proper redundant data back-up, or they relied on a single RAID5 array to provide it...

Cheers,
[a href=\"index.php?act=findpost&pid=171524\"][{POST_SNAPBACK}][/a]
Exactly.  All media will fail.  Plan for failure; expect failure.  That means redundancy.  There should never be a one and only copy of anything important.
Logged

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #24 on: February 02, 2008, 03:42:21 am »

Quote
I'd be interesting to hear the details on the drives...  Regardless, your back-up plan worked for you and that's what counts

I've had single hard drives fail, but having had all my data redundantly backed up, it was not disastrous.  in my experience the only folks I know who have permanently lost data from a hard drive failure either did not have proper redundant data back-up, or they relied on a single RAID5 array to provide it...

Cheers,
[a href=\"index.php?act=findpost&pid=171524\"][{POST_SNAPBACK}][/a]

Jack,
These are just single drives that have failed for one reason or another during the past 14 years I've been doing digital photography. Because I had all the important stuff bakced up on CD and later DVD, I didn't bother duplicating the data on other hard drives.

The most alarming failure occurred when the power supply in my computer exploded one day. It sounded almost like a gun shot. I replaced the power supply and was able to reboot the computer but wasn't able to open the folders on the D: drive, which was the bigger drive where most of the image files were stored.

I suspect the cause was geckos attracted by the heat of the power supply, crawling in through the wide spacing in the grill to lay their eggs. I taped a piece of mosquito netting to the ventilator grill of the new power supply and haven't had this occur again.

I know it's easier to have all data duplicated on different hard drives, which I'm now doing. I just find it curious that I've never come across an optical disc in my collection that once was readable and then became unreadable, except occasionally when there was a problem with a particular CD/DVD reader or its driver or firmware. Such problems have always been solved by using a different CD/DVD drive.
Logged

Jonathan Wienke

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5829
    • http://visual-vacations.com/
Next step for archive storage
« Reply #25 on: February 02, 2008, 10:39:57 am »

Quote
I just find it curious that I've never come across an optical disc in my collection that once was readable and then became unreadable, except occasionally when there was a problem with a particular CD/DVD reader or its driver or firmware.

You're just very lucky.
Logged

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
Next step for archive storage
« Reply #26 on: February 02, 2008, 02:30:29 pm »

Cavalry 1TB, (2 500Gb drives backing each other up) $239
http://www.newegg.com/Product/Product.aspx...N82E16822101084

a replacement drive for when the failure occurs $95
http://www.newegg.com/Product/Product.aspx...N82E16822136178

when one drive fails the other automatically rebuilds the replacement drive.
I ordered the replacement drive at the same time as the dual drive so when (not if!) the failure occurs I have a spare drive ready. $335 total for a little peace of mind.
Marc
« Last Edit: February 02, 2008, 06:42:17 pm by marcmccalmont »
Logged
Marc McCalmont

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #27 on: February 02, 2008, 11:39:12 pm »

Quote
You're just very lucky.
[a href=\"index.php?act=findpost&pid=171764\"][{POST_SNAPBACK}][/a]

Or perhaps very careful. I've never bought optical discs over the internet, for example. If the price looks too good to be true, it sometimes is.

Before setting off on this trip, I bought 4 spare BP-511A batteries through the internet, not because I needed 4 but because they were so damned cheap. The total cost including postage was less than the internet cost of one genuine Canon battery.

When I first charged them, one simply wouldn't charge. I was too busy with last minute packing and other matters to write letters of complaint or return the battery. 3 were sufficient. 2 months later, a second battery refused to charge and the remaining 2 are now not holding much charge another 2 months later, as though they are close to the end of their life. However, I'm still using the original batteries that came with my 20D and 5D.
Logged

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #28 on: February 03, 2008, 12:12:08 am »

Quote
...when one drive fails the other automatically rebuilds the replacement drive.
I ordered the replacement drive at the same time as the dual drive so when (not if!) the failure occurs I have a spare drive ready. $335 total for a little peace of mind.
Marc
[a href=\"index.php?act=findpost&pid=171809\"][{POST_SNAPBACK}][/a]

What happens in the event of a catastrophe; an unusual power surge, a direct lightning hit or a burglary?

For complete peace of mind, a separate copy of all the important files stored at a separate location is advisable. For me, the important files are the RAW images. They are all on optical media. However, I also keep a second external drive of organised RAW and processed images in a different location, but I do not back up my data regularly to this drive.

About a year ago I had an unfortunate experience with a new LaCie Big Disc (500GB). After many hours of work transferring images from a previous trip (from DVDs ) and after carefully organising the files into readily accessible subject matter, I was about 90% of the way through the task when the external drive malfunctioned and none of the data was able to be accessed. The computer just refused to recognise the drive. I suspect over-heating might have been the cause.

Fortunately, all I'd lost was my time. It could have been worse.
Logged

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4726
    • http://www.futtfuttfuttphotography.com
Next step for archive storage
« Reply #29 on: February 03, 2008, 03:26:56 pm »

Quote
Or perhaps very careful. I've never bought optical discs over the internet, for example. If the price looks too good to be true, it sometimes is.
[a href=\"index.php?act=findpost&pid=171920\"][{POST_SNAPBACK}][/a]
I've only ever bought good discs. I've had some fail.
A journalist on PCPro mag wrote one day he went back through his DVDs and found large No.s had deteriorated only six months later. Again good quality discs - in theory.

I find DVDs too slow and bulky, especially with laptop burners. If I'm away for 4 weeks and produce say 300G+ of data, that's probably 80-90 discs in reality.
My next laptop must have BluRay writing. So I'm hanging on to mine till then.


Quote
I suspect the cause was geckos attracted by the heat of the power supply, crawling in through the wide spacing in the grill to lay their eggs. I taped a piece of mosquito netting to the ventilator grill of the new power supply and haven't had this occur again.[a href=\"index.php?act=findpost&pid=171717\"][{POST_SNAPBACK}][/a]
If this had happened to early computers would we call problems in code, lizards?  
« Last Edit: February 03, 2008, 03:38:58 pm by jjj »
Logged
Tradition is the Backbone of the Spinele

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4726
    • http://www.futtfuttfuttphotography.com
Next step for archive storage
« Reply #30 on: February 03, 2008, 03:38:30 pm »

Quote
What happens in the event of a catastrophe; an unusual power surge, a direct lightning hit or a burglary?
[a href=\"index.php?act=findpost&pid=171926\"][{POST_SNAPBACK}][/a]
My back up computer is going in locked cellar on a separate mains ring [when builders have finished demolishing house and rebuilt it again, that is].
My office in attic is also locked with serious fire doors and very solid frame.
An RCD board is fitted and surge protectors are between mains and computer.
Backup machine is also very small and easily hidden.
Logged
Tradition is the Backbone of the Spinele

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #31 on: February 04, 2008, 12:49:52 am »

Quote
A journalist on PCPro mag wrote one day he went back through his DVDs and found large No.s had deteriorated only six months later. Again good quality discs - in theory.
[a href=\"index.php?act=findpost&pid=172032\"][{POST_SNAPBACK}][/a]

One has to wonder about such claims. What is meant by 'deteriorated'? Chemical break-down of the disc coatings? Were such apparently failed discs really recorded properly in the first instance? Did the journalist try the so-called 'deteriorated' discs in a few other drives?

I've written that I've never come across a failed disc in my collection of 2000+ CDs and DVDs, that I know was recorded properly and fully readable after recording.

But I have found that the occasional disc fails to read properly on one particular drive, but reads okay on another. I remember well the occasion when this first happened. It was a Kodak Photo-CD disc that was only 2 years old. I'd just updated my CD reader from a 4x to 20x, fully expecting that the 18MB Photo-CD files that used to take a full 2 minutes to open would then open in 30 seconds or less. Not so. Some files wouldn't open at all. Some did, taking about the same time of 2 minutes. Some looked as though they would open, eventually, but the process would slowly grind to a halt and they never made it.

Because the results were inconsistent from file to file, it was easy to jump to the conclusion that some kind of uneven deterioration of the disc had taken place. The problem was solved by returning to my old 4x CD drive. Those first Kodak Photo-CD discs, which are now about 15 years old, open flawlessly on my current Pioneer DVD burner. In fact, it would be true to say they actually perform better than they did 15 years ago. Each file used to take 2 minutes to be read and to open on the monitor. The process now takes about 5 seconds.
Logged

jjj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4726
    • http://www.futtfuttfuttphotography.com
Next step for archive storage
« Reply #32 on: February 04, 2008, 05:25:31 am »

Quote
One has to wonder about such claims. What is meant by 'deteriorated'? Chemical break-down of the disc coatings? Were such apparently failed discs really recorded properly in the first instance? Did the journalist try the so-called 'deteriorated' discs in a few other drives?
I've thrown the mag out now, but he had a batch that had in fact deteriorated, visibly IIRC.  I know you don't believe anything anyone else says [good up to a point], but sometimes, other people are actually correct.
As for using other drives  to read them, see my last point.

Quote
I've written that I've never come across a failed disc in my collection of 2000+ CDs and DVDs, that I know was recorded properly and fully readable after recording.
I doubt you've ever been murdered either, but I gather it's something that sadly happens to other people far too often.


Quote
Because the results were inconsistent from file to file, it was easy to jump to the conclusion that some kind of uneven deterioration of the disc had taken place. The problem was solved by returning to my old 4x CD drive. [a href=\"index.php?act=findpost&pid=172143\"][{POST_SNAPBACK}][/a]
Assuming you still have the old drive, it still fits in your computer and it also is still functional, that's still not  a good way to archive stuff, relying on old irreplacable, possibly incompatible kit.
Maybe it works in that drive as that is the drive that incorrectly wrote the discs in the first place, as that's another possible  reason for the read failure. The whole point of CDs, DVDs etc is they are usable by anyone with a CD/DVD reader. If that is not true they the archive has failed to be archival on quite a fundamental level.
Logged
Tradition is the Backbone of the Spinele

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #33 on: February 04, 2008, 07:52:44 am »

Quote
I've thrown the mag out now, but he had a batch that had in fact deteriorated, visibly IIRC.  I know you don't believe anything anyone else says [good up to a point], but sometimes, other people are actually correct.

Whatever the product, there are sometimes lemons that get through the quality control checks. Generally one might expect that the cheaper the product, the poorer the QC, but obviously that's not always the case.

There are also scams. I can imagine that in some disc factory in China, operating on a very narrow profit margin, a batch of CDs or DVDs produced with the wrong mix of chemical dyes, or with out-of-adjustment machinery, is officially consigned for destruction, but due to corruption at the work place the faulty consignment is sold to some shady dealer at a very low price who passes the discs on to another dealer who sells them to unsuspecting Americans or Englishmen on the internet.

Such is life! I think there's less risk of this happening if one buys from a reputable dealer.

Of course I know that other people are sometimes correct. But I also know they are sometimes incorrect. I also know that there's a very strong trait in human nature to blame others for one's own mistakes, to search for a scapegoat, and this is much easier when the scapegoat is an impersonal organisation.

Quote
I doubt you've ever been murdered either, but I gather it's something that sadly happens to other people far too often.

Murders are investigated and there's usually pretty conclusive evidence of the existence of a dead body. An incident of a disc that has chemically deteriorated in just 6 months should be investigated. They are supposed to last about 50 to 100 years, aren't they?

Quote
Assuming you still have the old drive, it still fits in your computer and it also is still functional, that's still not  a good way to archive stuff, relying on old irreplacable, possibly incompatible kit.

I think you missed my point. The solution at the time, 12 years ago, was to revert to using the previous CD drive that I'd had upgraded. The shop recognised that they'd bought a batch of substandard CD drives that had difficulty with the Photo-CD format and reinstalled my old drive free of charge. That was my first computer.

Quote
Maybe it works in that drive as that is the drive that incorrectly wrote the discs in the first place, as that's another possible  reason for the read failure. The whole point of CDs, DVDs etc is they are usable by anyone with a CD/DVD reader. If that is not true they the archive has failed to be archival on quite a fundamental level.

That's true and they are usable by anyone with a CD/DVD reader, except on those occasions when things go wrong. Incompatibility issues are rife in the computer world. You know that   .
« Last Edit: February 04, 2008, 07:55:18 am by Ray »
Logged

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #34 on: February 10, 2008, 05:06:56 am »

Just to get things in perspective, I thought I would add this piece of arcane information.

Having returned to Australia and whilst doing a bit of shopping I popped into one of my my favourite bricks & mortar stores for good bargains, Crazy Clark's, to buy a few more China made DVD wallets at A$2-3 each.

Whilst there, I notice DVD spindles of 50 discs, 2 for $35. These are recordable DVD-R, 16x with printable surfaces. They have a lifetime warranty.

Just out of curiosity I checked the current price of external hard drives at my favourite computer store in Brisbane. A 500GB Maxtor One Touch external hard drive is priced at $279.

How do these different media compare? On the one hand we have optical storage at 7.5 cents per gigabyte, guaranteed for a lifetime.

On the other hand we have 3.5" hard drive at 56 cents (Australian) per gigabyte guaranteed for 1 year.

Hmmm! I wonder which is the no-brainer?
Logged

Jonathan Wienke

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5829
    • http://visual-vacations.com/
Next step for archive storage
« Reply #35 on: February 10, 2008, 10:15:26 am »

Quote
Hmmm! I wonder which is the no-brainer?

Factor in the time you spend burning 100 DVDs and verifying the data, plus indexing and labeling the discs. I'm thinking that's at least 20 hours, probably more, depending on your burner speed, during which you and your computer are pretty much tied up and unable to do other things.  Then there's the hassle of digging through 100 DVDs trying to find one particular image. And copying all of those DVDs to new media before the DVD format becomes obsolete in a few years and DVD drive production ceases and the the drive you have quits working. There's also the factor of whether you are sufficiently dedicated to backing up your data to actually burn all of those DVDs and properly index and label them so that finding one particular image in the stack of discs is not an exercise in frustration. I figure my time is worth at least $15/hour, and spending 20+ hours burning DVDs vs a simple drag-and-drop file copy which can be done unattended once started is a very unattractive proposition. One might even say "no-brainer".
Logged

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #36 on: February 10, 2008, 11:09:15 am »

Quote
Factor in the time you spend burning 100 DVDs and verifying the data, plus indexing and labeling the discs. I'm thinking that's at least 20 hours, probably more, depending on your burner speed, during which you and your computer are pretty much tied up and unable to do other things.
[a href=\"index.php?act=findpost&pid=173713\"][{POST_SNAPBACK}][/a]

Jonathan,
The thread's about using optical media whilst travelling. You don't have to sit patiently waiting for your DVDs to burn. You can take a shower or have a meal. The only extra work really is organising folders to the right size to economically fill a disc. Once the folders are the right size it's just a matter of 'drag and drop' in the burning software and later a 30 second job of labelling each disc with a felt tipped pen. If you're prepared to waste a bit of disc space, you can save time by just recording the contents of each 4GB memory card.
Logged

Jonathan Wienke

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5829
    • http://visual-vacations.com/
Next step for archive storage
« Reply #37 on: February 10, 2008, 02:18:07 pm »

Quote
Jonathan,
The thread's about using optical media whilst travelling.

So what? For many types of shoots (weddings, concerts, charity events, school portraits, horse shows, etc) You're going to end up with several DVD's worth of images. I've covered all-day music festivals where I've shot more than 30GB of RAWs in a single day. Spending an hour or two at the end of a 14-hour day just to burn and verify DVD backups when I can simply drag-drop copy the files to a NAS and be done with it would be retarded.
« Last Edit: February 10, 2008, 02:20:05 pm by Jonathan Wienke »
Logged

Ray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 10385
Next step for archive storage
« Reply #38 on: February 10, 2008, 06:34:19 pm »

Quote
So what? For many types of shoots (weddings, concerts, charity events, school portraits, horse shows, etc) You're going to end up with several DVD's worth of images. I've covered all-day music festivals where I've shot more than 30GB of RAWs in a single day. Spending an hour or two at the end of a 14-hour day just to burn and verify DVD backups when I can simply drag-drop copy the files to a NAS and be done with it would be retarded.
[a href=\"index.php?act=findpost&pid=173774\"][{POST_SNAPBACK}][/a]

Circumstances vary. There's no single hard and fast rule for this sort of thing. Shooting 30GB of RAW in a day is probably unusual. Shooting 30GB of RAW day after day would be even more unusual. But if that's your life-style, then of course, forget about DVDs. If I were shooting weddings and horse shows, I don't think I'd be too concerned about archival issues anyway. Nor would I want the on-going chore of hard drive maintenance and periodic transfer to new drives in case a client wanted a particular image in 10 year's time.

The great thing about taking that initial trouble to record your RAW images on DVD, is that you can store them away and basically forget about them. No maintenance required. For the purpose of fast access to images, you're always going to try to have your images organised and available on hard drives, whether or not you've also got them on DVD.
Logged

Jonathan Wienke

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5829
    • http://visual-vacations.com/
Next step for archive storage
« Reply #39 on: February 10, 2008, 08:36:17 pm »

Quote
Shooting 30GB of RAW in a day is probably unusual. Shooting 30GB of RAW day after day would be even more unusual.

Not really; the OP specified that he's shooting 20GB in a typical day with his 1Ds-MkIII.
Logged
Pages: 1 [2] 3   Go Up