Pages: [1] 2   Go Down

Author Topic: An Archival Conundrum  (Read 14479 times)

John R Smith

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1357
  • Still crazy, after all these years
An Archival Conundrum
« on: September 30, 2010, 06:07:47 am »

Here is a topic for debate, speculation, and even (possibly) some hard facts. I don’t know the answer, but I would certainly appreciate some guidance.

You find that you will not be able to do any photography for the next 25 years. You wish to archive your current digital photograph collection onto some sort of media, lock it away in a cupboard in normal room conditions (avearage temperature and humidity), and come back to it in 25 years time with a reasonable expectation that you will be able to load all your old photos onto a new PC and start using Adobe CS19 or whatever version it is by then.

For the moment we will ignore the small matter of interfaces, and whether you would have a viable reader/socket/drive or whatever to pop your media back into. Given the current state of technology, which media would you choose for your long-term storage –

•   CDR or DVD-R (branded high-quality)
•   SD or CF cards
•   USB powered external hard drives
•   SSD drive
•   Magnetic tape cartridge

Or perhaps something else entirely?

John
« Last Edit: September 30, 2010, 07:00:01 am by John R Smith »
Logged
Hasselblad 500 C/M, SWC and CFV-39 DB
an

Christoph C. Feldhaim

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2509
  • There is no rule! No - wait ...
Re: An Archival Conundrum
« Reply #1 on: September 30, 2010, 06:35:24 am »

Magnetic tape.

martinreed22

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 50
Re: An Archival Conundrum
« Reply #2 on: September 30, 2010, 07:15:25 am »

I don't disagree with the response from Chris, mag tape has a good track record.

I would add that (as far as I know) no digital storage mechanism is guaranteed to last 25 years without attention. Mind you, by digital we are really covering a multitude of mechanisms: magnetic, charge (SD, CF), phase change (CDR). They all "rot" over time. Which means for safety you need to refresh the data every *mumble* years (*mumble* is a well known scientific term for more than a few, less than many).

Usual stuff: multiple copies, of course. Using multiple media types further reduces the risk. Use redundant encoding to allow for recovery from bit/block errors. Use multiple locations to avoid the perils of just one. Don't forget to clearly document what you have done and where, and share the info before a bus runs you over.

Just as important is whether your computer in 25 years time can provide power to, interface with and actually comprehend the data. Another argument for refreshing say every 5 years to a current technology.

Strangely enough, managed online storage may become sensible for long term storage as the providers have to refresh their systems regularly anyway. Just don't rely on one provider.

Paper tape or vellum would be safe mind you ... or you could even print them :)

Bottom line: I don't think anyone can safely predict more than 5 or so years forward.

cheers, martin
Logged

Christoph C. Feldhaim

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2509
  • There is no rule! No - wait ...
Re: An Archival Conundrum
« Reply #3 on: September 30, 2010, 07:21:51 am »

YUP.
Actually the raised question is one important of the many reasons why I decided to continue shooting film.

pegelli

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1664
    • http://pegelli.smugmug.com/
Re: An Archival Conundrum
« Reply #4 on: September 30, 2010, 07:35:07 am »

I think martinreed sums up the uncertainties and possible strategy (multiple technologies, copies, locations....) very well. Only thing to maybe also to consider is to add some hardware of today (again with some redundancy) in the room/boxes to increase the chances of doing something useful with the files after they're woken up.

I'm a bit puzzled by Christoph's response. I think film solves some durability problems, but is much harder to copy w/o quality loss, so a for instance a fire or flood can still wreak havoc without a contingency back-up, but maybe I'm overlooking something.
Logged
pieter, aka pegelli

mcbroomf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1534
    • Mike Broomfield
Re: An Archival Conundrum
« Reply #5 on: September 30, 2010, 08:05:04 am »

Yes I would do this with more than one media as well.

I would not use a USB external drive though, I'd just use a bare drive(s), and I'd also "archive" the physical computer that was used to write the files.

 
Logged

John R Smith

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1357
  • Still crazy, after all these years
Re: An Archival Conundrum
« Reply #6 on: September 30, 2010, 08:50:02 am »

Interesting.

I would have imagined that SSD might have been a preferred option - with no moving parts, read heads, or other bits to fail.

John
Logged
Hasselblad 500 C/M, SWC and CFV-39 DB
an

Christoph C. Feldhaim

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2509
  • There is no rule! No - wait ...
Re: An Archival Conundrum
« Reply #7 on: September 30, 2010, 09:12:08 am »

I'm a bit puzzled by Christoph's response. I think film solves some durability problems, but is much harder to copy w/o quality loss, so a for instance a fire or flood can still wreak havoc without a contingency back-up, but maybe I'm overlooking something.

B/W film, well processed and stored is said to live about 500 [!] years.
Actually I lost a bunch of negatives due to moisture and intelligent life forms.
However - the negatives and slides that survived are 20+ years old
and had simply been forgotten - I now am scanning these and it works.
Proper digital storage and turnover most likely is more reliable - but honestly:
Who will do that for the mentioned spans of time .... ?

John R Smith

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1357
  • Still crazy, after all these years
Re: An Archival Conundrum
« Reply #8 on: September 30, 2010, 09:17:53 am »

I have all the film negatives I have ever taken, since the age of 12. The oldest are now 52 years old, and they are still perfectly printable (6x9cm).

John
Logged
Hasselblad 500 C/M, SWC and CFV-39 DB
an

Christoph C. Feldhaim

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2509
  • There is no rule! No - wait ...
Re: An Archival Conundrum
« Reply #9 on: September 30, 2010, 10:14:04 am »

I have all the film negatives I have ever taken, since the age of 12. The oldest are now 52 years old, and they are still perfectly printable (6x9cm).

John

John, shouldn't this somewhat answer your question?

martinreed22

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 50
Re: An Archival Conundrum
« Reply #10 on: September 30, 2010, 10:37:28 am »

Interesting.

I would have imagined that SSD might have been a preferred option - with no moving parts, read heads, or other bits to fail.

John
Very fair question. Trouble is that nobody knows as the tech is too new. Some might say the same about inkjet prints :)

SSD is a form of flash memory. It holds tiny charges of electricity in tiny structures to encode your data. If nothing else, cosmic rays can degrade such structures. Bizzarely, solid state devices do actually wear out with use (although much less relevant in an archival role).

Like other media, refreshing regularly is a Good Thing. This is because a refresh should reduce the slow build up of correctable errors in the data.

SSD still has the issues about power and data connectivity, and data formats.

Magneto-optic has one of the better track records for archival data storage media.

Hard to beat the good ol' Second Law of Thermodynamics without putting some effort (energy) back in regularly.

cheers, martin
Logged

John R Smith

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1357
  • Still crazy, after all these years
Re: An Archival Conundrum
« Reply #11 on: September 30, 2010, 10:44:37 am »

Magneto-optical looks like it is a dead technology, sadly.
Logged
Hasselblad 500 C/M, SWC and CFV-39 DB
an

martinreed22

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 50
Re: An Archival Conundrum
« Reply #12 on: September 30, 2010, 11:57:00 am »

Magneto-optical looks like it is a dead technology, sadly.
I think the purveyors would like to think of it as "niche" rather then dead. I see the media is still available ex-stock from multiple sources. UDO is another esoteric option aimed at archive.

I would expect all media we have today to be obsolete/different in 25 years time. Planning on that assumption is, I think we've agreed, the key.

cheers, martin
Logged

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: An Archival Conundrum
« Reply #13 on: October 04, 2010, 10:12:47 pm »

Archival!!! Archiving!!

Cost and availability are the two daemons in Arching.

Well, there isn't anything that is certain. Everything we do is a compromise to minimise un-certainty. One thing about backing up is retrieving the data again, will the interface be available when I need it? Using SSD disks for archiving is great if cost isn't a factor, but try to archive 20+TB of data to SSD with redundancy, unless you are Bill Gates it ain't going to happen. So cheaper alternatives will have to be found.

Question, how quick do you want to retrieve your data, that is already archived? The faster you want it the more expensive it becomes.

Currently, I can live with multiple hard drive back-ups using hard drive cradles, this offers a fast and inexpensive way of backing up and retrieving my images fast, especially with Digital Assset Management software, with disks labeled and stored appropriately. This also offers me a thumbnail image to view the images, again this catalog also needs to be backed up multiple times with different time-stamps, as if one version of the catalog fails I can quickly roll back to the previous version.

Using multiple hard drives, say 3 2TB disks, use three different brand disks, Seagate, WD and Hitachi (in no particular order), so to avoid getting a bad batch of disks where suddenly all drives from a specific set fails you, oops!  ??? Someone did calculate the failure rate of 1, 2 and many disks.

Having an alternative backup solution to the above is a good choice, this is where your magnetic tape comes in-handy or similar devices. This can very well be a slower solution, but the fact that its a different media, gives it the opportunity to perhaps sustain different threads that the first back couldn't.

I have mentioned it above, Digital Asset Management, but i will do so again, if you can't find it, it isn't worth the space it takes up!!! So make sure you have a catalog or some way of finding what you or someone else is looking for some time later in the future. Imagine that person not know whats there, they still have to know how to find things, so write a little note about how you have structured your files.

This could be your retirement fund!!!

Henrik
Logged

bradleygibson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 828
    • http://GibsonPhotographic.com
Re: An Archival Conundrum
« Reply #14 on: October 05, 2010, 12:12:19 am »

Magnetic media is a risky strategy for a quarter-century--that's tape or disk.  The oxide coating can flake off, alignment differences from the writing drive and the reading drive, and/or the charge can fade.  Flash hasn't been around long enough to really understand how well the data (trapped electrons) will age.

Since you specifically said you do not want to maintain/periodically refresh the media, this would likely be my last choice.

My suggestion would be to get pressed optical media created in duplicate or triplicate.  This is not the media that you and I can burn at home, it is the media whose binary data is made of physical pits, pressed at a duplication house.  Obviously more expensive than burn media.

Presumably one will be able to find an optical disc reader in 25 years time, much the same way one can locate a 5 1/4" floppy drive from 1985 today on eBay or some such.  The challenge will be plugging it in--you can be sure SATA will not be an easy interface to connect to. :)

Solvable problem, but be prepared for a bit of work come 2035...  :)
« Last Edit: October 05, 2010, 06:04:28 pm by bradleygibson »
Logged
-Brad
 [url=http://GibsonPhotographic.com

John R Smith

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1357
  • Still crazy, after all these years
Re: An Archival Conundrum
« Reply #15 on: October 05, 2010, 04:23:59 am »

Thanks for your input, Henrik and Brad.

I am coming to the conclusion that as things stand, there is no real solution for a 25-year digital archive which will not be managed and refreshed at intervals. The catalogue is not a problem in this regard, as it is on paper and will stand the test of time. So there is no need for a DAM system.

For the present, I have decided that I will adopt a dynamic storage system for my digital photographs, where the collection is copied in its entirety at regular intervals to multiple hard drives, which will themselves be replaced at 3-year intervals. Which is all very fine, as long as I am around to do it of course, or I don't lose interest.

John
Logged
Hasselblad 500 C/M, SWC and CFV-39 DB
an

NikoJorj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1082
    • http://nikojorj.free.fr/
Re: An Archival Conundrum
« Reply #16 on: October 05, 2010, 04:41:14 am »

My suggestion would be to get pressed optical media created in duplicate or triplicate.  This is not the media that you and I can burn at home, it is the media whose binary data is made of physical pits, pressed at a duplication house.  Obviously more expensive than burn media.
Interesting! Do you got an order of magnitude of cost?

For the present, I have decided that I will adopt a dynamic storage system for my digital photographs, where the collection is copied in its entirety at regular intervals to multiple hard drives, which will themselves be replaced at 3-year intervals. Which is all very fine, as long as I am around to do it of course, or I don't lose interest.
And if one loses interest or isn't there anymore to do it, then what may count more are finished images, rather than digital originals ; and for that, prints you can make now are much more hassle-free than a digital archive.
For me, the main interest of a digital archive is to be able to re-process my images.
Logged
Nicolas from Grenoble
A small gallery

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: An Archival Conundrum
« Reply #17 on: October 05, 2010, 05:47:40 am »

Hi again John,

If you are looking at a 3 year cycle between updating your hard drives, I would definitely choose, quality drives, such as enterprise class disks with 5 years warranty and I would definitely replace early in the 4th year. This may sound silly but never the less. I think it is a good rule to follow.

Loosing interest, well that happens, just make sure your archive is one that someone else with interest can work out how to use. :-)

Henrik
Logged

Rick_Allen

  • Full Member
  • ***
  • Offline Offline
  • Posts: 183
    • http://www.rapdigital.net
Re: An Archival Conundrum
« Reply #18 on: October 05, 2010, 08:19:56 am »

Hard drives aren't archival and it doesn't matter if they are enterprise or not. When the Seagate firmware disaster hit me last year or the year before, luckily at the time I was backing everything up to a mirror set of western digital drives, I bit the bullet and bought a LTO4 tape solution. I now save my (I'm a digital tech for photographers) digital neg's ie. the whole unprocessed session to two tapes and one goes home with me at the end of each day. I then have my working sessions on a raid 5 plus spare connected to my main workstation. On the Raid5 I have a WIP folder that is undelivered working files that I do incremental backups of whenever I'm on the phone or at the coffee machine (often) this folder is never more than 2TB so it gets backed up to 2x 2tb bare drives (usually Western digital and Samsung but sometimes seagate) these drives are just cheapest that I can find and I have 6 sets that I rotate through as needed (these are just a third safety net and arent kept) Finally once the job's complete it's moved to a 2bArchived folder and that is automatically copied to the tape server drive raid1+0 and is burnt to a set of tapes.

just my 2c
Logged
Rick Allen
[url=http://www.rapdigital.ne

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: An Archival Conundrum
« Reply #19 on: October 05, 2010, 11:16:50 am »

Rick,

Good advise Mate, you probably haven't read the whole thread here, but it echo's what you are recommending.
One of the points above was not to use the same brand of disks as a backup pair to avoid exactly what you have experienced, e.g. the Seagate firmware disaster.

Having rotating backups is another great idea and provides that extra protection, should you by accident have overwritten your disks or otherwise damaged the backup set.

But just to go one over you, then use RAID 6 instead of RAID 5 where you have more redundant disks...and so on and so....

where do we draw the limits to protect our assets, you wisely take one of your backup's off site. but had you lived in Indonesia/Indian Ocean during the Tsunami, then that would have helped you diddle squad, only could only hope there was a fire/water proof safe to keep your backup's in.

having just lost 17GB to a failed/corrupt SD-card on a shoot, thinking I had set my camera up to write to two cards simultaneously, only to find that i was not doing so. Even at that level its important to have some sort of redundancy. (going to re-shoot! :-( )

One just can't be too careful these days with out digital assets, they are actually very fragile.

cheers

Henrik
Logged
Pages: [1] 2   Go Up