Pages: [1] 2   Go Down

Author Topic: Does anyone use LTO for storage or archiving image files?  (Read 26481 times)

MattBeardsley

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 58
    • http://mattbeardsleyphoto.com
Does anyone use LTO for storage or archiving image files?
« on: December 07, 2012, 01:01:52 pm »

So "tape" is kind of old-school technology, but LTO (linear tape open) is a technology that is alive and well in high-end data storage.  I came across a huge LTO system while shooting an editorial piece at Stanford University and started researching the technology.  I have a single Mac Pro in my studio with two 4-bay RAID boxes connected via eSATA.  I want to find a safe way to offload old RAW files and video clips, to archive them on the shelf for safe-keeping... also send a duplicate archive off to somewhere safe.  I think LTO might be the best, most cost-efective bet in 2012. Tapes are 1.5 TB and cost around $40 each, are archivally-safe for at least 30 years.  Does anyone else do this?

Here's a blog entry I wrote discussing the workflow in more depth.  I would love feedback from fellow pros about data storage!

http://photoartsmonthly.com/blog/2012/12/01/bru-desktop-lto5/

~Matt
Logged
Matt Beardsley, Oakland, CA
The Artist:  http://mattbeardsleyphoto.com
The Nerd:  http://photoartsmonthly.com

Phil Indeblanc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2017
Re: Does anyone use LTO for storage or archiving image files?
« Reply #1 on: December 07, 2012, 08:42:59 pm »

30 years is not something I would want to rely on for "body of work", but it would work just fine for long term clients archives.
I remember Verbatim was a popular tape and media sollution.  I have used in the past and did have a failed backup. but likley due to software.

HDD were going so low for a while. but looks like roughly .10cents or less perGB is still the norm.  If they were low enough, you can back it up and put it in a safe for future spin up.

I was thinking at some point BluRay or other standard would be able to do 1-2TB for the 100year archive? Maybe Laserdisc size would make a come back :-)??
Logged
If you buy a camera, you're a photographer...

alain

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 465
Re: Does anyone use LTO for storage or archiving image files?
« Reply #2 on: December 08, 2012, 02:18:21 pm »

Hi

When comparing to 3TB drives the break even point (the extra expensive tape drive is paid for) is after 75TB backup storage.

If you want to be save and you make three backups for everything and then the break even is after 25TB.  Although those people also want to have two tape drives to be sure ;-)

Logged

MattBeardsley

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 58
    • http://mattbeardsleyphoto.com
Re: Does anyone use LTO for storage or archiving image files?
« Reply #3 on: December 10, 2012, 04:02:08 pm »

It's true, the ever decreasing cost of hard drives is a compelling alternative... maybe 8 or 16-bay RAIDs daisy-chained via Thunderbolt is a cost-effective alternative.  By today's pricing, a 3 TB hard drive is around $150, making 75 TB roughly $3700 and, of course, there are many great enclosure and infrastructure options which drive up the cost to varying degrees (I'm curious about Nestor's new 16-bay Thunderbolt RAID boxes).

Once an LTO drive is in place (currently, they cost at least $1500 US) 1.5 TB LTO-5 tapes are around $40.  So 75 TB of storage, with no other backup, might be –as a very rough estimate– $3000 ($1500 for a drive, $2000 in tapes, $500 in software and SAS PCIe card).

You're totally right that the cost benefit only comes into play once the terabytes really start stacking up.  It's interesting to me to weigh other aspects too, like easy access vs. safer long-term storage

Thoughts?
Logged
Matt Beardsley, Oakland, CA
The Artist:  http://mattbeardsleyphoto.com
The Nerd:  http://photoartsmonthly.com

Sareesh Sudhakaran

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 546
    • The Indie Farm
Re: Does anyone use LTO for storage or archiving image files?
« Reply #4 on: December 10, 2012, 09:41:09 pm »

It doesn't matter if LTO tapes work 30 years from now if the format is extinct. It looks like replication is an important strategy to consider in any long term backup solution, plus cloud backup to the 'most trending' CDN.

By the way, the cost analysis might need reconsideration. The cost of media is immaterial to the archival period.
Logged
Get the Free Comprehensive Guide to Rigging ANY Camera - one guide to rig them all - DSLRs to the Arri Alexa.

chrismurphy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 77
Re: Does anyone use LTO for storage or archiving image files?
« Reply #5 on: December 17, 2012, 06:41:28 pm »

The comparisons are lopsided, because they only account for price based on the most inexpensive class of hard drives: high density consumer SATA. They also have by far the highest failure rates (full disk failure, unrecoverable read errors, and corruption).

The UER for consumer SATA is one sector in ~11TiB of data read. For a 3TB drive, that means you should be able to read that drive fully 4 times before encountering an unrecoverable read error, but you're not given a choice when this will happen.

The UER for LTO-5 is 1 bit in 11PiB, or ~11368 TB. So it's just a totally different category of media in terms of reliability. Expensive enterprise SAS drives, which have a UER of 1 sector lost in 1.1PiB read, are still an order of magnitude more likely to encounter an error than LTO. So if you start doing your comparisons based on how many enterprise disks you'd need to have, LTO gets cheaper faster than is being considered so far in the thread.

How Many Archive Copies Do You Really Need?

Consider the 2nd table, hours to reach hard error rate at (max) sustained data rate. A RAID 5 (striped, distributed parity, redundancy with one disk failure tolerated) you can actually have higher chance of array collapse than an equivalently sized RAID 0 (no redundancy).

And further the shelf life of a hard drive is not longer than it's spinning life. It ages, for different reasons, whether it's spinning or not. The life expectancy of hard drives is 1-7 years. Also one of the major points of LTO is that the most recent version of tape still can be read/written to, and two versions back can still be read. So an LTO-5 drive today can read LTO-3 tapes. And the expectation is LTO-8 drives will read today's LTO-6 tapes. So yeah in 15 years you won't have an LTO-5 drive, but you could easily have an LTO-7 drive to read those LTO-5 tapes and migrate them.

Or, the other extreme, but a whole lot easier, is to regress back to the days of artists's printing plates, and break them once you've printed your edition. In this case, make your prints, and destroy the Raw files and all derivatives. Digital photo archiving strategy solved.
Logged

Phil Indeblanc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2017
Re: Does anyone use LTO for storage or archiving image files?
« Reply #6 on: December 17, 2012, 09:58:18 pm »

I'm working on setting up a cloud at an off site location. I think I would rather do that than send files to any company that is providing that service.
Once you have a NAS/server with large enough storage, with a port between the two mach9ines, and some auto backup software that can login and copy to a "cloud" computer. Done. You have yourself a private cloud.
Logged
If you buy a camera, you're a photographer...

chrismurphy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 77
Re: Does anyone use LTO for storage or archiving image files?
« Reply #7 on: December 17, 2012, 10:27:16 pm »

The software is certainly there, and there are all sorts of ways to automate this. NAS 4 Free has schedulable snapshots that can be ZFS sent/received via rsync to another NAS 4 Free. If you had two on-site they even have fail over for high availability where the 2nd one takes over, sorta like a cluster (but without live replication). There's lsyncd which also uses rsync, although this is more of a home grown solution, no GUI. And there's also implementing an actual cluster with asynchronous geo replication off-site.

The big problem with cloud stuff are the pipes. For even a dinky library of only 50GB is around 12 days of uploading with U.S. average upload bandwidth. And that's at full saturation, which you can't do if you want usable internet service. Naturally with your own cloud and off-site, you can seed that storage locally, then move it. But if you have a 4TB library, just Raws, and a conservative 15% annual growth rate, that's 50GB per month on average. Apply QoS so that you can use other internet services without it feeling sluggish, and those 12 days will turn into nearly a full month of continuous  uploading.

So it greatly depends on the bandwidth, the ability to set up a decent router with QoS, and not busting the monthly cap if this is residential service.
Logged

alain

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 465
Re: Does anyone use LTO for storage or archiving image files?
« Reply #8 on: December 18, 2012, 06:49:26 am »

The comparisons are lopsided, because they only account for price based on the most inexpensive class of hard drives: high density consumer SATA. They also have by far the highest failure rates (full disk failure, unrecoverable read errors, and corruption).

The UER for consumer SATA is one sector in ~11TiB of data read. For a 3TB drive, that means you should be able to read that drive fully 4 times before encountering an unrecoverable read error, but you're not given a choice when this will happen.

The UER for LTO-5 is 1 bit in 11PiB, or ~11368 TB. So it's just a totally different category of media in terms of reliability. Expensive enterprise SAS drives, which have a UER of 1 sector lost in 1.1PiB read, are still an order of magnitude more likely to encounter an error than LTO. So if you start doing your comparisons based on how many enterprise disks you'd need to have, LTO gets cheaper faster than is being considered so far in the thread.

How Many Archive Copies Do You Really Need?

Consider the 2nd table, hours to reach hard error rate at (max) sustained data rate. A RAID 5 (striped, distributed parity, redundancy with one disk failure tolerated) you can actually have higher chance of array collapse than an equivalently sized RAID 0 (no redundancy).

And further the shelf life of a hard drive is not longer than it's spinning life. It ages, for different reasons, whether it's spinning or not. The life expectancy of hard drives is 1-7 years. Also one of the major points of LTO is that the most recent version of tape still can be read/written to, and two versions back can still be read. So an LTO-5 drive today can read LTO-3 tapes. And the expectation is LTO-8 drives will read today's LTO-6 tapes. So yeah in 15 years you won't have an LTO-5 drive, but you could easily have an LTO-7 drive to read those LTO-5 tapes and migrate them.

Or, the other extreme, but a whole lot easier, is to regress back to the days of artists's printing plates, and break them once you've printed your edition. In this case, make your prints, and destroy the Raw files and all derivatives. Digital photo archiving strategy solved.
Hi Chris

You have a point and tape is recommended for larger setup.  But I wouldn't start with a tape setup without having two operational tape drives available (the second one can be with a trusted other, at least at a different location) and testing backup restore with both tape drives (and software).

Considering the consumer drives, yes they will fail, but everything can fail. 

To avoid external hazards (fire,burglar,flood,...) you need backups on another site, which means that you need two sets of transportable backup media.
It's also important that one recent backup set is kept from the network (and power supply), can be done with those two sets if they rotate enough.

Locally it's best to have a on-line or near line storage and a raidZ2 system would fit in nicely (snapshots to avoid user error, scrubbing to find problems early, RAIDZ2 to get two parity disks).

--> I don't consider the consumer drive the major problem here.







Logged

Sareesh Sudhakaran

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 546
    • The Indie Farm
Re: Does anyone use LTO for storage or archiving image files?
« Reply #9 on: December 18, 2012, 09:28:22 am »


The UER for consumer SATA is one sector in ~11TiB of data read. For a 3TB drive, that means you should be able to read that drive fully 4 times before encountering an unrecoverable read error, but you're not given a choice when this will happen.

The UER for LTO-5 is 1 bit in 11PiB, or ~11368 TB. So it's just a totally different category of media in terms of reliability. Expensive enterprise SAS drives, which have a UER of 1 sector lost in 1.1PiB read, are still an order of magnitude more likely to encounter an error than LTO. So if you start doing your comparisons based on how many enterprise disks you'd need to have, LTO gets cheaper faster than is being considered so far in the thread.


Can't fight theory! But practice dictates otherwise. In the real world, hard disks don't do as badly as anticipated, especially because drives are always improving. All said and done, LTO does not guarantee anything, no matter what the underlying technology provides. If the statistical scale in theory were relevant (I don't deny its truth), cloud servers of the kind Akamai, Google, Amazon, etc. would be running on LTO, but that's not the case. And I haven't seen a single businessman (or an engineer for that matter) complaining about the lack of LTO systems for their data warehouses. I could be wrong, but to the average consumer, if LTO has to be more than a flash in the pan, it needs to put a guarantee in writing.

Also, one has to seriously consider the time spent to read and write LTO tapes. The more data you have, the more time you are going to spend hunting for something.

If I really have to back up something that was super-precious, I'd choose film over any other solution. Even if film becomes extinct, one can still read what one has saved with just the basic knowledge of the chemistry required, like a family recipe.

30 years for LTO? No, thank you.
Logged
Get the Free Comprehensive Guide to Rigging ANY Camera - one guide to rig them all - DSLRs to the Arri Alexa.

chrismurphy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 77
Re: Does anyone use LTO for storage or archiving image files?
« Reply #10 on: December 18, 2012, 01:08:48 pm »

But I wouldn't start with a tape setup without having two operational tape drives available

Why? Seems like overkill unless you need the capacity to do simultaneous backups to tape.

Quote
Considering the consumer drives, yes they will fail, but everything can fail. 

This statement ignores non-equal likelihood for failure between consumer drives and other drives. It's not merely double the failure rate, it's orders of magnitude higher.

Quote
To avoid external hazards (fire,burglar,flood,...) you need backups on another site, which means that you need two sets of transportable backup media.

OK. Compare the mass and volume of 2 LTO-5 tapes (3-5TB of storage) to that of a single SATA disk, the HDD is significantly bigger and heavier. Physical removal of HDDs compared to tapes is also higher risk.

Quote
Locally it's best to have a on-line or near line storage and a raidZ2 system would fit in nicely (snapshots to avoid user error, scrubbing to find problems early, RAIDZ2 to get two parity disks).

There are a number of ways to clobber the problem, RAIDZ2 is one way. But RAIDZ is non-growable, so you either have to build a 2nd non-growable array, and put both arrays into a pool, or you have to destroy and rebuild the array from scratch. Most photographers feel like growing their storage 1-2 disks at a time.

Quote
I don't consider the consumer drive the major problem here.

The context is archiving image files, and using LTO vs drives for that purpose. Drives are distinctly not archival media.
Logged

chrismurphy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 77
Re: Does anyone use LTO for storage or archiving image files?
« Reply #11 on: December 18, 2012, 02:52:56 pm »

In the real world, hard disks don't do as badly as anticipated, especially because drives are always improving.

There are numerous HDD storage research papers, based on drives used in the real world, clearly stating that consumer HDDs do worse than anticipated.

HDDs are improving in aerial density. The unrecoverable error rate is the same, which combined with higher density means you're more likely to encounter unrecoverable errors. The UER I quote is a manufacturer spec, there are other vectors for data loss the manufacturers don't put in their spec.

Quote
All said and done, LTO does not guarantee anything, no matter what the underlying technology provides. If the statistical scale in theory were relevant (I don't deny its truth), cloud servers of the kind Akamai, Google, Amazon, etc. would be running on LTO, but that's not the case.

Google and Amazon use LTO. There was a highly publicized Gmail failure a while back that was recovered from, by tape. Amazon provides storage services that are backed up to LTO, and even has one service only using LTO. Businesses that care even a little about data, are using tape as backup or at least for archive.

Quote
And I haven't seen a single businessman (or an engineer for that matter) complaining about the lack of LTO systems for their data warehouses. I could be wrong, but to the average consumer, if LTO has to be more than a flash in the pan, it needs to put a guarantee in writing.

Enterprise companies with enterprise storage tend to have autotape changers in the tens to hundreds of thousands of dollars so I don't expect they'd complain about a lack of LTO.

I don't know what sort of guarantee in writing you think you need. You either need one now, or you don't. Such things are very expensive. But I guarantee you, that if you contract with a company whose business it is to supply written guarantees of data integrity and reliability, they will use LTO to backup and archive your data whether you like it or not because that's the way it's done.


Quote
Also, one has to seriously consider the time spent to read and write LTO tapes. The more data you have, the more time you are going to spend hunting for something.

This is a backup and archive media, with a very high penalty for random access which is why its not used for that. It's used for offline, not online storage.

Whether HDDs or tapes, you still have to organize them. LTFS discloses what tape files are stored on, unlike conventional file systems like NTFS or HFSJ.


Quote
If I really have to back up something that was super-precious, I'd choose film over any other solution. Even if film becomes extinct, one can still read what one has saved with just the basic knowledge of the chemistry required, like a family recipe.

It's an idea. For small volume of images, and if metadata or dynamic range aren't important, it might make sense.

Quote
30 years for LTO? No, thank you.

It doesn't work for every situation. Maybe it doesn't work for most situations. But you say 30 years? What's your alternative? By definition that's not going to be either HDD or SSD. Going through another round of editorial work to find the 10% worthy of being imaged to film?

It's a valid question, how to do this. HDDs do not cover it. They are in no way an archive medium. And the overwhelming majority of photographers recycle their hard drives used for backups, guaranteeing that as corruption occurs, it finds its way into all backups.
Logged

alain

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 465
Re: Does anyone use LTO for storage or archiving image files?
« Reply #12 on: December 18, 2012, 04:57:08 pm »

...
There are a number of ways to clobber the problem, RAIDZ2 is one way. But RAIDZ is non-growable, so you either have to build a 2nd non-growable array, and put both arrays into a pool, or you have to destroy and rebuild the array from scratch. Most photographers feel like growing their storage 1-2 disks at a time.
...
The context is archiving image files, and using LTO vs drives for that purpose. Drives are distinctly not archival media.
Photographers that like to grow their storage 1 or 2 disk at a time, won't invest $2000 in a tape drive.

I have no problems with LTO, but I don't think it's for most photographers.  Maybe Blu-ray is better for archiving (not backup).

I would be very happy if most use a "regular" backup system that is verified by them.
Logged

Mike Boden

  • Full Member
  • ***
  • Offline Offline
  • Posts: 132
    • http://www.mikeboden.com
Re: Does anyone use LTO for storage or archiving image files?
« Reply #13 on: December 18, 2012, 08:08:53 pm »

Looks like I may be one of the only ones around here using LTO as a backup medium, which I've been using for about eight years now. In fact, I'm still using LTO2 technology and haven't upgraded yet. Anyway, my use of LTO isn't exclusive. I have two copies of all my files on hard drives, one of which is a RAID 5 array. Furthermore, I have two copies of the LTO archives, one of which is stored off-site.

So my workflow is organized such that the RAID 5 array is the backup to the original copies stored on local hard drives directly attached to my working computer. This RAID 5 array is installed in a separate and dedicated server. The LTO drive is installed in this server and acts as a backup/archive of the files on the RAID 5 array. So, if something happens to the directly attached hard drives, I can connect to the server to retrieve a backed up copy of any files. If something was to happen to the RAID 5 array, I can restore from LTO archive. If my office was to burn down or flood, then I can ultimately restore from the LTO archives stored off-site.

As far as cost is concerned, the price is inconsequential when it compares to losing files. Unfortunately, this happened to me early on before I put this workflow together and is the primary reason I went down this path.

I will also add that I use Retrospect for my backup software to/from the LTO tapes, and the data is stored on these tapes uncompressed and non-encrypted.
Logged
Mike Boden - www.mikeboden.com
Instagram

Richard Man

  • Full Member
  • ***
  • Offline Offline
  • Posts: 137
    • Richard Man Photography
Re: Does anyone use LTO for storage or archiving image files?
« Reply #14 on: December 18, 2012, 09:11:36 pm »

Very timely discussion, as I am looking at the issues right now. I have a couple 2 TB drives attached to the backup server, but one of them just died this weekend. The simple solution is get a drive dock and rotate two or three 3TB drives, but my digital images is going to hit up to 3TB by the end of 2013 (60% digital but 40% large scans of 617 and XPan and 120). With tapes, it seems like easier to store and less expensive to buy multiple set and rotate them.
Logged
// richard
[url=http://richardmanphoto.c

chrismurphy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 77
Re: Does anyone use LTO for storage or archiving image files?
« Reply #15 on: December 18, 2012, 10:30:59 pm »

Photographers that like to grow their storage 1 or 2 disk at a time, won't invest $2000 in a tape drive.

I don't see them as related. Logical volume managers, and RAID 1+linear can easily grow storage 1-2 drives at a time, and are scalable for very large storage. It's just that this is a bit more difficult with ZFS, which is not to say it should be avoided, it's just going to take some additional planning.


I have no problems with LTO, but I don't think it's for most photographers.  Maybe Blu-ray is better for archiving (not backup).

I think it's completely unworkable, just on a technical level.

Blu Ray and DVD are radically different media when commercially stamped vs burned with consumer recorders. They're not at all the same process. Commercial ROM disks have physical pits stamped into the receiving media. Consumer recordable optical media use dyes, and hence subject to dye fading. A test of 25 discs of various brands of archival recordable optical media in 2009 resulted in failure within 48 hours of accelerated testing for all dye based disks. Only one disk, which was not dye based, did not fail the test which was the Millenniata disks.

The m-disk requires specialized recording hardware, but can be read by any DVD reader. They're also less than 5GB per disk, while costing ~$2.60 each. So 1TB is over 200 disks at $500, not including the writer.

Practically, Blu Ray is more expensive per GB than simply buying good nearline hard drives, which will also be incredibly fast in comparison, and are reusable. Blu Ray once write media is slow, but perhaps tolerably slow for some people. Blu Ray rewritable media is untenable as an archive strategy in my view, it's way too slow.

I would be very happy if most use a "regular" backup system that is verified by them.

Sure two copies on most any media works pretty well and should be easy. People get lazy if this isn't really well automated, because overwhelmingly the penalty for not backing up is consequence free. Until it's not, and then the penalty seems really disproportionate.

It's sorta like "big sky theory" in aviation, except your chances are still probably better in the airplane. Again, until it's not. Going from good to bad in both realms is pretty abrupt.

Looks like I may be one of the only ones around here using LTO as a backup medium

I think it's a great story to share.

Also a good point is that LTO-5 isn't the only option. While LTO-6 is already out, some photographers may be better suited with an LTO-4 solution, although I haven't done a price difference comparison to see how much of an up front cost savings there is.

Last I'll mention that anyone considering tape should also consider software that supports LTFS so that you're not stuck in a situation where you need proprietary software to regain access to your data. It's licensed under the GPL, is supported by IBM, Quantum, Oracle, HP and others. And is cross platform.
Logged

Sareesh Sudhakaran

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 546
    • The Indie Farm
Re: Does anyone use LTO for storage or archiving image files?
« Reply #16 on: December 19, 2012, 11:28:23 am »

I don't know what sort of guarantee in writing you think you need. You either need one now, or you don't. Such things are very expensive. But I guarantee you, that if you contract with a company whose business it is to supply written guarantees of data integrity and reliability, they will use LTO to backup and archive your data whether you like it or not because that's the way it's done.

That's my point, too. How will I know that I need it now when I don't know it will work years from now? And by the way, businesses follow what is called 'best practices', which is nothing but 'pop practices' - whatever the accountants, insurers and lawyers think will protect them during that fiscal year. Is there a company that guarantees data longevity, come hell or high water? On what basis will they compensate if they fail to keep their promise?

Quote
This is a backup and archive media, with a very high penalty for random access which is why its not used for that. It's used for offline, not online storage.

But time is money nonetheless, and it needs to be quantified and monetized.

Quote
Whether HDDs or tapes, you still have to organize them. LTFS discloses what tape files are stored on, unlike conventional file systems like NTFS or HFSJ.

Yeah, but what about the guy reading the tape? He still needs to be organized! If he can manage that, he won't sweat with drives. I have heard cool things about LTFS, but let's compare it to ZFS, EXT3, or the like. In any case, I grant LTO's speed and cool file system, but they're not significantly ahead to justify the price differential. And one has to always be on one's toes with regards to disruptive technology. LTO does not guarantee its own survival!

Quote
What's your alternative? By definition that's not going to be either HDD or SSD. Going through another round of editorial work to find the 10% worthy of being imaged to film?

Why not? They have been around longer than LTO! Jokes aside, I find it's more convenient to reproduce data to the latest fab, and not rely on anybody. I can offer better odds to this strategy over any other, including film. After all, it is what life itself follows.

I want to make it clear that I do not deny LTO's general utility, only the specific belief about its longevity or future-worthiness. I'll give cave paintings better odds.
Logged
Get the Free Comprehensive Guide to Rigging ANY Camera - one guide to rig them all - DSLRs to the Arri Alexa.

chrismurphy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 77
Re: Does anyone use LTO for storage or archiving image files?
« Reply #17 on: December 19, 2012, 04:43:15 pm »

That's my point, too. How will I know that I need it now when I don't know it will work years from now?

LTO is designed with future support specifically in mind because it was designed to solve this particular archive problem. HDDs are not at all designed with this in mind.

Quote
And by the way, businesses follow what is called 'best practices', which is nothing but 'pop practices' - whatever the accountants, insurers and lawyers think will protect them during that fiscal year. Is there a company that guarantees data longevity, come hell or high water? On what basis will they compensate if they fail to keep their promise?

Yes. Insurance payout.

This category of enterprise storage is out of scope, it's very expensive. An LTO-5 drive is cheap in comparison to that. You can consider it a kind of insurance compared to using high density consumer hard drives as an archive media.

Quote
But time is money nonetheless, and it needs to be quantified and monetized.

You're ascribing a meaningful cost to using the hardware in a manner for which it's not designed. If you have a case where you need to be sifting through tape for one file to restore, you either have a bad design, or you had an edge case. If it's an edge case it does not need to be quantified and monetized.

Quote
Yeah, but what about the guy reading the tape? He still needs to be organized! If he can manage that, he won't sweat with drives.

Right except a bad hot swap, bad shutdown, or bad mount, resulting in a corrupted file system. You cannot make a hard drive fool proof read only. It can write garbage.

Quote
I have heard cool things about LTFS, but let's compare it to ZFS, EXT3, or the like.

Why? They aren't the same thing. The use cases are totally different.

Quote
In any case, I grant LTO's speed and cool file system, but they're not significantly ahead to justify the price differential.

Not significantly ahead of what? HDDs which aren't even archival? In the context of this thread, which is an archival context, HDD's simply do not qualify. So yes LTO is significanty ahead in this area. You can't even get a hard drive with a UER as low as LTO is. And you don't get the workflow with HDD that is implicit in LTO.

Now, if you have a small library, then indeed LTO may not make as much cost sense. But there is still a need to reduce the likelihood of data loss (incuding corruption) and that starts with not using high density consumer SATA drives if you care about your data.

Quote
And one has to always be on one's toes with regards to disruptive technology. LTO does not guarantee its own survival!

It's an open standard with an open source file system. What you're saying, is like saying Linux does not guarantee its own survival therefore you shouldn't use it for anything. It's ridiculous.

You're arguing against LTO because of an unknown that also applies to hard drives, yet you advocate hard drives over LTO. It's weird advice. And in fact wrong because LTO is designed by the LTO consortium to survive as an archiving platform. SNIA is not doing that for hard drives. They are thinking in 5 year terms, whereas LTO is thinking in 25-50 year terms.

Quote
Why not? They have been around longer than LTO! Jokes aside, I find it's more convenient to reproduce data to the latest fab, and not rely on anybody. I can offer better odds to this strategy over any other, including film. After all, it is what life itself follows.

If you can offer better odds over any other, you should produce your process. LTO has done this.


Quote
I want to make it clear that I do not deny LTO's general utility, only the specific belief about its longevity or future-worthiness.

First you said if the error rates were relevant then Google and Amazon would use LTO. Second you said they don't use LTO. Yet they do use LTO. But then you change your own metric for what makes such error rates relevant, and still conclude that LTO isn't actually meaningfully better than drives. This despite the fact drives aren't designed with this in mind, but LTO is. It's pretty weird.

Quote
I'll give cave paintings better odds.

If you take someone's digital photo and encoded it as a cave painting, as soon as one fleck of paint came off the whole encoding would be destroyed and the photo lost to time.

Uncompressed files are slightly more tolerant of corruption than compressed files, but digital photos need to be fairly exactly reproduced bit for bit in order to survive, unlike cave paintings. Your analogy just doesn't work. You're better off saying you'll give pigment inkjet prints on good media better odds, and totally forget the digital original.
Logged

FredBGG

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1630
Re: Does anyone use LTO for storage or archiving image files?
« Reply #18 on: December 31, 2012, 02:22:25 pm »

One thing to consider about LTO is that it needs to be stored in ideal conditions and just dust can be a problem.

Another thing to be considered is passive storage vs active storage.

Tape you record it and then store it.

With a NAS system you can keep it live. Drives always running and the system continuously
checks on itself as well as refreshes the data. You can also record to the NAS and turn it off...
then run the thing for a day or so per month so that it checks on itself and refreshes the data.
If there is a problem with a sector on one drive it is refreshed from the other drives.

We have an LTO for some backup work. Massive data amounts from TV show production. Tapes are made
and then sent to a data storage center that has a program of migrating the data to new tapes
every so many years (way less than 20/30).

I use Buffalo raids made in Japan that run on linux. They are operated from your web browser.

Raid in my Camera (shoot to two memory cards at a time).
Cards get downloaded to the workstation.
My workflow is RAID in my workstation for work in progress. Very important as I work against deadlines.
Files are immediately backed up to the local NAS raid and shuttle hard drive and copied to a second NAS
at a second location (can also be done over the internet). Once a week the raids are compared over
the internet just in case something was missed.

Logged

Sareesh Sudhakaran

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 546
    • The Indie Farm
Re: Does anyone use LTO for storage or archiving image files?
« Reply #19 on: January 01, 2013, 06:32:18 am »

I missed your reply, Chris. I think we are talking about different things. I'll try to clarify.

LTO is designed with future support specifically in mind because it was designed to solve this particular archive problem. HDDs are not at all designed with this in mind.

No, they are not. If they were, and the people making them were that convinced, they would offer longevity in writing. They don't. Please show me one independent study of LTO that proves its longevity for archival purposes.

Quote
Yes. Insurance payout.

This category of enterprise storage is out of scope, it's very expensive. An LTO-5 drive is cheap in comparison to that. You can consider it a kind of insurance compared to using high density consumer hard drives as an archive media.

Why not insure consumer hard drives at a higher premium? Which is cheaper? If LTOs need to be insured to maintain their value, what is their value? And if only the most expensive can insure, what about those who can't? Is it cost that prohibits them?

Quote
You're ascribing a meaningful cost to using the hardware in a manner for which it's not designed. If you have a case where you need to be sifting through tape for one file to restore, you either have a bad design, or you had an edge case. If it's an edge case it does not need to be quantified and monetized.

I don't understand. One is always sifting through tape, and that's the very nature of tape. The inherent disadvantages of tape will remain, no matter what. And if anybody has to pay for it, it is a meaningful cost, and something that every investor has to consider. How can you say it's an edge case? It can happen to anybody. It's like saying accidents are an edge case, so why insure?

Quote
Right except a bad hot swap, bad shutdown, or bad mount, resulting in a corrupted file system. You cannot make a hard drive fool proof read only. It can write garbage.

Everything that can happen to tape as well.

Quote
Why? They aren't the same thing. The use cases are totally different.

You're the one who compared it to NTFS. I just pointed out more robust file systems, especially ZFS which is very popular for NAS.

Quote
Not significantly ahead of what? HDDs which aren't even archival? In the context of this thread, which is an archival context, HDD's simply do not qualify.

Neither does LTO.

Quote
So yes LTO is significanty ahead in this area. You can't even get a hard drive with a UER as low as LTO is. And you don't get the workflow with HDD that is implicit in LTO.

That's just LTO marketing rehashed. Billions of people have perfectly fine workflows, of every magnitude imaginable, without even knowing about LTO. And, to repeat your last sentence with roles reversed: You don't get the workflow with LTO that is implicit in HDD.

Quote
Now, if you have a small library, then indeed LTO may not make as much cost sense. But there is still a need to reduce the likelihood of data loss (incuding corruption) and that starts with not using high density consumer SATA drives if you care about your data.

LTO data can be lost forever, too. So, if I really cared about my data, I'll just make more backups with hard drives. I don't see any flaw in this reasoning.

Quote
What you're saying, is like saying Linux does not guarantee its own survival therefore you shouldn't use it for anything. It's ridiculous.

Linux is free, LTO isn't. Linux does not claim archival. Big difference. Equating the two is ridiculous. Archival means it has to guarantee its own survival. Isn't that what you have been arguing all along?

Quote
You're arguing against LTO because of an unknown that also applies to hard drives, yet you advocate hard drives over LTO. It's weird advice.

No, it's only weird if you don't value money. If you value money, when two things offer the same value proposition, choose the cheaper one. It's common sense. Since when has that become weird?

Quote
And in fact wrong because LTO is designed by the LTO consortium to survive as an archiving platform. SNIA is not doing that for hard drives. They are thinking in 5 year terms, whereas LTO is thinking in 25-50 year terms.

It's 30 years maximum, and that is not archival. And definitely not a viable technology if it is more expensive to maintain and use than 6 consumer grade drives over that same period.

Quote
If you can offer better odds over any other, you should produce your process. LTO has done this.

Hard drives do. I offer better odds for hard drive duplication over LTO. Everyone's doing it.

Quote
First you said if the error rates were relevant then Google and Amazon would use LTO. Second you said they don't use LTO. Yet they do use LTO. But then you change your own metric for what makes such error rates relevant, and still conclude that LTO isn't actually meaningfully better than drives. This despite the fact drives aren't designed with this in mind, but LTO is. It's pretty weird.

I was wrong about Google and Amazon. Regarding technical matters, I fully realize, and have stated earlier, that I am not qualified to go into the details of it. But if its concept is flawed, why waste time? LTO, for all its benefits, cannot prove its ability to archive.

I will reconsider LTO only if there is sufficient independent evidence of its longevity and ability to hold data over a 30 year period, guaranteed. It does not, and therefore its utility is limited. You're right, it is weird.

Quote
If you take someone's digital photo and encoded it as a cave painting, as soon as one fleck of paint came off the whole encoding would be destroyed and the photo lost to time.

It's funny, and weird as it may sound, you cannot prove your statement. You see, cave paintings have been around far longer than the language we are conversing in. I wouldn't be so hasty in my condescension.

In any case, I feel I am going overboard with this argument. I have deep reservations about LTO, which could turn out to be wrong. It wouldn't be the first time I've blundered, so I hope all this is taken in good humor.

Wish you a happy new year!
Logged
Get the Free Comprehensive Guide to Rigging ANY Camera - one guide to rig them all - DSLRs to the Arri Alexa.
Pages: [1] 2   Go Up