Luminous Landscape Forum

Equipment & Techniques => Digital Cameras & Shooting Techniques => Topic started by: CliffSamys on January 03, 2007, 05:45:40 pm

Title: Now we're talkin STORAGE
Post by: CliffSamys on January 03, 2007, 05:45:40 pm
I thought some here may be interested in this article:

http://www.wired.com/news/technology/0,723...ml?tw=rss.index (http://www.wired.com/news/technology/0,72387-0.html?tw=rss.index)
Title: Now we're talkin STORAGE
Post by: alainbriot on January 03, 2007, 06:07:53 pm
Yes, things are moving :-) Seagate bought Maxtor. I just got several  Maxtor 1.5 TB OneTouch III Turbo Edition Hard Drives and they are great.  Small, silent, practical.  Far better than the LaCie 1TB I used before.  The new Maxtor are not as noisy, they are smaller and they stay cooler. They also come with Retrospect software, with One Touch Sync backup (couldn't be easier) are Mac formatted, and are ready to be configured as Raid 0 or 1.

http://www.maxtorsolutions.com/en/catalog/OTIII_Turbo/ (http://www.maxtorsolutions.com/en/catalog/OTIII_Turbo/)
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 03, 2007, 06:51:00 pm
Mais Alain, don't those drives consist of two 750GB drives chained together in one case, thereby doubling your chances of catastrophic drive failure for each one?

Nill
~~
www.toulme.net
Title: Now we're talkin STORAGE
Post by: Ray on January 03, 2007, 06:57:02 pm
Amazing and incredible! 50 terabits per square inch?? Wow! Couple that with a quantum computer and anything is possible   .
Title: Now we're talkin STORAGE
Post by: alainbriot on January 03, 2007, 07:17:53 pm
Quote
Mais Alain, don't those drives consist of two 750GB drives chained together in one case, thereby doubling your chances of catastrophic drive failure for each one?

Nill
~~
www.toulme.net
[a href=\"index.php?act=findpost&pid=93537\"][{POST_SNAPBACK}][/a]

I agree it is a risk, but I need large storage space.  Right now I have about 4 TB online.  Everything is backed up twice, so even if a drive crashes nothing is lost.
Title: Now we're talkin STORAGE
Post by: CliffSamys on January 03, 2007, 07:39:30 pm
Quote
Mais Alain, don't those drives consist of two 750GB drives chained together in one case, thereby doubling your chances of catastrophic drive failure for each one?

Nill
~~
www.toulme.net
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93537\")


If configured as RAID 1, this would be a mirrored 750GB RAID. This would be safe, as everything is stored on both drives.
RAID 0 is striping, where data is written alternately to each drive to increase speed.

By the way, this looks like a nice product. I've been pushing Samy's to start carrying a CalDigit item:

[a href=\"http://www.caldigit.com/FireWireVR.asp]http://www.caldigit.com/FireWireVR.asp[/url]

The Maxtor is better priced, though.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 03, 2007, 09:13:39 pm
Quote
I agree it is a risk, but I need large storage space.  Right now I have about 4 TB online.  Everything is backed up twice, so even if a drive crashes nothing is lost.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93542\")
I don't understand the point of these doubled drives though.  In what way are they better than two separate 750GB external drives?  It seems to me all they accomplish is doubling your chances of losing twice as much data.  (Does that equate to 4x the risk?)

If I had that much data (I only have about half a TB now) I'd have it all on a RAID5, backed up to another RAID5, with that backed up in turn to a bunch of 750GB external drives.  As it is, I have the live data on a 1.1TB RAID5, which backs up automatically nightly to two external 400GB drives, which are in turn backed up manually weekly to a single 750GB external that lives offsite.

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 03, 2007, 09:18:10 pm
Quote
...By the way, this looks like a nice product. I've been pushing Samy's to start carrying a CalDigit item:

http://www.caldigit.com/FireWireVR.asp (http://www.caldigit.com/FireWireVR.asp)
...[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93544\")
Something like that holding four drives instead of two, configurable as RAID5, and with a handle on top for portability, would seem just about ideal to me.  SATA might be preferable to firewire though.

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: Ray on January 03, 2007, 09:20:51 pm
Has it not been established that those who are the least bit paranoid about data loss, should back up their images on optical media like DVDs.

I know it's a bit tedious burning DVDs that hold only 4.4Gb, but we now have Blu-ray discs that hold 25Gb.
Title: Now we're talkin STORAGE
Post by: feppe on January 03, 2007, 09:44:40 pm
Quote
Has it not been established that those who are the least bit paranoid about data loss, should back up their images on optical media like DVDs.

I know it's a bit tedious burning DVDs that hold only 4.4Gb, but we now have Blu-ray discs that hold 25Gb.
[a href=\"index.php?act=findpost&pid=93560\"][{POST_SNAPBACK}][/a]

I don't see a reason to do DVD backups. Backing up 100GB, let alone 1TB on DVDs is not "a bit tedious," it's murder. Not to say unnecessary and ridiculously expensive compared to HDDs. Besides, you'll have to check the DVDs or re-burn them every year or so to ensure they're not rotting. Blu-ray and HD-DVDs seem to have higher (claimed) shelf-life but are not nearly as cost-effective as HDDs or even DVDs.

Hot-swappable RAID 0 or 5 setups are quite affordable these days, and external HDDs for off-site backing up cost even less. With HDD Mean-Time-To-Failure -rates at well over 1 million (!) hours and non-operating shock resistances in the hundreds of Gs you'll have to really try hard to get a failure leading to data loss with striped or double-backups.
Title: Now we're talkin STORAGE
Post by: alainbriot on January 03, 2007, 09:45:16 pm
Quote
Has it not been established that those who are the least bit paranoid about data loss, should back up their images on optical media like DVDs.

I know it's a bit tedious burning DVDs that hold only 4.4Gb, but we now have Blu-ray discs that hold 25Gb.
[a href=\"index.php?act=findpost&pid=93560\"][{POST_SNAPBACK}][/a]

I have everything backed up on DVD's as well.
Title: Now we're talkin STORAGE
Post by: feppe on January 03, 2007, 10:21:18 pm
Quote
I have everything backed up on DVD's as well.
[a href=\"index.php?act=findpost&pid=93564\"][{POST_SNAPBACK}][/a]

Umm..... 4 terabytes on DVDs? That's.... *calculates* .. *shocked* .. *doublechecks* ... almost one thousand (1,000) DVDs...

You, sir, either have the patience of a saint or a very patient assistant.

Well, I can see how it's not such a problem if you accumulate it over the years. But as I said, I would check that they work yearly in case they start rotting - even if I had double backups. Well, I guess if you check a DVD per day so that's a 3-year rotation which isn't that bad as long as you have two backups.

Still, a few external HDDs would be much more convenient.
Title: Now we're talkin STORAGE
Post by: Ray on January 03, 2007, 10:21:31 pm
Quote
Besides, you'll have to check the DVDs or re-burn them every year or so to ensure they're not rotting. [a href=\"index.php?act=findpost&pid=93563\"][{POST_SNAPBACK}][/a]

Nonsense! Have you ever thought of engaging in a bit of introspection to check your level of paranoia   .
Title: Now we're talkin STORAGE
Post by: feppe on January 03, 2007, 11:02:32 pm
Quote
Nonsense! Have you ever thought of engaging in a bit of introspection to check your level of paranoia   .
[a href=\"index.php?act=findpost&pid=93572\"][{POST_SNAPBACK}][/a]

Paranoia? It's a studied fact that DVDs rot. I have literally hundreds of DVD-Rs from less than 5 years back that are unreadable. Cheapest DVDs available with nothing I wouldn't miss on them, but the point remains. If you have two sets of backups you have to do some kind of rotated check to ensure they are still readable, and that you have time to copy them to new media when they start to do so without losing any data.

And as I pointed out, DVDs have currently nothing which makes them a better choice than HDDs as a backup media.

If a professional really wants to gamble with her income, be my guest. Investing a few hundred dollars on a decent backup system - whether it's DVD or HDD -based - is priceless in case of a equipment failure or fire.
Title: Now we're talkin STORAGE
Post by: CliffSamys on January 03, 2007, 11:19:58 pm
Quote
I don't understand the point of these doubled drives though.  In what way are they better than two separate 750GB external drives?  It seems to me all they accomplish is doubling your chances of losing twice as much data.  (Does that equate to 4x the risk?)
[a href=\"index.php?act=findpost&pid=93556\"][{POST_SNAPBACK}][/a]

For most people, doing a single backup is difficult enough. So to be able to write it to one device that automatically makes two copies makes sense. There are really no advantages to archiving to RAID 5 over RAID 1 in this type of scenario. In RAID 1 everything is written twice. It's only when people use RAID 0 (striping) that the failure rate increases.
Title: Now we're talkin STORAGE
Post by: CliffSamys on January 03, 2007, 11:24:08 pm
Quote
Something like that holding four drives instead of two, configurable as RAID5, and with a handle on top for portability, would seem just about ideal to me.  SATA might be preferable to firewire though.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93558\")

Hmmmm... Like this?

[a href=\"http://www.wiebetech.com/products/rt5.php]http://www.wiebetech.com/products/rt5.php[/url]

I like these too:

http://www.kanotechnologies.com/products/SV3X500R5S.cfm (http://www.kanotechnologies.com/products/SV3X500R5S.cfm)
Title: Now we're talkin STORAGE
Post by: Ray on January 03, 2007, 11:24:59 pm
Quote
Paranoia? It's a studied fact that DVDs rot. I have literally hundreds of DVD-Rs from less than 5 years back that are unreadable.[a href=\"index.php?act=findpost&pid=93578\"][{POST_SNAPBACK}][/a]

Wow! I can't understand that at all. All my ultra budget CDs recorded more than 10 years ago are still perfectly readable. I have no instances of DVDs that are unreadable, recorded 5 years ago (or so, or more).

Have you been the victim of fraud? Perhaps some company selling a batch of reject DVDs that they managed to acquire for nothing?
Title: Now we're talkin STORAGE
Post by: feppe on January 03, 2007, 11:37:24 pm
Quote
Wow! I can't understand that at all. All my ultra budget CDs recorded more than 10 years ago are still perfectly readable. I have no instances of DVDs that are unreadable, recorded 5 years ago (or so, or more).

Have you been the victim of fraud? Perhaps some company selling a batch of reject DVDs that they managed to acquire for nothing?
[a href=\"index.php?act=findpost&pid=93582\"][{POST_SNAPBACK}][/a]

CDs and DVDs aren't really comparable as DVDs have much higher data density and as thus are more susceptible to rotting.

It's possible that I'm a victim of fraud. But there are plenty of studies which suggest that "normal" recordable DVDs have a _practically_ limited shelf life. Limited in the sense that I nor any professional should trust them to work indefinitely. If you just burn two sets once, put them in a climate-controlled archiving cabinet and leave them for ten years, I wouldn't be surprised if there are quite a few ones that aren't readable.

Thankfully (?) tech advances at such a pace that people are re-burning or re-archiving files to different media every few years.
Title: Now we're talkin STORAGE
Post by: alainbriot on January 03, 2007, 11:45:14 pm
Quote
Umm..... 4 terabytes on DVDs? That's.... *calculates* .. *shocked* .. *doublechecks* ... almost one thousand (1,000) DVDs...

You, sir, either have the patience of a saint or a very patient assistant.
[a href=\"index.php?act=findpost&pid=93571\"][{POST_SNAPBACK}][/a]

I have been burning them as I go, starting around 1995 with CD's and now with DVD's :-)  Right now I am burning them at a rate of about 4 a week or so, with peaks and valleys, so to speak.  I also have everything on external disk drives.  4 copies of everything.  That keeps my paranoia in check.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 04, 2007, 09:00:18 am
Quote
For most people, doing a single backup is difficult enough. So to be able to write it to one device that automatically makes two copies makes sense. There are really no advantages to archiving to RAID 5 over RAID 1 in this type of scenario. In RAID 1 everything is written twice. It's only when people use RAID 0 (striping) that the failure rate increases.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93580\")
Now that does make a bit of sense, although (a) I suspect that most of these drives are being used as 1.5TB RAID 0 (or JBOD or something worse?) drives, not as 750GB RAID 1, and (b) there are still advantages to simply having two redundant external drives over RAID 1, e.g., operator error, etc.

Clearly there's no benefit of RAID 5 over RAID 1 in a two-disk array, but my whole point is that I don't *see* the point of two-disk arrays.  RAID 5's reliability benefits kick in at 3+ disks.

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 04, 2007, 09:06:37 am
Quote
Hmmmm... Like this?

http://www.wiebetech.com/products/rt5.php (http://www.wiebetech.com/products/rt5.php)

I like these too:

http://www.kanotechnologies.com/products/SV3X500R5S.cfm (http://www.kanotechnologies.com/products/SV3X500R5S.cfm)
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93581\")
Yes exactly!  Those are a little pricy though.     Mac Gurus has some enclosures that I've been looking at...

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: feppe on January 04, 2007, 10:30:23 am
Quote
Now that does make a bit of sense, although (a) I suspect that most of these drives are being used as 1.5TB RAID 0 (or JBOD or something worse?) drives, not as 750GB RAID 1, and ( there are still advantages to simply having two redundant external drives over RAID 1, e.g., operator error, etc.

Clearly there's no benefit of RAID 5 over RAID 1 in a two-disk array, but my whole point is that I don't *see* the point of two-disk arrays.  RAID 5's reliability benefits kick in at 3+ disks.

Nill
~~
www.toulme.net (http://www.toulme.net)
[a href=\"index.php?act=findpost&pid=93618\"][{POST_SNAPBACK}][/a]

I could be wrong, but my understanding is that with RAID 5 you lose all your data if your controller card malfunctions (short circuit, lightning damage, etc.). With RAID 1 you don't have that problem.
Title: Now we're talkin STORAGE
Post by: CliffSamys on January 04, 2007, 12:14:26 pm
Quote
Now that does make a bit of sense, although (a) I suspect that most of these drives are being used as 1.5TB RAID 0 (or JBOD or something worse?) drives, not as 750GB RAID 1, and ( there are still advantages to simply having two redundant external drives over RAID 1, e.g., operator error, etc.

Clearly there's no benefit of RAID 5 over RAID 1 in a two-disk array, but my whole point is that I don't *see* the point of two-disk arrays.  RAID 5's reliability benefits kick in at 3+ disks.

Nill
~~
www.toulme.net (http://www.toulme.net)
[a href=\"index.php?act=findpost&pid=93618\"][{POST_SNAPBACK}][/a]

I don't see the advantages of 2 drives vs RAID 1. Do you mean being able to take one off site?
RAID 5 requires a minimum of 3 drives by definition. You can't have a 2 drive RAID 5. But I still don't see its advantages over RAID 1 for what we are discussing. And as poited out, if the controller goes on a RAID 5, you will be very unhappy.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 04, 2007, 12:27:39 pm
Quote
I don't see the advantages of 2 drives vs RAID 1. Do you mean being able to take one off site?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=93662\")
There's that, plus if you accidentally permanently delete or otherwise hose a file or folder on a RAID 1 setup, by definition you do it on both disks, right?  With two separate disks — i.e., two separate backups — you reduce that risk significantly it seems to me.  

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: djgarcia on January 04, 2007, 04:25:04 pm
Quote
I could be wrong, but my understanding is that with RAID 5 you lose all your data if your controller card malfunctions (short circuit, lightning damage, etc.). With RAID 1 you don't have that problem.
[a href=\"index.php?act=findpost&pid=93639\"][{POST_SNAPBACK}][/a]
If a controller malfunctions there's no telling what could happen, no matter what RAID level - you may get lucky or not. If it just stops functioning, no harm, just replace it. But if it goes medieval on your drives, well that's that. But the same would apply to a normal disk controller, so there's no saving grace here, RAID or not. I mean, lightning bolts don't discriminate against RAID disks particularly - any drive can be fried equally . However, a fault-tolerant RAID array could minimize your probability of failure, because if only one drive is fried you can recover.

Some levels provide more than one drive tolerance. RAID 10 uses mirrored pairs, so in a multiple mirror array you could conceivably lose one drive in each pair and recover while still working in degraded performance mode. But lose both drives in the same mirror and you're hosed. I had that happen with a backup utility that couldn't handle a specific controller but had no problems with others.

Keep in mind there are many RAID configurations. Some address performance (RAID 0), some address fault tolerance (RAID 1), and some do both (10, 5, 6, 50, 60) in some way or another. But any way you cut it, RAID does not safeguard against my own stupidity - if I erase a folder and purge the trashcan, it's gone, RAID or not. Hence the existence of BACKUPS ... which RAID complements but does not substitute by any stretch of the imagination.

The main thing to remember about RAID is that it's transparent to the user and the OS (assuming hardware RAID, the only type worth having IMHO), until the problem occurs, then you normally get a chance to fix it relatively quickly and painlessly.

No, RAID isn't cheap, so it's up to you what it's worth to insure your images and to what point. But if a gas main explodes in your basement, well, may the Force be with you ...
Title: Now we're talkin STORAGE
Post by: narikin on January 04, 2007, 06:07:45 pm
no-one seems to be mentioning the platform difference here.

a windows based case can house many many drives internally, along with an advanced hardware raid card. it's one of the main reasons I switched to PC rather than Mac 4 years ago - Apple's ridiculous 2 drives internal for a top model was a joke, as is the idea of using endless firewire drives to store serious image library on.

In an upgrade a year ago, I settled on a PC case with 12 internal quick swap bays, (plus extra space for an OS drive etc) and put in an Areca raid card. total cost for 6Tb in Raid 6 (double redundancy of raid 5) was $2800. you would now get 9Tb for the same $. Throughput is simply amazing with 12 drives reading/writing all at once.

an x-raid was going to cost me over $10,000 more, for the same thing.  hmmm.
Title: Now we're talkin STORAGE
Post by: narikin on January 04, 2007, 06:12:49 pm
oh, and check out the warranty on that Maxtor. I suspect its one year. which is ridiculous.

they are using the Seagate 750gb drives, which you can buy for $300 each with a 5 year warranty and put them in your case, attach them to your onboard MB raid and get the exact same thing - 1.5Tb storage/ backup (and get them replaced for free if they break down in 4 years time...)
Title: Now we're talkin STORAGE
Post by: tedchoi11 on January 05, 2007, 12:48:55 am
Quote
CDs and DVDs aren't really comparable as DVDs have much higher data density and as thus are more susceptible to rotting.

It's possible that I'm a victim of fraud. But there are plenty of studies which suggest that "normal" recordable DVDs have a _practically_ limited shelf life. Limited in the sense that I nor any professional should trust them to work indefinitely. If you just burn two sets once, put them in a climate-controlled archiving cabinet and leave them for ten years, I wouldn't be surprised if there are quite a few ones that aren't readable.

Thankfully (?) tech advances at such a pace that people are re-burning or re-archiving files to different media every few years.
[a href=\"index.php?act=findpost&pid=93583\"][{POST_SNAPBACK}][/a]
i've had quite a few dvd-r discs rot over time. sometimes as soon as a few months. rough guess is that ~10% will fail within 1-3 years. it's harder to get catastrophic failure with hundreds of independent image files on a dvd-r, but a patch of rot will stop a movie cold. i see that all the time.

ted
Title: Now we're talkin STORAGE
Post by: Ray on January 05, 2007, 02:30:46 am
Quote
i've had quite a few dvd-r discs rot over time. sometimes as soon as a few months. rough guess is that ~10% will fail within 1-3 years. it's harder to get catastrophic failure with hundreds of independent image files on a dvd-r, but a patch of rot will stop a movie cold. i see that all the time.

ted
[a href=\"index.php?act=findpost&pid=93800\"][{POST_SNAPBACK}][/a]

Hmm! How odd! I was importing DVD movies from the US from the time the first DVD-ROM player became available in Australia; before there were hardly any titles available in Australia. I had my Creative Labs DVD player permanently set to Region 1, before I discovered a 'crack'.

I haven't come across a single disc, from those early years, that fails to play, whether from disc rot or any other cause. But I have bought the occasional pirated movie (from Malaysia and Thailand) that failed to play despite being new, and I have recorded images on a DVD disc which failed to play immediately after recording.

I have no reason to suppose that 'genuine' discs that have met standard QC requirements and that have been recorded properly should give any trouble. That's just my experience.
Title: Now we're talkin STORAGE
Post by: Jonathan Wienke on January 05, 2007, 03:50:04 am
You're just lucky, Ray.
Title: Now we're talkin STORAGE
Post by: Ray on January 05, 2007, 09:00:42 am
Quote
You're just lucky, Ray.
[a href=\"index.php?act=findpost&pid=93815\"][{POST_SNAPBACK}][/a]

Well, I'm certainly not going to argue that I'm not   .
Title: Now we're talkin STORAGE
Post by: kaelaria on January 05, 2007, 09:00:52 am
For anyone needing them/interested - Hitachi and Seagate just announced their 1TB drives at CES, starting at $399!
Title: Now we're talkin STORAGE
Post by: tedchoi11 on January 05, 2007, 09:04:23 am
Quote
You're just lucky, Ray.
[a href=\"index.php?act=findpost&pid=93815\"][{POST_SNAPBACK}][/a]
i have to concur. at first i thought it was a disc quality issue so i only used name brand, store bought discs. Verbatim, Maxell, Apple. Not that they make their own, or do much beyond repackaging. I'm sure there's a list of who makes what somewhere on the internet. Following discussions of disc rot on the internet led me to a couple of supposedly high quality, japanese original manufacturer 'gold label' discs. much higher cost, but would be worth it if they didn't fail. or let me stop worrying about failure in my backups.

but it didn't help. not noticably, anyway. i think the real problem is in the burners. i've used pioneer burners since the 104 series, and the matsushita burner in my iMac. my guess is that the quality of the dot created by the burner is nowhere near as good as what can be done in high quality mass production, and are much more sensitive to media breakdown. i've tried burning at the slowest speeds rather than the fastest speeds, but that doesn't make a difference.

also, the reader/player is another variable. some are much better at reading messy discs than others, but i'm concerned it's through some sort of lossy error correction. this is essentially unnoticable on a movie, may possibly be noticable on a still image, and would be a big concern in an excel or filemaker file. i'd rather have the whole excel file fail to open than to have one or two cells with corrupted data that would lead to intermittent erroneous calculations!

bottom line: i only backup onto hard drives. i migrate my data often- probably every 6 or 12 months- onto the second-biggest drives available. so these days, all my data are on duplicate 500gb drives, as the 750s are prohibitiively expensive. when the 1tb drives come out, i'll start buying 750gb drives and sell the 500gb drives. so far, this is working well to keep the number of drives at a reasonable level.

ted
Title: Now we're talkin STORAGE
Post by: Ray on January 05, 2007, 10:10:36 am
Ted,
I guess bad news is always more interesting than good news. The issue is complicated by the fact that, as a result of media reports of disc rot and physical deterioration becoming widely knowm, every time there's a problem reading a disc, the cause will likely be attributed to that physical deterioration.
Title: Now we're talkin STORAGE
Post by: feppe on January 05, 2007, 10:11:10 am
Quote
Hmm! How odd! I was importing DVD movies from the US from the time the first DVD-ROM player became available in Australia; before there were hardly any titles available in Australia. I had my Creative Labs DVD player permanently set to Region 1, before I discovered a 'crack'.

I haven't come across a single disc, from those early years, that fails to play, whether from disc rot or any other cause. But I have bought the occasional pirated movie (from Malaysia and Thailand) that failed to play despite being new, and I have recorded images on a DVD disc which failed to play immediately after recording.

I have no reason to suppose that 'genuine' discs that have met standard QC requirements and that have been recorded properly should give any trouble. That's just my experience.
[a href=\"index.php?act=findpost&pid=93807\"][{POST_SNAPBACK}][/a]

You're also comparin apples to oranges: DVDs and recordable DVDs (DVD-R and DVD+R) are very different beasts as the recording surface is not the same. In short, recordable DVDs are much more susceptible to rotting. That's the reasons why your legit DVDs haven't started rotting, and some of your pirated have - many pirated movies are actually recordable DVDs and not pressed in a factory.

Also, I've had the same issue as tedchoi11 below. I had an early Lite-On burner which did both DVD-R and DVD+R (feature which is pretty much standard these days). Now I have a batch of DVDs which can only be read with that burner as others won't work with them. That's only with one brand.

So, once again, DVDs are way too finicky for me to trust for backups. This holds even now when the tech is mature, as there is a much better alternative in external HDDs and RAID arrays.
Title: Now we're talkin STORAGE
Post by: jani on January 12, 2007, 07:57:41 pm
Quote
So, once again, DVDs are way too finicky for me to trust for backups. This holds even now when the tech is mature, as there is a much better alternative in external HDDs and RAID arrays.
So far, the "external HDDs and RAID arrays" crowd have been very good at dissing DVDs.

But you haven't been very good at showing that your alternative is better.

Harddisks are also prone to failure, they're susceptible to power surges, electromagnetic noise and physical shock, not to forget wear, tear and mechanical failures.

Perhaps you place too big a trust in some layman's reading of the term MTBF as the only meaningful measurement of a HDD's lifetime?

It is typical for hard drives to have a service life of three to five years (though most drives probably will live for ten or twelve, maybe more).

MTBF should only be considered in concert with the service life and/or warranty time of the drive.

So, what does this mean for you, in practice?

Well, you need to keep migrating your data to new media every few years, regardless of the choice of storage media.

And when you migrate your data to new media, the most important bit is to check that:

1) Your data isn't corrupt.
2) Your data isn't corrupt after migration.


An alternative to disk drive "backups" is to make tape backups, but then you need to avoid the bad kinds of backup tapes (DAT, for one) and try to find the ones that have a fairly long storage life (S-DLT, maybe, or LTO-3). But of course, even the cheap ones aren't really cheap in terms of absolute number of dollars (http://www.exabyte.com/products/products/get_products.cfm?prod_id=601).

And you're still not safe from the issue of at least checking your data integrity, or migrating your data.

Got a headache yet? Good, because I certainly do.
Title: Now we're talkin STORAGE
Post by: Ray on January 12, 2007, 09:22:51 pm
Quote
Now I have a batch of DVDs which can only be read with that burner as others won't work with them. That's only with one brand.
[a href=\"index.php?act=findpost&pid=93868\"][{POST_SNAPBACK}][/a]

That's not a major problem, feppe. Migrate the data to another computer. Reburn it. The only insurmountable problem is when physical deterioration takes place so the disc is unreadable by any device.

I've seen pictures of such deterioration on this forum, which made me wonder what sort of environment could cause that. I've never experienced this myself and I've recorded literally thousands of CDs and DVDs over the years, often on the cheapest, best value media I could find in the local stores.

I have never, however, bought any blank discs over the internet, and I'm beginning to wonder if this impersonal type of internet trading is the cause of some dissatisfaction. If you are a retail store, you can't afford to sell bodgy products. If you are trading on the internet, you could probably sell a batch of reject blanks you'd picked up for next to nothing (perhaps from China) and then disappear.

I recently watched a documentary on the effects of atomic radiation. There's been a huge amount of research on the effects of the Chernobyl atomic power station accident. When watching this, I was reminded of this 'bit rot' situation with optical media. Bad reports get extrapolated and exaggerated. They fire our imagination and appeal to our basic fears about everything. Atomic radiation is a classic example. Over the years, public opinion has shifted from the viewpoint that the very word radium itself was symbolic of everthing good, hi-tech and wonderful, to the position that any amount of radiation, however small, is harmful.

As a result, the Chernobyl accident caused a great deal of consternation amongst people living in the vicinity of the accident. Women had abortions, as it now seems, for no good reason at all. As some great American president (or important historical figure) said (who was that?), the only thing we have to fear is fear itself.

Whilst it's true that some people died who were very close to the accident and who received massive doses of radiation (and I hope that I am not seen as trivialising their families' grief), there are lots of people who received moderate amounts of radiation, who've been quaking in their boots for the past 20 years, and who are still as fit as a fiddle.

An analysis of the wildlife in the close vicinity of the accident revealed very surprising results. The researchers expected to find all sorts of mutant rats and mice with two heads and three ears, or perhaps no wildlife at all. On the contrary, wildlife was thriving without a hiccup. Moderately strong radiation levels just seemed to have bounced off their back. In fact, it now seems that relatively small amounts of radiation, up to 100 millisieverts (per year) are actually beneficial to us. They stimulate our immune response. An analysis of areas in the United States that have strong, natural background radiation, have shown that people who live in such areas have less cancer, statistically, than people who live in areas with low background radiation.

I can't remember how many X-rays a 100 millisievert dosage represents, but it's into the thousands.

The message from me is, if you are worried about bit rot, don't worry. Get a life! (Unless of course you've been in the habit of buying ultra-cheap discs from the internet. You know, if something's too good to be true, it often is   ).
Title: Now we're talkin STORAGE
Post by: feppe on January 12, 2007, 09:47:27 pm
Quote
An alternative to disk drive "backups" is to make tape backups, but then you need to avoid the bad kinds of backup tapes (DAT, for one) and try to find the ones that have a fairly long storage life (S-DLT, maybe, or LTO-3). But of course, even the cheap ones aren't really cheap in terms of absolute number of dollars (http://www.exabyte.com/products/products/get_products.cfm?prod_id=601).

And you're still not safe from the issue of at least checking your data integrity, or migrating your data.

Got a headache yet? Good, because I certainly do.
[a href=\"index.php?act=findpost&pid=95375\"][{POST_SNAPBACK}][/a]

I stand by my choice of HDD backups. As you said, there is a need to re-migrate to new media every few years whether one goes for CD, DVD or HDD. But as I've pointed out earlier, migrating hundreds - or even tens - of DVDs is an order of a magnitude more pain in the ass than migrating a few HDDs.

While tape is a decent solution, they are way too expensive as you point out, and due to them lacking random access are limited in their use.
Title: Now we're talkin STORAGE
Post by: Ray on January 12, 2007, 10:29:48 pm
Quote
I stand by my choice of HDD backups. As you said, there is a need to re-migrate to new media every few years whether one goes for CD, DVD or HDD. But as I've pointed out earlier, migrating hundreds - or even tens - of DVDs is an order of a magnitude more pain in the ass than migrating a few HDDs.

While tape is a decent solution, they are way too expensive as you point out, and due to them lacking random access are limited in their use.
[a href=\"index.php?act=findpost&pid=95397\"][{POST_SNAPBACK}][/a]

The first few hard drives I ever owned (the first being an 850mb Western Digital) have all failed, after 10 or 12 years and sometimes well before that. Not a single optical disc has yet failed in that period. When it does, I'll let you know. (Are you going to be around, on this site, in another 10 or 20 years? Maybe I'll catch up with you on another site   ).
Title: Now we're talkin STORAGE
Post by: feppe on January 12, 2007, 11:24:10 pm
Quote
The first few hard drives I ever owned (the first being an 850mb Western Digital) have all failed, after 10 or 12 years and sometimes well before that. Not a single optical disc has yet failed in that period. When it does, I'll let you know. (Are you going to be around, on this site, in another 10 or 20 years? Maybe I'll catch up with you on another site   ).
[a href=\"index.php?act=findpost&pid=95403\"][{POST_SNAPBACK}][/a]

I still haven't seen a single compelling argument to back up to optical media rather than hard disks. DVDs and CD-ROMs were the most convenient backup media until recently, but HDD tech has gone way past optical media in backups. When HD-DVD or BluRay recordables come down in price this _might_ change. But I have a feeling that HDD prices and capacities have galloped past by then, especially as it seems that multi-layer HD-DVD/BluRay recordables won't happen any time soon.

You're telling me you've used 10-year-old HDDs? That's, what, 300MB and slow as molasses? I don't know about you but the longest I've used a HDD was less than five years, and that's pushing it. Tech advances so fast that drives become cheaper and OS requirements steeper necessitating constant upgrading, unless one wants to stick to PS5 and 98SE.

Failure rate is largely a non-issue with redundant backups and integrity checks. And just as my failure rate is perhaps an anomaly, so is yours. Archival on newer optical media (DVDs and HD-DVD/BluRay) is arguably more prone to the elements and time than CD-ROMs due to higher data density (I know error correction has developed a lot).

Even if you do trust your current backups to last ten years redundancy is necessary, as there is the possibility of fire, flood, etc. which can only be avoided by redundant, off-site backups. But due to what I pointed above and in earlier posts, archival in the traditional sense is unnecessary these days as you can backup an entire hard disk from ten years ago - which cost hundreds of dollars - on a DVD which costs $1, or take up HDD space which costs cents.

In any case, I still can't believe anyone is backing up tens or hundreds of DVDs _and_ checking their integrity every once in a while - for that's necessary if you want to ensure your files stay intact, whether you use optical media or magnetic. It is obvious the convenience and cost of HDDs is unsurpassed. One can always gamble and rely on their backups working until they're needed, but that's asking for trouble.

Anyway, it appears to be a moot argument. It's your time and money you're throwing away.
Title: Now we're talkin STORAGE
Post by: kaelaria on January 13, 2007, 01:09:42 am
If you want the most secure solution, build a dedicated raid 5 array, and rebuild it with new discs every 5 years.  As they fail individually in between main builds, you lose nothing.

You can build a 1TB array with external cabinet for less than $750 right now, and that will only be cheaper in the future.  

Big deal.

I run a 1TB raid 5 array in my system right now, and it cost me $600 in drives.
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 01:21:59 am
Quote
If you want the most secure solution, build a dedicated raid 5 array, and rebuild it with new discs every 5 years.  As they fail individually in between main builds, you lose nothing.

You can build a 1TB array with external cabinet for less than $750 right now, and that will only be cheaper in the future. 

Big deal.

I run a 1TB raid 5 array in my system right now, and it cost me $600 in drives.
[a href=\"index.php?act=findpost&pid=95416\"][{POST_SNAPBACK}][/a]

If your RAID card fries, so does your data (most likely and unless you want to spend thousands on data recovery service which might or might not recover anything). So you still need backups even if you're running a redundant RAID setup.

But damn, prices really have come down. I'm definitely getting a RAID 5 finally when I get a new computer.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 02:49:45 am
Quote
You're telling me you've used 10-year-old HDDs? That's, what, 300MB and slow as molasses?

No. I wasn't clear. My practice has been to hand down my old computers to friends, relatives and ex-wife. Those initial hard drives (an 850MB and a 2.1GB) have long since given up the ghost. Another 2 or 3 hard drives around 5, 10 and 20GB have been the victim of a power surge. It's fortunate I wasn't relying upon such drives as a back-up for my images.

I had a selection of my slides transferred to Photo CD by Kodak a year or so before I bought my first computer. I used such discs to test computers in the store whilst shopping around for a suitable computer. This was the days when the average video card was 500kb. 1MB was a premium card and 2MB a rarity. That first computer was the most expensive I've ever bought and slow as molasses, as you say. It took a full 2 minutes for an 18mb image to be read and decompressed.

Those first CD-ROMs from Kodak, now about 14 or 15 years old, are still perfectly readable. My first hard drive (and 2nd, 3rd and 4th etc) are not.

Quote
Anyway, it appears to be a moot argument. It's your time and money you're throwing away.

I don't get you. It takes no more time to burn a CD or DVD than to brush your teeth. The cost is only significant when the technology is new. I wouldn't recommend burning to Blu-ray just yet.

However, in your situation I see the problem. To transfer 1TB of images to DVD would be a mammoth task. Pity you haven't been doing it regularly by degrees. You could then have peace of mind   .

On the cost issue, I should mention that a LaCie 300GB external hard drive, in Australia, costs about A$1 per gigabyte, maybe a bit less now. Blank DVDs are about 10-20 cents per gigabyte.
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 07:25:36 am
Quote
If your RAID card fries, so does your data (most likely and unless you want to spend thousands on data recovery service which might or might not recover anything).
That depends on how the controller "fries". If it destroys the information stored on the disks, too, then you're out of luck.

If it doesn't, however, you can usually replace it with an identical model controller or a newer controller from the same vendor.

At work, we have successfully upgraded a couple of our RAID 10 setups at least two times this way, and recovered from RAID controller failure once.

We have not experienced complete RAID failure from a fried RAID controller.

This is with the 3Ware 8xxx and 9xxx series controllers.

Unless other RAID controllers are worse, your worries are not very rational.

Worry about Windows corrupting your file system instead.

Quote
So you still need backups even if you're running a redundant RAID setup.

But damn, prices really have come down. I'm definitely getting a RAID 5 finally when I get a new computer.
As I've mentioned in another thread, RAID 5 isn't the "most secure" solution as kaelaria touts.

RAID 5 allows the failure of any single drive, but steals less capacity from the total number of drives; the equivalent of one drive per RAID is "lost" capacity.

RAID 6 beats RAID 5 by using another parity stripe set, and allows for failure of any two drives. The equivalent of two drives per RAID is "lost" capacity.

RAID 1 (mirror) and 10 (AKA RAID 1+0, stripe of mirrors) allows for failure of up to 50% of the drives in the RAID. The capacity of a RAID 1 or RAID 10 set is half of the total drive capacity.

3Ware sells controllers that are pretty efficient at RAID 6, and allow hot spares (unused drives that can be added automatically to the array if another drive fails). A home-building redundancy nut will probably either use mirrored RAID 5 or RAID 6 ("RAID 51" or "RAID 61"), or RAID 10, with such hot spares.

Redundancy nuts with money will purchase storage solutions from e.g. NetApp or EMC, with backup tape robots and a remote storage vault in a mountain.


This redundancy nut isn't quite that paranoid currently, I "mirror" manually to two or three different drives, plus two different DVDs, after each shoot. I expect to go move to dual RAID 10 or something like that in a while.
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 08:06:54 am
Quote
As a result, the Chernobyl accident caused a great deal of consternation amongst people living in the vicinity of the accident. Women had abortions, as it now seems, for no good reason at all. As some great American president (or important historical figure) said (who was that?), the only thing we have to fear is fear itself.

Whilst it's true that some people died who were very close to the accident and who received massive doses of radiation (and I hope that I am not seen as trivialising their families' grief), there are lots of people who received moderate amounts of radiation, who've been quaking in their boots for the past 20 years, and who are still as fit as a fiddle.
I am one of those people who have received moderate amounts of radiation because of that accident.

I have, however, not been "quaking in my boots", but others certainly have.

I feel comfortable that the doses of radiation haven't been critically high, and that the chances of me developing cancer aren't significantly higher because of that accident.

Quote
An analysis of the wildlife in the close vicinity of the accident revealed very surprising results. The researchers expected to find all sorts of mutant rats and mice with two heads and three ears, or perhaps no wildlife at all. On the contrary, wildlife was thriving without a hiccup. Moderately strong radiation levels just seemed to have bounced off their back. In fact, it now seems that relatively small amounts of radiation, up to 100 millisieverts (per year) are actually beneficial to us. They stimulate our immune response. An analysis of areas in the United States that have strong, natural background radiation, have shown that people who live in such areas have less cancer, statistically, than people who live in areas with low background radiation.
The "strong, natural background radiation" is peanuts compared to the radiation in areas that were close the Chernobyl accident.

Areas closest to Chernobyl, such as the town of Zaborye in Brjansk, Russia, had concentrations of cesium-137 of above 100 curies per square km.

In an area of approximately 10 300 square km areas around Chernobyl, the concentration of cesium-137 is greater than 15 curies per square km, or more than half a million becquerel per square meter.

An additional 28 600 square km in Russia, Ukraine and Belarus measured 5 curies or more per square km.

Certain areas of Norway received concentrations of about 100 000 becquerel per square meter, enough to make lots of reindeer, sheep and mushrooms unfit for human consumption. (Peruse a map if you don't know how far away that is ...)

The half-life of cesium-137 is about 30 years.

Gomel in Belarus, 130 km northwest of Chernobyl, still has a concentration of 1-5 curies per square km.

A total of eight to ten million people still live under conditions that are more radioactive, but it's suspected that tens of millions risk serious health consequences because of radioactive materials transported by water. How great these risks are is debated, however.

(Feel free (http://www10.antenna.nl/wise/index.html?http://www10.antenna.nl/wise/449-450/4.html) to read  (http://www.stuk.fi/sateilytietoa/sateily_ymparistossa/itameri/en_GB/laskeuma/) a bit more (http://en.wikipedia.org/wiki/Chernobyl_accident))
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 08:35:39 am
Quote
However, in your situation I see the problem. To transfer 1TB of images to DVD would be a mammoth task. Pity you haven't been doing it regularly by degrees. You could then have peace of mind   .

On the cost issue, I should mention that a LaCie 300GB external hard drive, in Australia, costs about A$1 per gigabyte, maybe a bit less now. Blank DVDs are about 10-20 cents per gigabyte.
[a href=\"index.php?act=findpost&pid=95423\"][{POST_SNAPBACK}][/a]

Mr Briot backs up terabytes of data on DVDs as well, but he's doing just that: copying stuff as he goes. This is IMO the only sane way to do DVD backups. But that doesn't solve the issue of verifying their integrity every few years, something I don't consider paranoid at all. I'm in a position to choose my backup regimen as I don't have size-hungry (photographic) backups dating back years, and currently HDDs are the most convenient.

I just checked prices here in Europe. 3.5" hard drives cost ~€0.20 per gigabyte (with €20-50 for the external casing, bringing an external 500GB drive to 25 eurocents per gig) while DVDs cost €0.10 in 100 DVD spindles - their price has gone down drastically here in the past 6 months. So you were right, DVDs are cheaper. But if one puts any money on time spent backing up and verifying it's a no-brainer.

BTW, I doubt hard drives cost three times as much in Australia than in already expensive Europe as your numbers suggest. Perhaps you're looking at the 2.5" prices or prices of some ultra-fast server drives?
Title: Now we're talkin STORAGE
Post by: kaelaria on January 13, 2007, 09:47:45 am
WOW I can't believe the level of absolute PARANOIA around here...lol!!  Either that or some of you should have auditioned for Monk

Some of you can find a theoretical fault with anything, so there IS NO solution for you, pure and simple.

Here in the corporate IT world we use the time tested methods, and are only bound by our departmental budgets.

My web, file and SQL servers are raid 5 arrays with monthly master and daily incrimental tape backups.  OMG the raid card may go out!  OMG the SCSI card could go out!  OMG a meteor could crash through the roof and kill the whole system!!  

At home I backup critical files one and a while from my raid array to an external drive that's only powered on for transfers.  OMG that might break too!!!

You guys sure are entertaining when you try and give computer advice
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 10:05:49 am
Quote
I just checked prices here in Europe. 3.5" hard drives cost ~€0.20 per gigabyte (with €20-50 for the external casing, bringing an external 500GB drive to 25 eurocents per gig) while DVDs cost €0.10 in 100 DVD spindles - their price has gone down drastically here in the past 6 months. So you were right, DVDs are cheaper. But if one puts any money on time spent backing up and verifying it's a no-brainer.
Since you're using harddrives, you'll be replacing them every three years or so anyway.

And for integrity's sake, let's assume that you're using RAID 1 or 10 (dual DVDs the competition).

Hard disk densities roughly quadruple every three years for the same price at a certain price point (around €100-200), let's assume that DVDs have a similar price curve, but last at least 20 years if stored properly.

If you shoot 10,000 images of 15 MB each year, and that you now currently have three year's worth of images, that's 450 GB of storage, requiring 2x500 GB drives at roughly €140 each. Then you need a RAID controller, that costs €37 for a four-port controller. (Prices from komplett.nl.)

In three years, you'll need another 2x1000 GB drives at €100 each (because prices don't drop that fast). In three more years, you'll need 2x1500 GB drives at €70 each, if prices for drives keep dropping. In three more years, 2x2000 GB at €50 each.

Hardware cost twelve years, presuming that you change no other components: €720.

DVD writer: €28
200 DVDs €174 + 200 DVDs €120 + 200 DVDs €90 + 200 DVDs €70.

DVD cost, presuming that you change no other components: €442.

But the drives use about 1.5 W in standby mode each. Over twelve years, that adds up to about 120 kWh, if they're turned on 25% of the day. Add 25% power supply overhead, and it's another €20.

So, basically, hard drives are not that much more expensive that it would matter much financially as long as we're sticking to a simple mirrored setup. But if you want your working storage to be online and available, the cost doubles again.

The problem for hard drives is remote location storage of backups, where DVDs win for convenience of transport. If you have access to networked storage, hard drives win on convenience, too.

And arguably, it's easier to verify the data integrity with hard drives.

But given the very small additional cost -- both in time and money -- of creating DVD backups as you go, it is prudent to do both: redundant hard drives and redundant DVDs.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 10:38:21 am
Quote
BTW, I doubt hard drives cost three times as much in Australia than in already expensive Europe as your numbers suggest. Perhaps you're looking at the 2.5" prices or prices of some ultra-fast server drives?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=95446\")

I'm looking at prices at [a href=\"http://www.computeralliance.com.au/parts.aspx?qrySubCat=HBK]http://www.computeralliance.com.au/parts.aspx?qrySubCat=HBK[/url] which is a local computer store in Brisbane with some of the best prices in town. Internal hard drives vary from around 40 cents to 70 cents per GB, but external hard drives are more expensive, ranging from a bit under a dollar per GB to a bit more than a dollar. A$1 is 0.60 Euros. There are always a few run-out models that are on special and might bring those prices down a bit.

Currently in that store, the best value DVDs are by LG, 16x in spindles of 50 for A$19. That's just 8.4 cents per GB.
Title: Now we're talkin STORAGE
Post by: larryg on January 13, 2007, 10:51:54 am
Strange things happen (what you least expect some times)

Our Novel system crashed.  The Novel program itself.  We have Raid (not sure which one) but the backup software only backed up the data (every night and on Friday with offsite storage)   But since the system itself (Novel) crashed we couldn't get anything going.  

Still working on restoring the Novel System so that we can get the whole system back up.

Redundancy seems to be essential in any arrangement.  Backup Backup and have other supporting systems that will preserve the integrety of the data and the systems.

I Currently:  Have a Raid system  (raid 0) then backup on an external Maxtor
that stays on site another backup on Maxtor goes home (along with archival DVD Gold backups at home).

I can see moving to a 1.5 TB   raid system with other drives as backup to that system.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 10:54:13 am
Quote
200 DVDs €174 + 200 DVDs €120 + 200 DVDs €90 + 200 DVDs €70.

DVD cost, presuming that you change no other components: €442.

[a href=\"index.php?act=findpost&pid=95457\"][{POST_SNAPBACK}][/a]


Those prices seem a bit inflated, Jani. Right at the moment I could buy 200 DVDs for just 46 Euros at a local store. Are you referring to the gold plated variety?  
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 10:58:09 am
Quote
Strange things happen (what you least ex
I Currently:  Have a Raid system  (raid 0) then backup on an external Maxtor
that stays on site another backup on Maxtor goes home (along with archival DVD Gold backups at home).
RAID 0 isn't really RAID.

RAID = Redundant Array of Inexpensive Disks

There is no redundancy in RAID 0.
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 11:01:08 am
Quote
Those prices seem a bit inflated, Jani. Right at the moment I could buy 200 DVDs for just 46 Euros at a local store. Are you referring to the gold plated variety? 
No, definitely not gold plated.

Samsung Pleomax 8x (http://www.komplett.nl/k/ki.asp?sku=317072)
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 11:06:17 am
Quote
WOW I can't believe the level of absolute PARANOIA around here...lol!!  Either that or some of you should have auditioned for Monk

...

You guys sure are entertaining when you try and give computer advice
[a href=\"index.php?act=findpost&pid=95454\"][{POST_SNAPBACK}][/a]

Having two backups, one at home and one at a friend's place or the bank's vault isn't paranoia, it's prudent practice when it comes to your living - if you're a pro - or your memories - if you're an amateur. The added cost and time of making the second backup and taking it off-site once a month or so is negligible compared to the financial and emotional heartache in case of a fire, flood or theft. Sure, there are other things to worry about with such cases, but why gamble on something you _can_ protect?

And no need for ad hominems.
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 11:09:48 am
Quote
I'm looking at prices at http://www.computeralliance.com.au/parts.aspx?qrySubCat=HBK (http://www.computeralliance.com.au/parts.aspx?qrySubCat=HBK) which is a local computer store in Brisbane with some of the best prices in town. Internal hard drives vary from around 40 cents to 70 cents per GB, but external hard drives are more expensive, ranging from a bit under a dollar per GB to a bit more than a dollar. A$1 is 0.60 Euros. There are always a few run-out models that are on special and might bring those prices down a bit.

Currently in that store, the best value DVDs are by LG, 16x in spindles of 50 for A$19. That's just 8.4 cents per GB.
[a href=\"index.php?act=findpost&pid=95459\"][{POST_SNAPBACK}][/a]

You can buy the external HDD as a package and pay premium, or you can buy a "normal" 3.5" HDD and an external casing on the cheap. My prices are based on the latter as I pointed out in my earlier post, and are significantly cheaper than buying an expensive external HDD package.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 11:18:02 am
Quote
I am one of those people who have received moderate amounts of radiation because of that accident.

I have, however, not been "quaking in my boots", but others certainly have.

I feel comfortable that the doses of radiation haven't been critically high, and that the chances of me developing cancer aren't significantly higher because of that accident.
The "strong, natural background radiation" is peanuts compared to the radiation in areas that were close the Chernobyl accident.

Areas closest to Chernobyl, such as the town of Zaborye in Brjansk, Russia, had concentrations of cesium-137 of above 100 curies per square km.

In an area of approximately 10 300 square km areas around Chernobyl, the concentration of cesium-137 is greater than 15 curies per square km, or more than half a million becquerel per square meter.

An additional 28 600 square km in Russia, Ukraine and Belarus measured 5 curies or more per square km.

Certain areas of Norway received concentrations of about 100 000 becquerel per square meter, enough to make lots of reindeer, sheep and mushrooms unfit for human consumption. (Peruse a map if you don't know how far away that is ...)

The half-life of cesium-137 is about 30 years.

Gomel in Belarus, 130 km northwest of Chernobyl, still has a concentration of 1-5 curies per square km.

A total of eight to ten million people still live under conditions that are more radioactive, but it's suspected that tens of millions risk serious health consequences because of radioactive materials transported by water. How great these risks are is debated, however.

(Feel free (http://www10.antenna.nl/wise/index.html?http://www10.antenna.nl/wise/449-450/4.html) to read  (http://www.stuk.fi/sateilytietoa/sateily_ymparistossa/itameri/en_GB/laskeuma/) a bit more (http://en.wikipedia.org/wiki/Chernobyl_accident))
[a href=\"index.php?act=findpost&pid=95440\"][{POST_SNAPBACK}][/a]

These are exactly the sort of figures that put the wind up people. They sound frightening. In the recent documentary I saw, scientist who had been involved in the research were interviewed. At this point, 20 years after the accident, the only significant increase in cancer has been in Thyroid cancer in children, due to radioactive iodine in milk, it is thought. That's bad enough of course and a real tragedy, but the massive outbreak of cancers such as leukemia doesn't appear to have occurred.

According to the first article you referred to, in Belarus there are 1.8 million people still living in heavily polluted areas and receiving 15 milliSieverts of radiation a year. Is that a typo? It is now thought that doses up to 100 milliSieverts can be beneficial.

Quote
Practically the whole territory was heavily contaminated by radionuclides. More than 1.8 million people are still living in heavy polluted territories. The radiation dose they receive on average amounts to 15 mSv (milliSievert) a year.

Again from Wikipedia:

Quote
Nearly 20 years after the disaster, according to the Chernobyl Forum, no evidence of increases in the solid cancers and, possibly more significantly, none of the widely expected increases in leukemia have been found in the population.
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 11:19:34 am
Quote
The problem for hard drives is remote location storage of backups, where DVDs win for convenience of transport. If you have access to networked storage, hard drives win on convenience, too.

And arguably, it's easier to verify the data integrity with hard drives.

But given the very small additional cost -- both in time and money -- of creating DVD backups as you go, it is prudent to do both: redundant hard drives and redundant DVDs.
[a href=\"index.php?act=findpost&pid=95457\"][{POST_SNAPBACK}][/a]

Excellent, much-needed analysis on potential costs. HD-DVD and/or BluRay might change the picture even further in a year or two, especially if they come up with reasonably-priced multi-layer recordables.

I agree that creating DVD backups as you go is not an issue. But as I've been saying from the beginning, creating the same on HDDs wins purely from the convenience POV - especially when verifying their integrity down the line is taken into account -, even though the price for DVDs is cheaper.

One thing I'd like to point out that HDDs aren't necessarily any harder to transport than DVDs if you design your system for it. You can buy cheap hot-swappable HDD bays for PCs - and easy to retro-fit, also -, which is just as fast to take out as a DVD and just as easy to transport - well, I'd bring static-free pouches with me.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 11:26:52 am
Quote
You can buy the external HDD as a package and pay premium, or you can buy a "normal" 3.5" HDD and an external casing on the cheap. My prices are based on the latter as I pointed out in my earlier post, and are significantly cheaper than buying an expensive external HDD package.
[a href=\"index.php?act=findpost&pid=95471\"][{POST_SNAPBACK}][/a]

Well, let's not quibble about a few cents here and there. The point is, DVD storage is considerably cheaper than either internal or external hard drive storage.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 13, 2007, 12:19:13 pm
I backed up to DVD for about six months then lost interest.  There just got to be too many of the darn things.  And as much of a PITA as backing up to DVD is, I don't even want to think about having to restore half a TB or more from DVD.  Yikes!

And I still don't understand the appeal of a mirroring RAID system (e.g., RAID10) over, say, RAID5 plus redundant on- and off-site HDD backups.  Yes it provides more protection against some failure in the RAID itself, but not against all the other stuff that can bite you... OS burps, viruses, surges, operator headspace errors, etc.  Anything that writes a bad file to, or deletes a good file from, one side of the mirror does the same to the other side, does it not?  What good is that?  I'm sure I'm missing something here.

My live data is on a 1.1TB RAID5.  It backs up, automatically and nightly, to external firewire drives.  Those get backed up, manually and weekly or so, to another external drive that otherwise stays at my next door neighbor's.

Hmmm... I probably shouldn't be backing up from my backups, should I.  Maybe I'll change that.

Nill
~~
www.toulme.net
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 01:08:56 pm
Quote
According to the first article you referred to, in Belarus there are 1.8 million people still living in heavily polluted areas and receiving 15 milliSieverts of radiation a year. Is that a typo? It is now thought that doses up to 100 milliSieverts can be beneficial.
When you write "it is now thought that ..." you imply that this is the general consensus.

The claim is hardly uncontroversial, it's still only comparatively few researches who believe this may be the case. That isn't necessarily 100 milliSieverts, but "low dosages", which from what I gather means between 1 mSv and a few tens of mSv.

Typical background radiation is around 2.4 mSv per year.

Since you were so fond of quoting only a small part of a pretty long Wikipedia article, only to support your personal view, here's another bit of a Wikipedia article for you:

Radiation poisoning - table of exposure levels and symptoms (http://en.wikipedia.org/wiki/Radiation_poisoning#Table_of_exposure_levels_and_symptoms)

As you can see (http://www.scienceboard.net/community/perspectives.122.html), the levels mentioned as "healthy" in the Taiwan incident were on average 0.4 Sv/21 years, or slightly less than 20 mSv/year on average.

Another article (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=10453435&query_hl=6&itool=pubmed_docsum) suggests 0.4 cGy/month. 0.4 cGy/month is the equivalent of 4 mSv/month absorbed.

I've been unable to find supporting evidence that continued exposure to approximately 100 mSv/year is beneficial. Smaller doses, maybe, but not undisputedly (http://en.wikipedia.org/wiki/Hormesis#Non-acceptance).

BTW, I guess your source is BBC's Horizon (http://news.bbc.co.uk/1/hi/sci/tech/5173310.stm).

Obviously, more research is needed, but it's WAY too early to call such doses generally "safe" or even "healthy", even though it makes for good headlines and punchlines. Keep in mind that journalists tend to present conflicting views of a case, with no regard for which is the most accepted view, in order to seem "neutral".
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 01:32:42 pm
Quote
I backed up to DVD for about six months then lost interest.  There just got to be too many of the darn things.  And as much of a PITA as backing up to DVD is, I don't even want to think about having to restore half a TB or more from DVD.  Yikes!
Yes, that is a bummer, isn't it?

Considering that most people don't want a backup, they want a restore when their data is gone, that may be pretty significant.

Quote
And I still don't understand the appeal of a mirroring RAID system (e.g., RAID10) over, say, RAID5 plus redundant on- and off-site HDD backups.  Yes it provides more protection against some failure in the RAID itself, but not against all the other stuff that can bite you... OS burps, viruses, surges, operator headspace errors, etc.  Anything that writes a bad file to, or deletes a good file from, one side of the mirror does the same to the other side, does it not?  What good is that?  I'm sure I'm missing something here.
That is correct, but what good is RAID 5 or any kind of redundancy in the same system?

Sure, in-system redundancy only takes care of problems with the physical drives themselves, not with software or users. PEBCAK can't be solved with hardware.

However, OS "burps" that destroy already stored data are extremely rare, even with Windows, at least as long as you use NTFS as your file system instead of FAT. Malware that destroys data is perhaps a bit more frequent.

But you can be certain that one of your hard disks will fail sometime in the future, you just don't know when. Using several hard drives increases the risk significantly, of course.

Surges? You don't have a surge protector? That's irresponsible, and it's such a cheap measure, too.

You might also want to consider a UPS to protect against brown-outs and brief black-outs.

Quote
My live data is on a 1.1TB RAID5.  It backs up, automatically and nightly, to external firewire drives.  Those get backed up, manually and weekly or so, to another external drive that otherwise stays at my next door neighbor's.

Hmmm... I probably shouldn't be backing up from my backups, should I.  Maybe I'll change that.
As long as you verify that the data is the same in both backups (use e.g. MD5 checksums of each file, stored to a list, and an MD5 checksum of that file, too), there is nothing inherently wrong with that.

There is also a benefit with that kind of setup; you can make sure your main system remains available and unencumbered while making the second backup.
Title: Now we're talkin STORAGE
Post by: narikin on January 13, 2007, 02:26:31 pm
Quote
If your RAID card fries, so does your data (most likely and unless you want to spend thousands on data recovery service which might or might not recover anything). So you still need backups even if you're running a redundant RAID setup
[a href=\"index.php?act=findpost&pid=95418\"][{POST_SNAPBACK}][/a]
no it doesn't.
a raid card can go down, and you can just put in a new (same model) raid card and its all back as it was.

hope this clears that up.

(+another reason to use 3rd party raid card, rather than onboard raid)
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 02:44:12 pm
Quote
You might also want to consider a UPS to protect against brown-outs and brief black-outs.
As long as you verify that the data is the same in both backups (use e.g. MD5 checksums of each file, stored to a list, and an MD5 checksum of that file, too), there is nothing inherently wrong with that.
[a href=\"index.php?act=findpost&pid=95505\"][{POST_SNAPBACK}][/a]

What's the checksum of a checksum for? I don't know much about the MD5 algorithm itself, but doesn't it include data in itself to verify its own integrity? This would yield a checksum of a checksum useless.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 13, 2007, 04:34:04 pm
Quote
That is correct, but what good is RAID 5 or any kind of redundancy in the same system?
Reliability.  In the nine months I've been running that RAID I've had a drive fail twice.  Now maybe that's just exceptionally bad luck, but it happened.  But when it happened, I lost absolutely nothing other than the small amount of time it took to yank out one drive and plug in another.  Then the RAID rebuilt itself.  No downtime, no restore, no "Yikes, is my backup good?" — just roll on.


Quote
However, OS "burps" that destroy already stored data are extremely rare, even with Windows, at least as long as you use NTFS as your file system instead of FAT. Malware that destroys data is perhaps a bit more frequent.

...Surges? You don't have a surge protector? That's irresponsible, and it's such a cheap measure, too.

You might also want to consider a UPS to protect against brown-outs and brief black-outs.
I have those things, of course.  User stupidity is my greatest exposure, and as I am the user, that exposure is exceptionally high in my case.  So I need systems that protect against that exposure as well as all the others.  ;-)


Quote
As long as you verify that the data is the same in both backups (use e.g. MD5 checksums of each file, stored to a list, and an MD5 checksum of that file, too), there is nothing inherently wrong with that.
That makes sense.  I am using checksums on my compares, but not against a stored list.  I'll have to figure out how to do that.

Thanks,

Nill
~~
www.toulme.net (http://www.toulme.net)
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 04:45:18 pm
Quote
What's the checksum of a checksum for? I don't know much about the MD5 algorithm itself, but doesn't it include data in itself to verify its own integrity? This would yield a checksum of a checksum useless.
No, it's not a checksum of a checksum.

It's a checksum of the file containing all the other checksums.

That way, you can check -- with a reasonable degree of reliability -- whether your list of checksums is correct or corrupt.

Of course, that doesn't help you much if you find that the list is corrupt, but if you have another copy of the list, ... Storage is cheap for this kind of things, so having multiple copies of the checksum list is easy and prudent practice.
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 04:51:56 pm
Quote
Reliability.  In the nine months I've been running that RAID I've had a drive fail twice.  Now maybe that's just exceptionally bad luck, but it happened.  But when it happened, I lost absolutely nothing other than the small amount of time it took to yank out one drive and plug in another.  Then the RAID rebuilt itself.  No downtime, no restore, no "Yikes, is my backup good?" — just roll on.
In that case I misunderstood you, I thought you objected to this, because this is the point of a stripe of mirrors (RAID 10), too, only that you'll have better write and read performance, as well as better protection against failures (up to fully half the number of disks may fail without data loss).

The downside is that RAID 10 doesn't scale very well, whereas e.g. RAID 6 scales pretty well.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 13, 2007, 04:59:28 pm
Quote
In that case I misunderstood you, I thought you objected to this, because this is the point of a stripe of mirrors (RAID 10), too, only that you'll have better write and read performance, as well as better protection against failures (up to fully half the number of disks may fail without data loss).
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=95557\")
Well I suppose I'm thinking also from the standpoint of cost/benefit at the very bottom end of the scale (where I dwell).  Of course if cost is no object then RAID10 or even RAID6 affords additional reliability.  It just seems to me that RAID5, at the cost of only one additional drive's capacity, provides a very significant reliability enhancement.  Beyond that, I'd personally rather spend my few available dollars on redundant backups that protect me better against my biggest threat — my own screw-ups.  ;-)

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 05:27:32 pm
Quote
No, it's not a checksum of a checksum.

It's a checksum of the file containing all the other checksums.

That way, you can check -- with a reasonable degree of reliability -- whether your list of checksums is correct or corrupt.

Of course, that doesn't help you much if you find that the list is corrupt, but if you have another copy of the list, ... Storage is cheap for this kind of things, so having multiple copies of the checksum list is easy and prudent practice.
[a href=\"index.php?act=findpost&pid=95555\"][{POST_SNAPBACK}][/a]

Well, that's splitting hairs.

But I'm still under the impression that an MD5 checksum already includes integrity verification in itself. ie. if the MD5 checksum file is corrupted, it won't work nor give false results with the actual backed up data. Therefore getting checksums of checksums isn't necessary or even useful - just having multiple copies is enough. Correct me if I'm wrong.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 06:29:18 pm
Quote
Since you were so fond of quoting only a small part of a pretty long Wikipedia article, only to support your personal view, here's another bit of a Wikipedia article for you:
[a href=\"index.php?act=findpost&pid=95497\"][{POST_SNAPBACK}][/a]

It's not a personal view, Jani. More a personal summary of the thrust of that documentary, which reminds me in some ways of our current predicament regarding the reliability of recorded optical media.

We don't really know how reliable they are, statistically. We have the occasional report of discoloration on the surface of the disc and unplayability associated with 'bit rot'. We probably have many, many more instances of software and hardware incompatibility resulting from either the disc not being recorded properly in the first instance, or not being playable on all systems, perhaps due to someone not ticking the 'full compatibility' box, or perhaps not. That's an unfornate condition of the state of computers that we have to live with.

What I'm suggesting here is, if it were possible to collect the facts on all the DVD discs that have ever been produced, remove all the scams from the equation where known rejects have been sold, and remove all the instances where people have unwitingly applied adhesive labels to their discs which have chemically attacked the disc etc, we might find that only, say 0.000001% of those billions of discs have suffered from physical deterioration that makes them now unreadable, ie. one in a million. Those could be considered pretty good odds against things going wrong. But we simply don't know what the actual figure is. In the absense of the facts, people's fears often seem to take over.

From my personal experience, I'm very happy with the readability and longevity of all my optical discs. However, there have been many occasions, more frequently in the past that at present, when I have not been happy with the burn success rate. I have a stack of failed recordings about 9" high. I've kept the discs because I thought I might find a decorative use for them   .
Title: Now we're talkin STORAGE
Post by: larryg on January 13, 2007, 07:00:03 pm
Quote
RAID 0 isn't really RAID.

RAID = Redundant Array of Inexpensive Disks

There is no redundancy in RAID 0.
[a href=\"index.php?act=findpost&pid=95466\"][{POST_SNAPBACK}][/a]


Exactly, this is why I also beckup to external maxtors and the DVD's

Raid 0 (as best I can surmise) is mostly good when one hard drive crashes.  You then have all the data on another mirror drive.  This (at least in my situation) probably will not help if your system crashes.
Title: Now we're talkin STORAGE
Post by: kaelaria on January 13, 2007, 07:06:46 pm
OMG you guys are too much!

raid 0 is for performance ONLY!  If one drive crashes, you lose everything.  You do have ~40% better transfer rates though - it makes a nice difference in some applications!

Some of you guys are great free entertainment!  
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 07:25:15 pm
Quote
Well, that's splitting hairs.
Eh, no, it isn't.

There's a pretty big difference between doing a checksum of each checksum, and doing a checksum of a huge list of checksums.

Quote
But I'm still under the impression that an MD5 checksum already includes integrity verification in itself. ie. if the MD5 checksum file is corrupted, it won't work nor give false results with the actual backed up data. Therefore getting checksums of checksums isn't necessary or even useful - just having multiple copies is enough. Correct me if I'm wrong.
You're wrong; the checksum isn't self-correcting.

It's only a checksum of other data.

My point is that the list of checksums -- which for 500 GB of data can very well mean a pretty large file -- is also exposed to the same risk of bit rot per byte as any other kind of data.

Bit rot may change the printed hash of d41d8cd98f00b204e9800998ecf8427e to d41d8cd98f00b204e98%0998ecf8427e (obviously wrong) or d41d8cd98f00b204e9810998ecf8427e (possibly correct for some file).

Now if you have a checksum that's wrong, how can you know if it's the checksum or the file that's wrong?

Knowledge of the MD5 hash algorithm may rescue you, as may software coded by someone with knowledge of the algorithm.

I know that if I change _one_ bit in the middle of a JPEG, it will probably still display seemingly identically to the original, but that the checksum will be very different:

91f4fa04f6a82e7ec87e664eef1e3165  9366_Kurt_Maflin.jpg (http://jani.at.ifi.uio.no/hobbies/photo/portraits/9366_Kurt_Maflin.jpg)
823375454caf2a0a3670a858c8cfc87e  9366_Kurt_Maflin2.jpg (http://jani.at.ifi.uio.no/hobbies/photo/portraits/9366_Kurt_Maflin2.jpg)

However, checking this for a large file of checksums may be impractical, and it's really cheap to make a checksum of the file itself, and just store that, too.

There are several ways of doing it; I kindof like storing the md5sum of the checksum file in the file name of the checksum file (example is based on the bash shell in a unix or cygwin environment):

1) Create a list of checksums of JPEGs, raw files and Photoshop files in the file md5$sums:
$ md5sum *.jpg *.CR2 *.psd > md5sums
2) Generate a checksum of the file md5sums:
$ sum=$(md5sum md5sums|cut -f1 -d' ')
3) Rename the file:
$ mv md5sum $sum
4) Check the result:
$ ls -l md5*
-rw-r--r--  1 jani jani 168 Jan 14 01:14 md5sums.8d11a346ebd32e93a5a6adcebd55dee6

Now if I do a checksum of the file, and I get something else than what follows the dot, I know I have a problem.
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 07:29:03 pm
Quote
Exactly, this is why I also beckup to external maxtors and the DVD's

Raid 0 (as best I can surmise) is mostly good when one hard drive crashes.  You then have all the data on another mirror drive.
No, that's exactly what you haven't got.

If you run RAID 0, and one hard drive crashes, you've lost, because with RAID 0, the data is striped across several drives, with -- I repeat myself here -- no redundancy.

So if you have a RAID 0 of three drives, and one drive crashes, you suddenly have 0 drives worth of data.

However, if you run RAID 1 -- which is a real RAID solution -- you will get the behaviour you describe; one disk may crash, but the mirror drive is (probably) okay.


Nobody should be calling "RAID 0" for RAID at all, as it's just confusing you and many others who don't really know what's going on behind the scenes. But it's too late for that.
Title: Now we're talkin STORAGE
Post by: jani on January 13, 2007, 07:32:34 pm
Quote
What I'm suggesting here is, if it were possible to collect the facts on all the DVD discs that have ever been produced, remove all the scams from the equation where known rejects have been sold, and remove all the instances where people have unwitingly applied adhesive labels to their discs which have chemically attacked the disc etc, we might find that only, say 0.000001% of those billions of discs have suffered from physical deterioration that makes them now unreadable, ie. one in a million.
While it's unlikely that the figures would be this good -- one in ten thousand or one in a thousand seems more likely -- I agree with the principle.

Quote
I have a stack of failed recordings about 9" high. I've kept the discs because I thought I might find a decorative use for them   .
You could, I suppose, run them carefully (hehe) through your microwave, the patterns will be different for each disk, and they might actually be cool to look at.

Caveat lector: I cannot take responsibility for what may happen with you, your microwave, kitchen, furniture, apartment and/or house if you actually try this.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 07:39:11 pm
I see that I've been exaggerating when describing my first Kodak Photo CD disc as being 14 to 15 years old. I've just pulled out the first disc I ever had burned, still in its original case, and notice a date of 28th January 1995 on the cover, just a few days short of 12 years. It seems I've been using computers for not much more than 11 years.

I popped the disc in my Pioneer DVD drive and opened the first image on the disc. On my first computer, it took 2 minutes to open the full res file in 8 bit mode(18mb). Today it took just 10 seconds to read the file and display on monitor in 16 bit mode. Below is the image just as Kodak scanned it, unmodified apart from jpeg compression. This is the very first Kodachrome slide I had transferred to CD-ROM.

Image shot 42 years ago (Nepal); transferred to CD 12 years ago; displayed on LL today.

[attachment=1535:attachment]
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 13, 2007, 08:00:07 pm
Nice image — glad it still loads.  ;-)

Nill
~~
www.toulme.net (http://www.toulme.net)
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 08:59:47 pm
Quote
Nice image — glad it still loads.  ;-)

Nill
~~
www.toulme.net (http://www.toulme.net)
[a href=\"index.php?act=findpost&pid=95605\"][{POST_SNAPBACK}][/a]

Thanks! I've got a suspicion my photographic skills haven't improved much in the past 42 years   .

If this image had refused to load, and refused to load on every other DVD drive I have, it wouldn't be a major concern, because I've rescanned this image more than once with higher resolution scanners than Kodak used at that time.

I see here in this thread, concerns that might not apply to the amateur. If an image is half decent, I'll convert it and play around with it. It will exist in both RAW and TIFF formats on probably 2 or more different optical media discs, as well as a hard drive or two. If I consider the image to be really good, there'll be even more duplicates of it in one format or another, one rendition or another.

It really would be paranoia for me to worry about losing digitised images that are important to me. I'm far more worried about the physical deterioration of unscanned slides and negatives that go back to the days when my father was a school boy. Is it worth scanning them to preserve the memory for future generations that might not give a stuff? Should I scan them purely for the experience of nostalgia I might get and the satisfaction of doing the best job I can in extracting the most detail? Have I got better things to do?
Title: Now we're talkin STORAGE
Post by: feppe on January 13, 2007, 09:31:22 pm
Quote
Eh, no, it isn't.

There's a pretty big difference between doing a checksum of each checksum, and doing a checksum of a huge list of checksums.
You're wrong; the checksum isn't self-correcting.

I never said anything about MD5 being self-correcting; I know the difference between them and PAR files.

I know how checksums work, but I don't know if the software can tell if it's the checksum itself that's corrupt, or the data. If MD5 algorithm was designed cleverly - to have a checksum of itself in it - your practice would be unnecessary.
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 09:58:21 pm
Here's image no. 14 on the same disc. Taken with Pentax Spotmatic and 135mm lens. Unfortunately the image is not as sharp as I would have liked it to be, a result of the slow film speed and telephoto lens (not my inability to hold the camera steady, I'll add   ).

Notice the equality of the sexes in this underdeveloped country. Both men and women carrying the same load   .

[attachment=1536:attachment]

ps. I don't bother with check sums. I'm a cave man   .
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 13, 2007, 10:03:49 pm
Quote
...Is it worth scanning them to preserve the memory for future generations that might not give a stuff? Should I scan them purely for the experience of nostalgia I might get and the satisfaction of doing the best job I can in extracting the most detail? Have I got better things to do?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=95612\")
Yes; yes; possibly, right at the moment, but at some point...

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 13, 2007, 10:09:57 pm
Quote
Here's image no. 14 on the same disc. ...
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=95617\")
Another excellent image, and the scan might respond to some sharpening.  

Dang, if your eye was that good 42 years ago, I don't even want to see any of your current work.

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Title: Now we're talkin STORAGE
Post by: Ray on January 13, 2007, 10:54:02 pm
Quote
Yes; yes; possibly, right at the moment, but at some point...

Nill
~~
www.toulme.net (http://www.toulme.net)
[a href=\"index.php?act=findpost&pid=95619\"][{POST_SNAPBACK}][/a]


It'll have to be at some point. There's an insidious trap to the digital game. One can easily end up taking huge numbers of digital images, simply because one can with no additional expense.

But there is an expense, and that's the time taken to sort through them all and categorise them.

I'm currently at an impasse. I simply have too many images.
Title: Now we're talkin STORAGE
Post by: Ray on January 14, 2007, 12:20:42 am
Hey! Look at this image, no. 13 from the same disc. I have to thank you guys for provoking me into a controversial stance, otherwise I wouldn't have bothered revisiting these images   .

[attachment=1537:attachment]
Title: Now we're talkin STORAGE
Post by: Ray on January 14, 2007, 12:37:28 am
Quote
Dang, if your eye was that good 42 years ago, I don't even want to see any of your current work.

Nill
~~
www.toulme.net (http://www.toulme.net)
[a href=\"index.php?act=findpost&pid=95620\"][{POST_SNAPBACK}][/a]


Thank you for your kind comments but I fear I may have gone backwards  since those days.
Title: Now we're talkin STORAGE
Post by: Craig Arnold on January 14, 2007, 05:57:41 am
5 pages of thread about backups and no-one has even mentioned a tape drive?

If the data is important then backing up to tape is still the staple method. Particularly with this sort of data which is really just an archive with very little daily change a tape is ideal.

The LTO drives can manage hundreds of Gb per hour and hold up to 400Gb on a single tape. Backing up the size archive you are talking about could easily be done in a week, and maintenance could be done on a weekly schedule. Not onerous.

It's not cheap, but not much more expensive than maintaining everything on multiple hard disks.
Title: Now we're talkin STORAGE
Post by: djgarcia on January 14, 2007, 11:51:57 am
Quote
And I still don't understand the appeal of a mirroring RAID system (e.g., RAID10) over, say, RAID5 plus redundant on- and off-site HDD backups.  Yes it provides more protection against some failure in the RAID itself, but not against all the other stuff that can bite you... OS burps, viruses, surges, operator headspace errors, etc.[a href=\"index.php?act=findpost&pid=95485\"][{POST_SNAPBACK}][/a]
The main advantages of RAID 10 are the better redundancy (under certain circumstances), faster write times, and the fact that you could recover the whole array from a single disk from each mirror pair. I use it in my data array but my backup array which has the last 3-5 images of each of my logical system & data disks is RAID 5. My system disk is striped (RAID 0, which as Jani mentions has no redundancy).

When working your storage strategy, you need to keep in mind everyone's storage requirements have differences. My basic average requirement per single photo image adds up to around 220MB, 20MB for the raw 17MPxl file and 200MB for the working PSD. I could store maybe 18-20 in a DVD ... This is very different from Nil's, who has much smaller files but many, many more. How the storage configuration responds to his files will be very different from my files.

To sumarize:

- It cannot be over-emphasized, RAID (especially 0!) does not replace backups. It's complementary technology that can make your life more trouble-free and save you time when used properly.

- RAID 0 is strictly for performance, the more spindles (drives) the faster, but also more fragile. You lose one drive and you've lost the whole array content, not just the one drive's. 2 drives are almost not worth it but you get to 4 and you're flying. Get smaller drives because they add up. And DO NOT use more than one partition actively or you'll lose the performance advantage. IMO best for system and scratch disks.

- RAID 1 uses mirrored pairs, so you can lose one drive of each mirror and survive. You lose half the storage capacity of the combined disks. It has noticeably better write performance than RAID 5 because it doesn't have to calculate parity, but slower reads because it is not striped. Safe but slow.

- RAID 5 uses parity for redundancy and stripes the drives. You lose only 1 drive's worth of storage capacity so you can get relatively inexpensive huge capacities. Fast reads, writes similar to a single drive. You can lose 1 drive and function. IMO good for backup of several drive images.

- RAID 6 is like RAID 5 but uses a second drive for redundant parity. You can lose 2 drives and function, but you lose 2 drives from storage capacity. IMO best for backup if you can afford it.

- RAID 10 uses two or more RAID 1 mirrored pairs and stripes them (RAID 1, + RAID 0 on top) so it's much faster than RAID 1. Needs at least 4 drives. A 4-drive RAID 10 reads as fast as a 3-drive RAID 5 because it's in pairs, but writes faster than a 4-drive RAID 5. IMO best for data if you can afford it.

- Keep in mind that in striped arrays (0, 10, 5, 6) speed is gained from separate drives being accessed. A 4-drive RAID 5 will read faster than a 4-drive RAID 10 because the RAID 5 is accessing 3 spindles and the R10 only 2 (one in each mirror).

Hope I got it down clearly and correctly ...
Title: Now we're talkin STORAGE
Post by: djgarcia on January 14, 2007, 12:20:47 pm
Sorry, I mistakenly placed all parity in RAID 5 in a drive but that is what RAID 3 does. RAID 5 actually distributes parity among all the drives. I edited the original message to that effect.
Title: Now we're talkin STORAGE
Post by: Nill Toulme on January 14, 2007, 12:46:39 pm
Has tape really gotten price competitive with HD's?  I had a SCSI tape drive a few years ago, and it was a PITA.

Nill
~~
www.toulme.net (http://www.toulme.net)
Title: Now we're talkin STORAGE
Post by: jani on January 14, 2007, 02:40:52 pm
Quote
Has tape really gotten price competitive with HD's?
Yes and no.

If you want an LTO-3 or S-DLT drive for personal use, you'll probably not find it worth your while.

Now consider the possibility that you instead are a company with larger storage needs.

If you want to build a 12-disk RAID 5 with 750 GB drives, that will cost quite a bit of money.  If you want to purchase a ready-made solution, it will cost a bit more.

Tape robots may win simply on convenience and storage density.
Title: Now we're talkin STORAGE
Post by: narikin on January 21, 2007, 07:00:02 am
Quote
If you want to build a 12-disk RAID 5 with 750 GB drives, that will cost quite a bit of money.  If you want to purchase a ready-made solution, it will cost a bit more.
[a href=\"index.php?act=findpost&pid=95707\"][{POST_SNAPBACK}][/a]
I put a 12 disk raid 6 array for $3500 into my desktop computer.
its working right now on the machine I'm writing this from..

no biggie to put together, took about an hour to install, and I have enough storage for 3 years, with double redundancy.

one disk went down after 3 months, and it rebuilt the array in 2 hours. trouble free for last 9 months, including a relocate.

you still need deep backup for fire/flood/theft emergencies, but otherwise its a near bulletproof solution.