Pages: 1 [2] 3 4 5   Go Down

Author Topic: Now we're talkin STORAGE  (Read 26794 times)

Nill Toulme

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 738
    • http://www.toulmephoto.com
Now we're talkin STORAGE
« Reply #20 on: January 04, 2007, 09:06:37 am »

Quote
Hmmmm... Like this?

http://www.wiebetech.com/products/rt5.php

I like these too:

http://www.kanotechnologies.com/products/SV3X500R5S.cfm
[{POST_SNAPBACK}][/a]
Yes exactly!  Those are a little pricy though.     Mac Gurus has some enclosures that I've been looking at...

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Logged

feppe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2906
  • Oh this shows up in here!
    • Harri Jahkola Photography
Now we're talkin STORAGE
« Reply #21 on: January 04, 2007, 10:30:23 am »

Quote
Now that does make a bit of sense, although (a) I suspect that most of these drives are being used as 1.5TB RAID 0 (or JBOD or something worse?) drives, not as 750GB RAID 1, and ( there are still advantages to simply having two redundant external drives over RAID 1, e.g., operator error, etc.

Clearly there's no benefit of RAID 5 over RAID 1 in a two-disk array, but my whole point is that I don't *see* the point of two-disk arrays.  RAID 5's reliability benefits kick in at 3+ disks.

Nill
~~
www.toulme.net
[a href=\"index.php?act=findpost&pid=93618\"][{POST_SNAPBACK}][/a]

I could be wrong, but my understanding is that with RAID 5 you lose all your data if your controller card malfunctions (short circuit, lightning damage, etc.). With RAID 1 you don't have that problem.

CliffSamys

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 63
    • http://samys.com
Now we're talkin STORAGE
« Reply #22 on: January 04, 2007, 12:14:26 pm »

Quote
Now that does make a bit of sense, although (a) I suspect that most of these drives are being used as 1.5TB RAID 0 (or JBOD or something worse?) drives, not as 750GB RAID 1, and ( there are still advantages to simply having two redundant external drives over RAID 1, e.g., operator error, etc.

Clearly there's no benefit of RAID 5 over RAID 1 in a two-disk array, but my whole point is that I don't *see* the point of two-disk arrays.  RAID 5's reliability benefits kick in at 3+ disks.

Nill
~~
www.toulme.net
[a href=\"index.php?act=findpost&pid=93618\"][{POST_SNAPBACK}][/a]

I don't see the advantages of 2 drives vs RAID 1. Do you mean being able to take one off site?
RAID 5 requires a minimum of 3 drives by definition. You can't have a 2 drive RAID 5. But I still don't see its advantages over RAID 1 for what we are discussing. And as poited out, if the controller goes on a RAID 5, you will be very unhappy.
Logged
Cliff
Samy's Camera, Pro Digital Mana

Nill Toulme

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 738
    • http://www.toulmephoto.com
Now we're talkin STORAGE
« Reply #23 on: January 04, 2007, 12:27:39 pm »

Quote
I don't see the advantages of 2 drives vs RAID 1. Do you mean being able to take one off site?
[{POST_SNAPBACK}][/a]
There's that, plus if you accidentally permanently delete or otherwise hose a file or folder on a RAID 1 setup, by definition you do it on both disks, right?  With two separate disks — i.e., two separate backups — you reduce that risk significantly it seems to me.  

Nill
~~
[a href=\"http://www.toulme.net]www.toulme.net[/url]
Logged

djgarcia

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 341
    • http://improbablystructuredlayers.net
Now we're talkin STORAGE
« Reply #24 on: January 04, 2007, 04:25:04 pm »

Quote
I could be wrong, but my understanding is that with RAID 5 you lose all your data if your controller card malfunctions (short circuit, lightning damage, etc.). With RAID 1 you don't have that problem.
[a href=\"index.php?act=findpost&pid=93639\"][{POST_SNAPBACK}][/a]
If a controller malfunctions there's no telling what could happen, no matter what RAID level - you may get lucky or not. If it just stops functioning, no harm, just replace it. But if it goes medieval on your drives, well that's that. But the same would apply to a normal disk controller, so there's no saving grace here, RAID or not. I mean, lightning bolts don't discriminate against RAID disks particularly - any drive can be fried equally . However, a fault-tolerant RAID array could minimize your probability of failure, because if only one drive is fried you can recover.

Some levels provide more than one drive tolerance. RAID 10 uses mirrored pairs, so in a multiple mirror array you could conceivably lose one drive in each pair and recover while still working in degraded performance mode. But lose both drives in the same mirror and you're hosed. I had that happen with a backup utility that couldn't handle a specific controller but had no problems with others.

Keep in mind there are many RAID configurations. Some address performance (RAID 0), some address fault tolerance (RAID 1), and some do both (10, 5, 6, 50, 60) in some way or another. But any way you cut it, RAID does not safeguard against my own stupidity - if I erase a folder and purge the trashcan, it's gone, RAID or not. Hence the existence of BACKUPS ... which RAID complements but does not substitute by any stretch of the imagination.

The main thing to remember about RAID is that it's transparent to the user and the OS (assuming hardware RAID, the only type worth having IMHO), until the problem occurs, then you normally get a chance to fix it relatively quickly and painlessly.

No, RAID isn't cheap, so it's up to you what it's worth to insure your images and to what point. But if a gas main explodes in your basement, well, may the Force be with you ...
« Last Edit: January 04, 2007, 04:27:19 pm by djgarcia »
Logged
Over-Equipped Snapshooter - EOS 1dsII &

narikin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1371
Now we're talkin STORAGE
« Reply #25 on: January 04, 2007, 06:07:45 pm »

no-one seems to be mentioning the platform difference here.

a windows based case can house many many drives internally, along with an advanced hardware raid card. it's one of the main reasons I switched to PC rather than Mac 4 years ago - Apple's ridiculous 2 drives internal for a top model was a joke, as is the idea of using endless firewire drives to store serious image library on.

In an upgrade a year ago, I settled on a PC case with 12 internal quick swap bays, (plus extra space for an OS drive etc) and put in an Areca raid card. total cost for 6Tb in Raid 6 (double redundancy of raid 5) was $2800. you would now get 9Tb for the same $. Throughput is simply amazing with 12 drives reading/writing all at once.

an x-raid was going to cost me over $10,000 more, for the same thing.  hmmm.
Logged

narikin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1371
Now we're talkin STORAGE
« Reply #26 on: January 04, 2007, 06:12:49 pm »

oh, and check out the warranty on that Maxtor. I suspect its one year. which is ridiculous.

they are using the Seagate 750gb drives, which you can buy for $300 each with a 5 year warranty and put them in your case, attach them to your onboard MB raid and get the exact same thing - 1.5Tb storage/ backup (and get them replaced for free if they break down in 4 years time...)
« Last Edit: January 04, 2007, 06:13:37 pm by narikin »
Logged

tedchoi11

  • Newbie
  • *
  • Offline Offline
  • Posts: 20
Now we're talkin STORAGE
« Reply #27 on: January 05, 2007, 12:48:55 am »

Quote
CDs and DVDs aren't really comparable as DVDs have much higher data density and as thus are more susceptible to rotting.

It's possible that I'm a victim of fraud. But there are plenty of studies which suggest that "normal" recordable DVDs have a _practically_ limited shelf life. Limited in the sense that I nor any professional should trust them to work indefinitely. If you just burn two sets once, put them in a climate-controlled archiving cabinet and leave them for ten years, I wouldn't be surprised if there are quite a few ones that aren't readable.

Thankfully (?) tech advances at such a pace that people are re-burning or re-archiving files to different media every few years.
[a href=\"index.php?act=findpost&pid=93583\"][{POST_SNAPBACK}][/a]
i've had quite a few dvd-r discs rot over time. sometimes as soon as a few months. rough guess is that ~10% will fail within 1-3 years. it's harder to get catastrophic failure with hundreds of independent image files on a dvd-r, but a patch of rot will stop a movie cold. i see that all the time.

ted
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Now we're talkin STORAGE
« Reply #28 on: January 05, 2007, 02:30:46 am »

Quote
i've had quite a few dvd-r discs rot over time. sometimes as soon as a few months. rough guess is that ~10% will fail within 1-3 years. it's harder to get catastrophic failure with hundreds of independent image files on a dvd-r, but a patch of rot will stop a movie cold. i see that all the time.

ted
[a href=\"index.php?act=findpost&pid=93800\"][{POST_SNAPBACK}][/a]

Hmm! How odd! I was importing DVD movies from the US from the time the first DVD-ROM player became available in Australia; before there were hardly any titles available in Australia. I had my Creative Labs DVD player permanently set to Region 1, before I discovered a 'crack'.

I haven't come across a single disc, from those early years, that fails to play, whether from disc rot or any other cause. But I have bought the occasional pirated movie (from Malaysia and Thailand) that failed to play despite being new, and I have recorded images on a DVD disc which failed to play immediately after recording.

I have no reason to suppose that 'genuine' discs that have met standard QC requirements and that have been recorded properly should give any trouble. That's just my experience.
Logged

Jonathan Wienke

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5829
    • http://visual-vacations.com/
Now we're talkin STORAGE
« Reply #29 on: January 05, 2007, 03:50:04 am »

You're just lucky, Ray.
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Now we're talkin STORAGE
« Reply #30 on: January 05, 2007, 09:00:42 am »

Quote
You're just lucky, Ray.
[a href=\"index.php?act=findpost&pid=93815\"][{POST_SNAPBACK}][/a]

Well, I'm certainly not going to argue that I'm not   .
Logged

kaelaria

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2223
    • http://www.bgpictures.com
Now we're talkin STORAGE
« Reply #31 on: January 05, 2007, 09:00:52 am »

For anyone needing them/interested - Hitachi and Seagate just announced their 1TB drives at CES, starting at $399!
Logged

tedchoi11

  • Newbie
  • *
  • Offline Offline
  • Posts: 20
Now we're talkin STORAGE
« Reply #32 on: January 05, 2007, 09:04:23 am »

Quote
You're just lucky, Ray.
[a href=\"index.php?act=findpost&pid=93815\"][{POST_SNAPBACK}][/a]
i have to concur. at first i thought it was a disc quality issue so i only used name brand, store bought discs. Verbatim, Maxell, Apple. Not that they make their own, or do much beyond repackaging. I'm sure there's a list of who makes what somewhere on the internet. Following discussions of disc rot on the internet led me to a couple of supposedly high quality, japanese original manufacturer 'gold label' discs. much higher cost, but would be worth it if they didn't fail. or let me stop worrying about failure in my backups.

but it didn't help. not noticably, anyway. i think the real problem is in the burners. i've used pioneer burners since the 104 series, and the matsushita burner in my iMac. my guess is that the quality of the dot created by the burner is nowhere near as good as what can be done in high quality mass production, and are much more sensitive to media breakdown. i've tried burning at the slowest speeds rather than the fastest speeds, but that doesn't make a difference.

also, the reader/player is another variable. some are much better at reading messy discs than others, but i'm concerned it's through some sort of lossy error correction. this is essentially unnoticable on a movie, may possibly be noticable on a still image, and would be a big concern in an excel or filemaker file. i'd rather have the whole excel file fail to open than to have one or two cells with corrupted data that would lead to intermittent erroneous calculations!

bottom line: i only backup onto hard drives. i migrate my data often- probably every 6 or 12 months- onto the second-biggest drives available. so these days, all my data are on duplicate 500gb drives, as the 750s are prohibitiively expensive. when the 1tb drives come out, i'll start buying 750gb drives and sell the 500gb drives. so far, this is working well to keep the number of drives at a reasonable level.

ted
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Now we're talkin STORAGE
« Reply #33 on: January 05, 2007, 10:10:36 am »

Ted,
I guess bad news is always more interesting than good news. The issue is complicated by the fact that, as a result of media reports of disc rot and physical deterioration becoming widely knowm, every time there's a problem reading a disc, the cause will likely be attributed to that physical deterioration.
Logged

feppe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2906
  • Oh this shows up in here!
    • Harri Jahkola Photography
Now we're talkin STORAGE
« Reply #34 on: January 05, 2007, 10:11:10 am »

Quote
Hmm! How odd! I was importing DVD movies from the US from the time the first DVD-ROM player became available in Australia; before there were hardly any titles available in Australia. I had my Creative Labs DVD player permanently set to Region 1, before I discovered a 'crack'.

I haven't come across a single disc, from those early years, that fails to play, whether from disc rot or any other cause. But I have bought the occasional pirated movie (from Malaysia and Thailand) that failed to play despite being new, and I have recorded images on a DVD disc which failed to play immediately after recording.

I have no reason to suppose that 'genuine' discs that have met standard QC requirements and that have been recorded properly should give any trouble. That's just my experience.
[a href=\"index.php?act=findpost&pid=93807\"][{POST_SNAPBACK}][/a]

You're also comparin apples to oranges: DVDs and recordable DVDs (DVD-R and DVD+R) are very different beasts as the recording surface is not the same. In short, recordable DVDs are much more susceptible to rotting. That's the reasons why your legit DVDs haven't started rotting, and some of your pirated have - many pirated movies are actually recordable DVDs and not pressed in a factory.

Also, I've had the same issue as tedchoi11 below. I had an early Lite-On burner which did both DVD-R and DVD+R (feature which is pretty much standard these days). Now I have a batch of DVDs which can only be read with that burner as others won't work with them. That's only with one brand.

So, once again, DVDs are way too finicky for me to trust for backups. This holds even now when the tech is mature, as there is a much better alternative in external HDDs and RAID arrays.

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Now we're talkin STORAGE
« Reply #35 on: January 12, 2007, 07:57:41 pm »

Quote
So, once again, DVDs are way too finicky for me to trust for backups. This holds even now when the tech is mature, as there is a much better alternative in external HDDs and RAID arrays.
So far, the "external HDDs and RAID arrays" crowd have been very good at dissing DVDs.

But you haven't been very good at showing that your alternative is better.

Harddisks are also prone to failure, they're susceptible to power surges, electromagnetic noise and physical shock, not to forget wear, tear and mechanical failures.

Perhaps you place too big a trust in some layman's reading of the term MTBF as the only meaningful measurement of a HDD's lifetime?

It is typical for hard drives to have a service life of three to five years (though most drives probably will live for ten or twelve, maybe more).

MTBF should only be considered in concert with the service life and/or warranty time of the drive.

So, what does this mean for you, in practice?

Well, you need to keep migrating your data to new media every few years, regardless of the choice of storage media.

And when you migrate your data to new media, the most important bit is to check that:

1) Your data isn't corrupt.
2) Your data isn't corrupt after migration.


An alternative to disk drive "backups" is to make tape backups, but then you need to avoid the bad kinds of backup tapes (DAT, for one) and try to find the ones that have a fairly long storage life (S-DLT, maybe, or LTO-3). But of course, even the cheap ones aren't really cheap in terms of absolute number of dollars.

And you're still not safe from the issue of at least checking your data integrity, or migrating your data.

Got a headache yet? Good, because I certainly do.
Logged
Jan

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Now we're talkin STORAGE
« Reply #36 on: January 12, 2007, 09:22:51 pm »

Quote
Now I have a batch of DVDs which can only be read with that burner as others won't work with them. That's only with one brand.
[a href=\"index.php?act=findpost&pid=93868\"][{POST_SNAPBACK}][/a]

That's not a major problem, feppe. Migrate the data to another computer. Reburn it. The only insurmountable problem is when physical deterioration takes place so the disc is unreadable by any device.

I've seen pictures of such deterioration on this forum, which made me wonder what sort of environment could cause that. I've never experienced this myself and I've recorded literally thousands of CDs and DVDs over the years, often on the cheapest, best value media I could find in the local stores.

I have never, however, bought any blank discs over the internet, and I'm beginning to wonder if this impersonal type of internet trading is the cause of some dissatisfaction. If you are a retail store, you can't afford to sell bodgy products. If you are trading on the internet, you could probably sell a batch of reject blanks you'd picked up for next to nothing (perhaps from China) and then disappear.

I recently watched a documentary on the effects of atomic radiation. There's been a huge amount of research on the effects of the Chernobyl atomic power station accident. When watching this, I was reminded of this 'bit rot' situation with optical media. Bad reports get extrapolated and exaggerated. They fire our imagination and appeal to our basic fears about everything. Atomic radiation is a classic example. Over the years, public opinion has shifted from the viewpoint that the very word radium itself was symbolic of everthing good, hi-tech and wonderful, to the position that any amount of radiation, however small, is harmful.

As a result, the Chernobyl accident caused a great deal of consternation amongst people living in the vicinity of the accident. Women had abortions, as it now seems, for no good reason at all. As some great American president (or important historical figure) said (who was that?), the only thing we have to fear is fear itself.

Whilst it's true that some people died who were very close to the accident and who received massive doses of radiation (and I hope that I am not seen as trivialising their families' grief), there are lots of people who received moderate amounts of radiation, who've been quaking in their boots for the past 20 years, and who are still as fit as a fiddle.

An analysis of the wildlife in the close vicinity of the accident revealed very surprising results. The researchers expected to find all sorts of mutant rats and mice with two heads and three ears, or perhaps no wildlife at all. On the contrary, wildlife was thriving without a hiccup. Moderately strong radiation levels just seemed to have bounced off their back. In fact, it now seems that relatively small amounts of radiation, up to 100 millisieverts (per year) are actually beneficial to us. They stimulate our immune response. An analysis of areas in the United States that have strong, natural background radiation, have shown that people who live in such areas have less cancer, statistically, than people who live in areas with low background radiation.

I can't remember how many X-rays a 100 millisievert dosage represents, but it's into the thousands.

The message from me is, if you are worried about bit rot, don't worry. Get a life! (Unless of course you've been in the habit of buying ultra-cheap discs from the internet. You know, if something's too good to be true, it often is   ).
Logged

feppe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2906
  • Oh this shows up in here!
    • Harri Jahkola Photography
Now we're talkin STORAGE
« Reply #37 on: January 12, 2007, 09:47:27 pm »

Quote
An alternative to disk drive "backups" is to make tape backups, but then you need to avoid the bad kinds of backup tapes (DAT, for one) and try to find the ones that have a fairly long storage life (S-DLT, maybe, or LTO-3). But of course, even the cheap ones aren't really cheap in terms of absolute number of dollars.

And you're still not safe from the issue of at least checking your data integrity, or migrating your data.

Got a headache yet? Good, because I certainly do.
[a href=\"index.php?act=findpost&pid=95375\"][{POST_SNAPBACK}][/a]

I stand by my choice of HDD backups. As you said, there is a need to re-migrate to new media every few years whether one goes for CD, DVD or HDD. But as I've pointed out earlier, migrating hundreds - or even tens - of DVDs is an order of a magnitude more pain in the ass than migrating a few HDDs.

While tape is a decent solution, they are way too expensive as you point out, and due to them lacking random access are limited in their use.

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
Now we're talkin STORAGE
« Reply #38 on: January 12, 2007, 10:29:48 pm »

Quote
I stand by my choice of HDD backups. As you said, there is a need to re-migrate to new media every few years whether one goes for CD, DVD or HDD. But as I've pointed out earlier, migrating hundreds - or even tens - of DVDs is an order of a magnitude more pain in the ass than migrating a few HDDs.

While tape is a decent solution, they are way too expensive as you point out, and due to them lacking random access are limited in their use.
[a href=\"index.php?act=findpost&pid=95397\"][{POST_SNAPBACK}][/a]

The first few hard drives I ever owned (the first being an 850mb Western Digital) have all failed, after 10 or 12 years and sometimes well before that. Not a single optical disc has yet failed in that period. When it does, I'll let you know. (Are you going to be around, on this site, in another 10 or 20 years? Maybe I'll catch up with you on another site   ).
Logged

feppe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2906
  • Oh this shows up in here!
    • Harri Jahkola Photography
Now we're talkin STORAGE
« Reply #39 on: January 12, 2007, 11:24:10 pm »

Quote
The first few hard drives I ever owned (the first being an 850mb Western Digital) have all failed, after 10 or 12 years and sometimes well before that. Not a single optical disc has yet failed in that period. When it does, I'll let you know. (Are you going to be around, on this site, in another 10 or 20 years? Maybe I'll catch up with you on another site   ).
[a href=\"index.php?act=findpost&pid=95403\"][{POST_SNAPBACK}][/a]

I still haven't seen a single compelling argument to back up to optical media rather than hard disks. DVDs and CD-ROMs were the most convenient backup media until recently, but HDD tech has gone way past optical media in backups. When HD-DVD or BluRay recordables come down in price this _might_ change. But I have a feeling that HDD prices and capacities have galloped past by then, especially as it seems that multi-layer HD-DVD/BluRay recordables won't happen any time soon.

You're telling me you've used 10-year-old HDDs? That's, what, 300MB and slow as molasses? I don't know about you but the longest I've used a HDD was less than five years, and that's pushing it. Tech advances so fast that drives become cheaper and OS requirements steeper necessitating constant upgrading, unless one wants to stick to PS5 and 98SE.

Failure rate is largely a non-issue with redundant backups and integrity checks. And just as my failure rate is perhaps an anomaly, so is yours. Archival on newer optical media (DVDs and HD-DVD/BluRay) is arguably more prone to the elements and time than CD-ROMs due to higher data density (I know error correction has developed a lot).

Even if you do trust your current backups to last ten years redundancy is necessary, as there is the possibility of fire, flood, etc. which can only be avoided by redundant, off-site backups. But due to what I pointed above and in earlier posts, archival in the traditional sense is unnecessary these days as you can backup an entire hard disk from ten years ago - which cost hundreds of dollars - on a DVD which costs $1, or take up HDD space which costs cents.

In any case, I still can't believe anyone is backing up tens or hundreds of DVDs _and_ checking their integrity every once in a while - for that's necessary if you want to ensure your files stay intact, whether you use optical media or magnetic. It is obvious the convenience and cost of HDDs is unsurpassed. One can always gamble and rely on their backups working until they're needed, but that's asking for trouble.

Anyway, it appears to be a moot argument. It's your time and money you're throwing away.
Pages: 1 [2] 3 4 5   Go Up