Luminous Landscape Forum

Equipment & Techniques => Digital Cameras & Shooting Techniques => Topic started by: whawn on December 23, 2008, 02:35:45 am

Title: Store Images On Flash Cards?
Post by: whawn on December 23, 2008, 02:35:45 am
...he says in the latest Photo Techniques issue.  Is this really practical?  Has anyone studied how long flash memory is good for if left full but otherwise unused?  I've googled a bit and have found only stats on read/write passes.  

If practical, the idea could be cost effective, given that flash prices keep dropping.

I agree with Vestal, BTW, about keeping all images shot.  You just never know...
Title: Store Images On Flash Cards?
Post by: dwdallam on December 23, 2008, 03:01:02 am
Quote from: whawn
...he says in the latest Photo Techniques issue.  Is this really practical?  Has anyone studied how long flash memory is good for if left full but otherwise unused?  I've googled a bit and have found only stats on read/write passes.  

If practical, the idea could be cost effective, given that flash prices keep dropping.

I agree with Vestal, BTW, about keeping all images shot.  You just never know...


I just read that essay too. I found it a bit hmmm, strange that he would discard flash cards just to store images. What a waste. Why not get a BlueRay DVD writer and store them on DVDs? BlueRay DVDs hold 50GBs. Or wqhy not buy a 1TB hard drive and use it externally--as a third drive-- to store all of your images? Come home, copy all the images to the external HD, unplug the hard drive, and you still have your flash cards to use. A 1TB hard drive is around 120 bucks, and that's for a good one.

I like some of the articles in PT but I find the mag increasingly dated. It's as if the contributors just found digital. In fact, there was a huge essay this month about wet darkroom printing. I doubt I'll renew my subscription. However, they have been looking for professional photographers and technical people to write updated essays. If that proves fruitful for them, I'd love to stay subscribed. As it is now, I haven't learned much of anything in the last several issues. As another example of dated information, was the article about airy disks and aperture diffraction. Wow.

I really think they should stay away from the technical because they end up reinventing the wheel that has been on the net for a long time, and is always available for reference, for instance, the airy disk and diffraction essay this month.

The essays I have found most interesting are those that David writes about being a human in relation to photography. I did find his thoughts on keeping photos interesting because, even though I know you never delete images for the most part, he shares his personal thinking on the matter, and that's the interesting part. That type of rumination is unique to each individual, although similar to all of us, and it allows me to relate to the person as a fellow human being.
Title: Store Images On Flash Cards?
Post by: whawn on December 23, 2008, 02:01:00 pm
Quote from: dwdallam
I like some of the articles in PT but I find the mag increasingly dated.
Seems to me the mag is trying to find a new identity.  They sent me a survey a couple of months ago with questions that seemed aimed at charting a total make-over, and this issue especially seemed to be vacillate between catering to newbies and what the 'puter biz calls 'power users.'  I honestly hope they return to the heavily technical and esoteric for the main part of the mag.  That's why I subscribed to begin with.  But, they may be just hangin' on; the circulation statement in this issue shows they print fewer than 30k copies of each issue, with a total paid circulation of around 18k.

Ctien's stuff is always interesting to me, and I enjoyed the essay on the darkroom, though I dismantled mine years ago.  Some days I sorta wish I hadn't.

About flash, though.  If the cards will hold data for a long, long time, it might be cost and labor effective to use them for storage.  At 50 bucks for 8G, my cost would be around .27 a shot, with no need to manage electronics, which compares very favorably to my out-of-pocket film and processing cost of almost .50 per shot, before storage materials.  Hard-drive space is cheaper, but it's also pretty delicate and is a PITA to manage.  I'm leery of DVD storage.  A DVD is nothing but tightly packed ('way more data per square inch, and thus much less robust) CD, and CDs don't have a good longevity record.  The lack of robustness is also true of high-capacity HDs.  The engineers are packing a terabyte into where once only a quarter gig dared to go, and the result is a delicate data condition.
Title: Store Images On Flash Cards?
Post by: dwdallam on December 24, 2008, 02:06:41 am
Quote from: whawn
Seems to me the mag is trying to find a new identity.  They sent me a survey a couple of months ago with questions that seemed aimed at charting a total make-over, and this issue especially seemed to be vacillate between catering to newbies and what the 'puter biz calls 'power users.'  I honestly hope they return to the heavily technical and esoteric for the main part of the mag.  That's why I subscribed to begin with.  But, they may be just hangin' on; the circulation statement in this issue shows they print fewer than 30k copies of each issue, with a total paid circulation of around 18k.

Ctien's stuff is always interesting to me, and I enjoyed the essay on the darkroom, though I dismantled mine years ago.  Some days I sorta wish I hadn't.

About flash, though.  If the cards will hold data for a long, long time, it might be cost and labor effective to use them for storage.  At 50 bucks for 8G, my cost would be around .27 a shot, with no need to manage electronics, which compares very favorably to my out-of-pocket film and processing cost of almost .50 per shot, before storage materials.  Hard-drive space is cheaper, but it's also pretty delicate and is a PITA to manage.  I'm leery of DVD storage.  A DVD is nothing but tightly packed ('way more data per square inch, and thus much less robust) CD, and CDs don't have a good longevity record.  The lack of robustness is also true of high-capacity HDs.  The engineers are packing a terabyte into where once only a quarter gig dared to go, and the result is a delicate data condition.


I want to say that I finished reading PT today and I really did like some of the information I got from it. I do remember that they were searching for a new personality too, and I think now upon further reflection that they may be on to something. The thing about the technical aspect is that for people who want that type of information, it's readily available online and in copious amounts. I'll give them another shot when my subscriptin expires, given tehy stay on the track they are seemingly on now.

As for storage, come on man. Hard Drives are REALLY safe these days. They've never been safer. I haven't had a HD drive in years and years and decades and crap I'm getting old.

I would say buy something like the Western Digital passport, which is a rubber encased 2.5 inch hard drive that needs no power cord as it runs off of the power in the USB connection it uses. If you're worried about a HD failing to the point it destroys your data, then what would be a solution is to buy several 80GB quality 2,5 inch USB drives, such as the passport, and use those. I think you can get a very good quality 80GB 2.5 inch HD for about 40 bucks. Remember to that if you only use the HD for backing up your images, it's only going to be on as long as it takes to back them up each time, or take them back off for use. That means the time on is going to be so incredibly small that a drive head to platter crash, or memory wipe due to some electronic surge or other phenomenon out of your control is millions to 1.

80GB for 50.00 dollars is 0.000625 per megabyte or 0.625 per GB. At that price buy three backup HD for the same files. Then you have triple backups for less than a penny a MB.
Title: Store Images On Flash Cards?
Post by: giles on December 24, 2008, 07:01:55 pm
Quote from: dwdallam
As for storage, come on man. Hard Drives are REALLY safe these days. They've never been safer. I haven't had a HD drive in years and years and decades
You haven't used enough disk drives.  

I come from an enterprise IT background, and all disks fail.  With enough disks online, disk failures are routine.  Perhaps no one of us with our own images only will have "enough" disks to expect a failure, but that the LL readership as a whole will lose a few disks in 2009 is a completely safe prediction.

Despite this I agree with you: hard disks are the most practical storage option we have, but as they fail, take precautions:

1. keep copies of your data on multiple drives
2. keep at least one copy of everything off site
3. update the disks from time to time, especially as connectivity options change (SCSI, IDE -> SATA, firewire seemingly on its way out, USB3 is on the horizon, eSATA may yet find some market share, who knows what next)
4. yes, #3 implies that even old "archived" images need to be migrated to new media
5. checksums to be sure that the data you stored is the data you read back are a good idea (sometimes disks lie), and painful to manage manually so I hope that Apple and Microsoft enhance their file systems along the lines of what Sun has done with ZFS

If I had CDs and DVDs that had images not on hard disks at all I'd be thinking of copying them; optical media can last a long time but not for ever and it was always hard to be sure if you had good media or not.

Directly on the original topic of long term storage on flash memory:

a. isn't that ridiculously expensive?
b. I'd want to see some studies of longevity
c. I'd take 'b' with a pinch of salt as the technology develops and changes so fast that long term studies will be out of date by the time they're published

Flash memory was designed to store data without a power source, but I don't think it was designed for long term storage.

Giles
Title: Store Images On Flash Cards?
Post by: dwdallam on December 25, 2008, 02:23:44 am
Quote from: giles
I come from an enterprise IT background, and all disks fail.  With enough disks online, disk failures are routine.  Perhaps no one of us with our own images only will have "enough" disks to expect a failure, but that the LL readership as a whole will lose a few disks in 2009 is a completely safe prediction.

Yes, but when you say fail, do you mean complete data loss? What I meant is that HDs yes do fail, but rarely do they fail to the point of complete or catastrophic data loss. Plus, like I said, if you only turn the disk on to copy files to it, what are the chances of it failing with such a low on line time per year, plus unplugged inbetween backups? About the only thing that would wipe it would be an electromagnetic bomb! I mean it would be on only minutes a year, not hours, and not under a server type load. And if you back your files up to two HDs, what are the chances then? It's just not reasonable to worry about it after that. And I think your posts does agrees with me too, so I understand what you are saying about failures.
Title: Store Images On Flash Cards?
Post by: joedecker on December 25, 2008, 06:08:46 pm
Quote from: giles
Directly on the original topic of long term storage on flash memory:
a. isn't that ridiculously expensive?

Cheaper than 35mm slides.  I did the calculations a couple years back and it was already true, so I'd guess it's true now (even if my file sizes have gone up.)

I never considered slides "ridiculously expensive."  All a matter of perspective, I guess.

But yeah, I agree with you--I don't know of any good information on how well flash devices hold data for long-term storage.

I have used flash devices as moderate-term storage (weeks)--I usually back up to two HD-backed devices each day during travel (a laptop and a hard drive gadget, typically Colorspace or, in earlier days, Epson), on a trip through Patagonia I had the power adapter (but not the laptop itself) blow a fuse (sizzle, sizzle!) from some electrical Badness in the Hosteria--for the rest of the trip, to make sure that I had two backups of everything, I just used the flash cards I had, which were enough for the rest of the trip.  Worked great.

Even then, I was smart enough to never connect the two devices during a trip, or have them both plugged into wall power at the same time.   I was certain that I was indulging in overparanoia until the Hosteria incident.   Now, I unplug 'em (on trips) when they're not charging, too.  

--Joe
Title: Store Images On Flash Cards?
Post by: whawn on December 25, 2008, 11:30:15 pm
Quote from: dwdallam
If you're worried about a HD failing to the point it destroys your data,
One bit dropped will ruin a photo file.  HDs -- as they come off the line and are certified for shipping -- fail to the point of destroying data.  They ALL do it, early and often.  The drive circuitry has error correction built in that usually corrects the trouble, but there is a threshold, and once that threshold is passed, your photo is destroyed for all time.

The tighter the data-packing, the greater the need for error correction and the more imminent the threshold limit becomes.  

[!--quoteo(post=246850:date=:name=from giles)--][div class=\'quotetop\']QUOTE (from giles) [a href=\"index.php?act=findpost&pid=246850\"][{POST_SNAPBACK}][/a][div class=\'quotemain\'][!--quotec--]1. keep copies of your data on multiple drives
2. keep at least one copy of everything off site
3. update the disks from time to time, especially as connectivity options change (SCSI, IDE -> SATA, firewire seemingly on its way out, USB3 is on the horizon, eSATA may yet find some market share, who knows what next)
4. yes, #3 implies that even old "archived" images need to be migrated to new media
5. checksums to be sure that the data you stored is the data you read back are a good idea (sometimes disks lie), and painful to manage manually so I hope that Apple and Microsoft enhance their file systems along the lines of what Sun has done with ZFS[/quote]
Did I mention that HDs are a PITA to manage?

I, too, have never had an HD fail.  I have, however, had a lot of files corrupted one way or another in ordinary operation.  Something as simple as moving a file from one directory to another (which, back in the old days, involved only the changing of a pointer in the FAT) has caused problems.  Disk optimization can bring on corruption.  Simple age will cause the magnetic bounds to spread and the fields to weaken, increasing the error rate.  Disk expansion/contraction from heat introduces tracking errors, exacerbated by the incredibly narrow tracks in use today.  And so on.  

In line with #5 above, I surely wish that MS (IBM is really to blame, but MS is easier to pick on) had included at least checksum validation in its file copy/transfer protocol.  But they didn't, haven't and most likely won't, so we must rely on 3rd party ware to do that for us, and that too is a PITA.

As for CD/DVD storage:  The medium of an RW disk is a kind of jelly sandwiched within plastics and garnished with a thin (nearly transparent) aluminum sheet.  The jelly is only semi-rigid, and is compacted when heated by the writing laser, leaving little (I mean tiny) pits and bumps, each one a bit, along a track.  When read, the laser heat is less, so it supposedly has no effect on the jelly.  Even so, over time (leaving aside oxidation of the metal), the jelly spreads, as any goo will, and begins to refill the pits.  After awhile: data destruction.  The only real question is the length of the while.  Gold foil (because it doesn't oxidize) increases longevity.  Certain jellies are said to resist spreading longer than others.  But... how long before CDs, et al., go the way of ten inch floppies (or half-inch tape) and become unreadable by way of technological change, regardless of data longevity?

My question, in the beginning (rephrased, but unchanged in intent): Has anyone tested long-term data integrity on flash media?  So far, it looks like NOT is the answer.
Title: Store Images On Flash Cards?
Post by: joedecker on December 26, 2008, 01:02:06 am
Quote from: whawn
My question, in the beginning (rephrased, but unchanged in intent): Has anyone tested long-term data integrity on flash media?  So far, it looks like NOT is the answer.

Not that I know of.  However, I do note that Sandisk gives an MTBF spec of 1 million hours, which is 114 years, in this white paper, PDF:
http://www.sandisk.com/Assets/File/OEM/Man...lashPMv12.0.pdf (http://www.sandisk.com/Assets/File/OEM/Manuals/OEMCFlashPMv12.0.pdf)

But I haven't found anything that really serves as a "Yes, we've tested this."


Title: Store Images On Flash Cards?
Post by: dwdallam on December 26, 2008, 02:11:36 am
Quote from: joedecker
Not that I know of.  However, I do note that Sandisk gives an MTBF spec of 1 million hours, which is 114 years, in this white paper, PDF:
http://www.sandisk.com/Assets/File/OEM/Man...lashPMv12.0.pdf (http://www.sandisk.com/Assets/File/OEM/Manuals/OEMCFlashPMv12.0.pdf)

But I haven't found anything that really serves as a "Yes, we've tested this."


I don't think the internet has that information. You're probably going to have to get to a computer science department on a University campus to answer that theoretical question in technical terms regarding physics and metallurgy, magnetics, etc., not anecdotal terms. Lots of people like to talk out of their asses trying to answer that question.

Some manufactures say "1 million hours" which is about 100+ years. However, thta may mean many things besides simple write a file and keep it out of weather and count the years the file can still be used.

Good question though.
Title: Store Images On Flash Cards?
Post by: giles on December 26, 2008, 08:57:14 am
Quote from: dwdallam
Yes, but when you say fail, do you mean complete data loss? What I meant is that HDs yes do fail, but rarely do they fail to the point of complete or catastrophic data loss.
Yes.  Failures like hearing the disk ring as the heads scrape along the platters.  Utterly impossible to recover any of the data off the disk.

Quote
Plus, like I said, if you only turn the disk on to copy files to it, what are the chances of it failing with such a low on line time per year, plus unplugged inbetween backups? About the only thing that would wipe it would be an electromagnetic bomb!
Electronic equipment ages even while powered off, so while the data available indicates increased chance of failure with increased use, you're still rolling the dice.  (A disk drive consists of its platters with the data on them, the heads that read that data, and controlling electronics.  It's often the electronics that go to la-la land in my experience, e.g. a "now you see it, now you don't" disk that keeps coming and going before finally and catastrophically stopping talking to the world.)

Quote
I mean it would be on only minutes a year, not hours, and not under a server type load. And if you back your files up to two HDs, what are the chances then? It's just not reasonable to worry about it after that. And I think your posts does agrees with me too, so I understand what you are saying about failures.
Ultimately it's risk analysis problem: how much effort (and expense) am I prepared to to spend to reduce my chances of data loss?  As ever, the law of diminishing returns applies.  Some of my experience has been at sites with high degrees of paranoia, e.g. a business that will be immediately insolvent if they lose their data, or a business that is subject to regulatory requirements and must be able to demonstrate a high degree of care.

Back in the real world, if I were to lose all my current images, I'd regret losing some (ones with sentimental value, e.g. family snaphots) and a few "good" images that I'm proud of, but the most of them ... well ... if even I don't think they're terribly good, they're probably not.

Giles
Title: Store Images On Flash Cards?
Post by: joedecker on December 26, 2008, 02:58:37 pm
Quote from: dwdallam
I don't think the internet has that information. You're probably going to have to get to a computer science department on a University campus to answer that theoretical question in technical terms regarding physics and metallurgy, magnetics, etc., not anecdotal terms. Lots of people like to talk out of their asses trying to answer that question.

Of course.  The academics wouldn't have an entire answer for the question either, though, because how long the charge will stay intact within the chip (say) is only one of the potential failure modes of such a device, and the other failure modes (degradation of the connections to the rest of the card, just to pick one) are going to be very dependent on manufacturing.

Anyway, not trying to argue, just pretty much claim "there isn't a good solid answer out there that I know of", what this problem really needs is someone who is willing to bring theory and experiment to bear on the problem, a la Wilheim.
Title: Store Images On Flash Cards?
Post by: whawn on December 26, 2008, 03:30:06 pm
Quote from: joedecker
I do note that Sandisk gives an MTBF spec of 1 million hours, which is 114 years, in this white paper, PDF:
http://www.sandisk.com/Assets/File/OEM/Man...lashPMv12.0.pdf (http://www.sandisk.com/Assets/File/OEM/Manuals/OEMCFlashPMv12.0.pdf)
<thread drift>
Now, THAT is a white paper!  My goodness, that takes me back to the good 'ol days.  It gives rejuvinated meaning to the term 'documentation.'  

Seems to me that manufacturers should find some happy medium, between what is now in the package and the real, full-meal deal, to provide routinely to customers.  

I bought one of those WD 'Passbook' 2.5 inch externals for my daughter's Xmas present (a student, with much need for music room) and the 'docs' occupied about 6 square feet of paper (both sides covered) in every language known to man, with the 'warranty' and disclaiming any responsibility for killing anyone.  Then, the paper said, one should plug it in and access the backup or transfer software provided on the disk.  That was it. No other word.  Nothing on the disk formatting (FAT32, I suppose, but don't know).  Nothing about not swinging it about one's head while operating.  Nothing about shock limits, the number of tracks, the sector size, or encoding method.  Certainly not a bad sector chart.  

Back in the day, drives and mobos and all came with stacks of paper similar to the Sandisk work and that was much too much, but now, somewhere along the line, everyone has gone too far the other way.

FWIW, my daughter is not tech-oriented, but is knowledgeable enough to set up and administer a small wireless/wired LAN, to understand the directory structure and how to manipulate it, and so on.  She was vastly disappointed and almost felt insulted by the lack of real information in this case.  Me, I was just plain shocked.
</thread drift>
Title: Store Images On Flash Cards?
Post by: dwdallam on December 27, 2008, 04:34:09 am
Quote from: giles
Yes.  Failures like hearing the disk ring as the heads scrape along the platters.  Utterly impossible to recover any of the data off the disk.


Electronic equipment ages even while powered off, so while the data available indicates increased chance of failure with increased use, you're still rolling the dice.  (A disk drive consists of its platters with the data on them, the heads that read that data, and controlling electronics.  It's often the electronics that go to la-la land in my experience, e.g. a "now you see it, now you don't" disk that keeps coming and going before finally and catastrophically stopping talking to the world.)


Ultimately it's risk analysis problem: how much effort (and expense) am I prepared to to spend to reduce my chances of data loss?  As ever, the law of diminishing returns applies.  Some of my experience has been at sites with high degrees of paranoia, e.g. a business that will be immediately insolvent if they lose their data, or a business that is subject to regulatory requirements and must be able to demonstrate a high degree of care.

Back in the real world, if I were to lose all my current images, I'd regret losing some (ones with sentimental value, e.g. family snaphots) and a few "good" images that I'm proud of, but the most of them ... well ... if even I don't think they're terribly good, they're probably not.

Giles

Take the disk to a shop doing forensics. If the data is on the disk, they'll get it off. And not only electronic equipment degrades over time when off, so do rocks and steel girders, and titanium submarines, and PU238. The point is if that is your position, nothing is safe and I agree with you 100%. It's a world of impermanence, Buddhism 101.


Use a external RAID rack. If a drive or two fails at the same time, you still have all the information. If all the drives fail catastrophically, then you lose your data. That's dice I'll roll any day because the chances of a 4 DISK RAID system failing to the point of losing data is slim, especially if you have it ALSO backed up on your primary HDs.  About the only way you can be sure your images will be safe from all problems for around 100 years is to print them using archival paper printing medium and store them in a vacuum in a tooled steel fire and bomb proof safe in Fort Knox or something, a safe within a safe, vacuum sealed, for later retrival and scanning.

The point is, telling people that multiple hard drive backups isn't really safe is wrongheaded. Sure it is. It's as safe as you can get barring extreme measures like the one I suggest above. If you're really worried about it, get a fireproof home safe and rotate the backups into that each day. Then floods and fire won't get you either, unless the safe is defective. And so on a so forth. By the time you get done protecting everything you have to that extreme point,  you'll be dead.

Alternately, if you have the money, buy a T3 connection and upload all your images to an online source that triple backs their data on HDs and tape and mirrors it in seven different countries around the globe. That's about as safe as you can get, even if they are on hard drives because of the redundancy. And each time a drive fails they replace it and your files eventually get moved around to new drives so data never has a chance to degrade sitting on one drive. Or you could upload your collections to NEWSGROUPS and then they propagate all over the world on millions of servers. Just encrypt your files. Free storage!  Just remember to reupload them about every three months.

Before a cheap way to share data, my friends and I use to use Nwsgroups to store our private files and share them with each other. We'd just encrypt them and use the world of News as secondary hard drives. Safe as long as they are retained by some newsgroup server. Slow if you don't have high bandwidth upload privileges.

Just get a few hard drives and rotate your backups, store the backup drives in a "safe" place, and your done.
Title: Store Images On Flash Cards?
Post by: Slough on December 27, 2008, 02:52:47 pm
It sounds like David Vestal is having a Ken Rockwell moment. As others have said, storing on CF is expensive compared to the alternatives. It also limits you to say 4 GB per card, so you then have issues with storing and retrieving the correct card.

One point worth making is if you do have backup disks, make sure you use a surge protector to protect against power spikes, and it might even make sense to keep the backup drives unplugged. Internal drives as backups are a bad idea as they share the same power supply as the main board. I had two internal drives blow at the same time. I remain suspicious about RAID unless the drives do not share the same power supply.

Regarding actual data loss on a HD being rare, perhaps that is true, but it usually costs a fortune to send the failed HD to a specialist who can recover the data, by for example replacing the controller circuitry.

It's quite sobering to think how long the fruits of our civilisation would last were society to collapse. Not long is my belief. The Dead Sea Scrolls lasted thousands of years and are largely readable. Some of our data would struggle to last 100 years.
Title: Store Images On Flash Cards?
Post by: dwdallam on December 27, 2008, 05:41:58 pm
Quote from: Slough
It sounds like David Vestal is having a Ken Rockwell moment. As others have said, storing on CF is expensive compared to the alternatives. It also limits you to say 4 GB per card, so you then have issues with storing and retrieving the correct card.

One point worth making is if you do have backup disks, make sure you use a surge protector to protect against power spikes, and it might even make sense to keep the backup drives unplugged. Internal drives as backups are a bad idea as they share the same power supply as the main board. I had two internal drives blow at the same time. I remain suspicious about RAID unless the drives do not share the same power supply.

Regarding actual data loss on a HD being rare, perhaps that is true, but it usually costs a fortune to send the failed HD to a specialist who can recover the data, by for example replacing the controller circuitry.

It's quite sobering to think how long the fruits of our civilisation would last were society to collapse. Not long is my belief. The Dead Sea Scrolls lasted thousands of years and are largely readable. Some of our data would struggle to last 100 years.


4GB? Where do you live? They are up to 100GB now.

You would buy an external RAID bay with its own power supply, and shut the power off to it--or unplug it--between each backup.

There is a National Geographic movie about what would happen to modern civilization's presence in the world, including buildings, roads, and data, if the human race were to go extinct. They say in a very short time, plant life and natural events would cover or destroy all roads and most buildings. Essentially, modern society would be wiped from history in a very, very short time. We don't have pyramids made out of rock. Modern society is much more "flux" oriented than ancient societies, which valued permanence  because of the stability it gave, especially along the Nile river delta. Anyway, I'm no historian, but it's a really good movie.

Title: Store Images On Flash Cards?
Post by: Slough on December 27, 2008, 06:17:44 pm
Quote from: dwdallam
4GB? Where do you live? They are up to 100GB now.

I live on Planet Earth. The largest CF card I can see on Warehouse Express, a respected UK dealer, is 32 GB. And as far as I can see you pay a premium for large size. Smaller cards are much cheaper on a per GB basis. Still, of you are a wealthy amateur living in Dubai ...
Title: Store Images On Flash Cards?
Post by: dwdallam on December 27, 2008, 10:27:15 pm
Quote from: Slough
I live on Planet Earth. The largest CF card I can see on Warehouse Express, a respected UK dealer, is 32 GB. And as far as I can see you pay a premium for large size. Smaller cards are much cheaper on a per GB basis. Still, of you are a wealthy amateur living in Dubai ...

Earth based information:

Source: http://en.wikipedia.org/wiki/CompactFlash#Technical_details (http://en.wikipedia.org/wiki/CompactFlash#Technical_details)

Capacities and compatibility

As of 2008[update], CompactFlash cards are generally available in capacities from about 512 MB to 100 GB, with perhaps the most popular choices in Europe and North America being between 1 GB and 16 GB. Lower capacity cards, below 512 MB, are becoming rare in stores as higher capacity cards are readily available at the same price. The largest CompactFlash cards commonly available currently are the 32 GB models from various manufacturers — SanDisk launched its 16 GB Extreme III card at the 2006 Photokina trade fair, Transcend announced its 32 GB card on January 15, 2008.[4] Samsung launched 16, 32 and 64 GB CF cards soon after. Pretec announced 48 GB cards in January 2008 and 100GB cards in September.[5][6][7] These cards, and almost all cards over 2 GB, require the host device to support the FAT32 file system (if the camera is using a FAT file system). The largest cards, however, are usually not among the fastest ones.
---------------

Buy now!!

Pretec's Stumps Up 64GB Compact Flash Card, and 100GB World's Largest
Posted by Kit Eaton at 5:58 PM on September 23, 2008

100GB CF cards.... holy moly that's huge. Pretec's saying it's the world's largest capacity CF, and I'm not going to argue. It's a 233x speed card, capable of access rates at 35MB/s and it, along with its smaller sibling, is being shown at Photokina this week. Not satisfied with that though, Pretec's also extending its range of ultra-fast cards (apparently the world's fastest) with 333x speed and 50MB/s data rates in 32GB and 50GB capacities. The 333x 32GB and 233x 64GB are due to ship now, for $US630 and $US400, and the 233x 100GB and 333x 50GB by the end of the year. But Pretec's being shy about pricing those, or indeed even showing a product pic. Press release below.


 

    COLOGNE, Germany & TAIPEI, Taiwan —(Business Wire)— Sep 22, 2008 Pretec, creator of the highest capacity and fastest CompactFlash card in the world, will demonstrate even higher capacities including a 64GB CF card, in the Leaf booth (West Hall 4.2, B009) and CFA (Hall 5.1, G-019) at Photokina 2008.

    Pretec today releases 64GB and 100GB, 233X CF cards with access speed of up to 35MB/s, overtaking the Pretec 48GB CF card, the previous world's record holder; and super high speed 333X 32GB and 50GB CF cards capable of running up to 50 MB per second of Read/Write speed, the highest speed CF card in the world.

    Pretec CF 233X and 333X CF cards use metal housing and ruggedized construction to provide extensive ruggedness. Pretec 64GB CF cards can be configured or reused to make a 256GB SATA, the highest capacity 2.5" SSD, by using Pretec Q-SATA, a patent-pending technology of flexible, scalable and modular SATA/PATA design based on up to 4 CF cards.

    Pretec CF cards will be demonstrated together with high performance digital cameras from Leaf, a business unit of Eastman Kodak. Their new Leaf AFi-II system delivers the fastest shooting speeds of any camera in its class. These speeds are enabled by advanced communication and CF technologies and Leaf Capture software. "We are pleased to have chosen Pretec as our exclusive storage partner at Photokina 2008 and to witness the excitement as Pretec breaks the CF card speed and capacity world records," said Seth Greenberg, Leaf Marketing Director.

    Pretec 233X 64GB and 333X 32GB CF cards with suggested retail price of $399 and $630 are slated to start delivery by Photokina 2008.

Source: http://www.gizmodo.com.au/2008/09/pretecs_..._largest-2.html (http://www.gizmodo.com.au/2008/09/pretecs_stumps_up_64gb_compact_flash_card_and_100gb_worlds_largest-2.html)

A little pricey for me, especially for data storage (cough).

-----------------

But no worries, here is a 64GB USB thumb drive for 110.00 on Amazon:

http://www.amazon.com/Kingston-Technology-...606&sr=1-18 (http://www.amazon.com/Kingston-Technology-DT150-64GB-DataTraveler/dp/B001JJBCBW/ref=sr_1_18/179-1310664-9315959?ie=UTF8&s=pc&qid=1230434606&sr=1-18)

Kingston Technology DT150/64GB 64GB DataTraveler 150 USB 2.0 Flash Drive (Red/Black)
Other products by Kingston

List Price:    $129.99
Price:    $113.99
------------

Earth is larger than Great Britain man! I mean haven't you heard of the country called AMAZON?

I'm joking you, okay?

Title: Store Images On Flash Cards?
Post by: Ben Rubinstein on December 28, 2008, 05:46:47 am
Quote from: dwdallam
Alternately, if you have the money, buy a T3 connection and upload all your images to an online source that triple backs their data on HDs and tape and mirrors it in seven different countries around the globe. That's about as safe as you can get, even if they are on hard drives because of the redundancy. And each time a drive fails they replace it and your files eventually get moved around to new drives so data never has a chance to degrade sitting on one drive. Or you could upload your collections to NEWSGROUPS and then they propagate all over the world on millions of servers. Just encrypt your files. Free storage!  Just remember to reupload them about every three months.

Of course that is only as safe as the finances of the company running the service, witness Digital Railroad...
Title: Store Images On Flash Cards?
Post by: Slough on December 28, 2008, 06:47:19 am
Quote from: dwdallam
Earth based information:

Blah blah blah.

I used UK suppliers because I am based in the UK and I know the UK cost of hard drives as reference. For UK people, using CF cards is an expensive and not very practical way to store data.

And when I look on Amazon in the land of apple pie and all that is good, I see ~$80 for a 64GB CF card. For the same price you can get a 500GB external drive. That still makes it an expensive and inconvenient alternative to hard drives. Of course there might be valid reasons why you might prefer CF disks. Being able to carry them around in the field and use them without the need for a power socket is one such reason.


Title: Store Images On Flash Cards?
Post by: dwdallam on December 28, 2008, 07:10:03 am
Quote from: pom
Of course that is only as safe as the finances of the company running the service, witness Digital Railroad...

NOt for Newsgroups it isn't. They propagate over thousands of servers besides the one you upload to or subscribe to.
Title: Store Images On Flash Cards?
Post by: Ben Rubinstein on December 28, 2008, 01:30:49 pm
Quote from: dwdallam
NOt for Newsgroups it isn't. They propagate over thousands of servers besides the one you upload to or subscribe to.

Tell me more, seriously...

BTW I hold you personally responsible (the CS4 thread) for me upgrading to 64 bit, buying 8GB more of RAM and now I've made a striped disk just for scratch/page file/Bridge cache. Appreciate your advice there.  
Title: Store Images On Flash Cards?
Post by: jjj on December 30, 2008, 04:56:41 pm
Quote from: joedecker
Cheaper than 35mm slides.  I did the calculations a couple years back and it was already true, so I'd guess it's true now (even if my file sizes have gone up.)

I never considered slides "ridiculously expensive."  All a matter of perspective, I guess.
Ah, but you forget that when people shoot digital, they will shoot a lot more photos than when using slide. A lot more.
I spent a couple of week in China, just before I switched to digital and shot just over 40 rolls of film.  Approximately 1600 images [and one battery would last almost exactly 40 rolls]. That was considered a lot of film.
People casually shoot a thousand pictures in a day now!
I also had a Canon Ixus 2.1M camera with a 'huge' and quite expensive 256M card and made it and the 64M card last the entire trip!
I've done jobs that I could not have afforded to do on film, as there was so much to shoot and for extended periods of time. I've even shot for 24hrs straight, with a different model or models every 20mins - 84 in total I think.

I gave up on DVDs for back up a long time ago as 4G was simply too small and slow to bother with. So compact flash would be very expensive a proposition. Last big trip I had 2x320G and 1x500G laptop drives with me.

Title: Store Images On Flash Cards?
Post by: lumpidu on January 03, 2009, 10:30:02 am
I am in the embedded computer business and we use CF cards as a reliable data storage for our data. We guarantee our customers 10 years lifetime for our products. We are using industrial grade CF cards which also have internal capacitors to survice a sudden power loss, so these are able to store what is already in the internal buffers.

IMHO what is the main problem with CF cards is flash wear / wear levelling and the used file system. FAT16 or FAT32 don't use any journalling, so these file systems can get corrupted if unconnected from the computer unexpectedly. Formattting CF cards with an appropriate file system is key for reliable backup. Mac OS X has AFS, Linux has ext3, Windows has NTFS. These are reliable file systems. Used on a rugged industrial grade CF card, these should be safe for almost 10 years.

But then again using these file systems for continuous write access is also not good, because the update of the file system journal causes wear on the CF card flash and the internal wear levelling will mark the worn out sectors as defect and copies the contents to some new place until no capacity is free anymore.

Best is to use FAT32 formatted industrial grade CF cards for the Camera and copy these over to industrial grade CF cards which use a journalling file system. Industrial grade CF have a wider temperature range (-30 - +85 degree Celsius) and a much higher MTBF. Buy two or three different CF card readers for USB 2.0 and you should be fine for the next 10 years, as USB will stay at least for that time.

As for price per GB: High reliability costs more and is more inconvenient, but you get what you pay for.

Title: Store Images On Flash Cards?
Post by: dwdallam on January 04, 2009, 12:51:27 am
Quote from: pom
Tell me more, seriously...

BTW I hold you personally responsible (the CS4 thread) for me upgrading to 64 bit, buying 8GB more of RAM and now I've made a striped disk just for scratch/page file/Bridge cache. Appreciate your advice there.  


Probably won't work--hehehe. Really though, it is much more crisp and tight feeling. Plus you get the new ACR with all the tools LR2 has, except you don;t get dual up windows, before and after live view with ACR. Let us know what you think.

Newsgroups have been around I think before the "web" actually. They were around before AOL first launched. I would google "Newsgroups wiki" and read teh wiki file on them. It's thorough.
LINK: http://en.wikipedia.org/wiki/Newsgroup (http://en.wikipedia.org/wiki/Newsgroup)
 
Here is an excerpt:

How newsgroups work

Newsgroup servers are hosted by various organizations and institutions. Most Internet service providers host their own news servers, or rent access to one, for their subscribers. There are also a number of companies who sell access to premium news servers.

Every host of a news server maintains agreements with other news servers to regularly synchronize. In this way news servers form a network. When a user posts to one news server, the message is stored locally. That server then shares the message with the servers that are connected to it if both carry the newsgroup, and from those servers to servers that they are connected to, and so on. For newsgroups that are not widely carried, sometimes a carrier group is used for crossposting to aid distribution. This is typically only useful for groups that have been removed or newer alt.* groups. Crossposts between hierarchies, outside of the Big 8 and alt.* hierarchies, are failure prone.

So you can see that your information will be stored on millions of servers for up to 3+ months at a time. Find a newsgroup that isn't active, like alt.pears.salad and upload your files using RAR split into chunks of 100MG each. Password protect the split file so no one can get into it. Uploading to Newsgroups is an art, so you'll need to understand how it works. There are programs that can do all the preparatory crap for you. The problem is that New servers don't allow you to upload a huge file. They want it broken down into smaller files, and the size is relatively small for each chunk. With large files you literally end up with 1, 000s of small chunks, but using a news reader, you don't see all of that./ You just see the entire file. Anyway, good luck on that :0

You can, however, use the news servers (hosts) for your own file storage because there are literally no limits on newsgroups. As long as you are posting to a group that allows file uploads, you're ok. It's pretty much the wild wild west and last area that is totally unmonitored and "anything goes." Oh boy, you find some real winners on that format. Many of them so vitriolic that they can;t post anywhere else on the internet, so they find news  groups to vent their hate and ignorance.

On the other hand, you can get some really good information that you will not get ANYWHERE else. A good example is war footage from Iraq. If you live in the US, our current administration has issued a black out of "negative" reporting on the war. So if you want to see what it's really like in Iraq when an IUD goes off and cremates a Bradley, you can find that stuff on Newsgroups--uncut, uncensored, etc. I don't remember the name of the American that had his head decapitated on TV, as he was sitting there under several terrorist, but of course you will never see any of that in the US. I saw the entire captioned file footage after finding the video uncensored on Usenet. It brought a whole new meaning to the word "violence" to my vocabulary, among other things, since it was no movie and very visceral . Once things states don't want in public view gets to the Usenet, it's there forever pretty much, as long as people keep "upping it" and your country doesn't start censoring information on the internet--such as China does by blocking specific pathways.  In that sense, it's a really good service. Also, many corporations use Newgroups instead of or in addition too forums, since they are free and have no maintenance fee associated with them--or bandwidth fees. In fact, Adobe uses them, as does MS.
Title: Store Images On Flash Cards?
Post by: dwdallam on January 04, 2009, 01:10:47 am
Quote from: jjj
Ah, but you forget that when people shoot digital, they will shoot a lot more photos than when using slide. A lot more.
I spent a couple of week in China, just before I switched to digital and shot just over 40 rolls of film.  Approximately 1600 images [and one battery would last almost exactly 40 rolls]. That was considered a lot of film.
People casually shoot a thousand pictures in a day now!
I also had a Canon Ixus 2.1M camera with a 'huge' and quite expensive 256M card and made it and the 64M card last the entire trip!
I've done jobs that I could not have afforded to do on film, as there was so much to shoot and for extended periods of time. I've even shot for 24hrs straight, with a different model or models every 20mins - 84 in total I think.

I gave up on DVDs for back up a long time ago as 4G was simply too small and slow to bother with. So compact flash would be very expensive a proposition. Last big trip I had 2x320G and 1x500G laptop drives with me.


Use Blueray disks. They hold 50GB.
Title: Store Images On Flash Cards?
Post by: dwdallam on January 04, 2009, 01:15:47 am
Quote from: lumpidu
I am in the embedded computer business and we use CF cards as a reliable data storage for our data. We guarantee our customers 10 years lifetime for our products. We are using industrial grade CF cards which also have internal capacitors to survice a sudden power loss, so these are able to store what is already in the internal buffers.

IMHO what is the main problem with CF cards is flash wear / wear levelling and the used file system. FAT16 or FAT32 don't use any journalling, so these file systems can get corrupted if unconnected from the computer unexpectedly. Formattting CF cards with an appropriate file system is key for reliable backup. Mac OS X has AFS, Linux has ext3, Windows has NTFS. These are reliable file systems. Used on a rugged industrial grade CF card, these should be safe for almost 10 years.

But then again using these file systems for continuous write access is also not good, because the update of the file system journal causes wear on the CF card flash and the internal wear levelling will mark the worn out sectors as defect and copies the contents to some new place until no capacity is free anymore.

Best is to use FAT32 formatted industrial grade CF cards for the Camera and copy these over to industrial grade CF cards which use a journalling file system. Industrial grade CF have a wider temperature range (-30 - +85 degree Celsius) and a much higher MTBF. Buy two or three different CF card readers for USB 2.0 and you should be fine for the next 10 years, as USB will stay at least for that time.

As for price per GB: High reliability costs more and is more inconvenient, but you get what you pay for.


What about the new SSDs coming out now? Would they offer the same type of reliability for storage? And again, why not just use hard drives? Buy a few 120GB 2.5 inch laptop drives, copy the files, and put them in a small fireproof, waterproof safe? 120Gb for like 50 bucks.
Title: Store Images On Flash Cards?
Post by: jjj on January 04, 2009, 06:04:51 pm
Quote from: dwdallam
Use Blueray disks. They hold 50GB.
I was talking about travelling - Not so good then as laptop doesn't have a BluRay burner and I'm not going to bother with the bulk of of an external one, when I want to reduce weight not increase it. My next laptop will have a BluRay burner, depends if Apple add them, if not and they stick with the stupid mirrors they currently have for screens, it'll be a PC laptop. The other thing is the slowness of burning data, as opposed to copying it to a hard drive.