Topic: Archiving - Help Also retrieving Old RAW formats (Read 5595 times)

kyle · « **on:** April 18, 2006, 06:44:05 am »

I'm interested in what members are doing these days in achiving their shoots.

With cameras like the 1Ds2, 5D and MF digital backs even archiving on to DVD's is causing problems. It's not unrealistic tio have 1TB (1000 * 1GB) or more and if you want to save .PSD (the finished image) as well then you'll need tons and tons of storage.

Obviously these days first level archiving can be done on external Hard Disks as they are cheap enough (350GB external HD less than 100 GBP / 170 USD).

However what is the best method of LONG TERM storage, Off site storage and methods of retrieval.

Also if storing in the original RAW format how does one retrieve pics when the old camera RAW is no longer supported - as I believe some early DSLR's are no longer supported by CS2 or even the original Camera manufacturers software.

I'm sure this issue has been put off for a long time by a lot of shooters but I'm getting to a stage where meddling around with EXCEL and searching through loads of Disks isn't good enough anymore.

I don't care if it's Windows or Mac based but I'm sure this is definitely a Non Trivial topic.

Thanks

-K

boku · « **Reply #1 on:** April 18, 2006, 07:33:10 am »

1) I now convert all my various raw files to DNG format for posterity.
2) My main archive is a 1 TB Buffalo NAS setup for 1 TB (no Raid).
3) I backup to 2 sets of offline USB hard drives (8 Lacie 250 GB). One set is stored offsite.
4) I have abandoned DVD backup altogether.
5) I am a casual shooter. If I were a busy pro I would need more storage capacity, but the methods would still be the same.
6) In general, I suggest the book "The DAM Book" for good up-to-date advice.

Lisa Nikodym · « **Reply #2 on:** April 18, 2006, 12:06:00 pm »

My method:

Convert RAWs to DNGs (same reason as boku). Have RAWs, DNGs and final post-processed TIFFs on my working hard drive.

Periodically copy everything on my working hard drive to a portable hard drive, which is usually kept at a different location. (This is mostly in case my working hard drive suddenly fails before I've finished working with a set of pictures.)

When I finish a set of pictures (for example, for one trip), I write all images associated with that set (RAWs, DNGs, & TIFFs) to two (for redundancy) gold DVDs, label them with a felt tip pen made especially for writing on CDs & DVDs, and store them in a binder. This is my permanent archive. It is certainly reasonable to write them to a spare hard drive instead of DVDs, if that's what you prefer.

Lisa

kyle · « **Reply #3 on:** April 19, 2006, 04:28:24 am »

Discs IMO are a temporary "Holding" solution. We do need something like small optical permanent media for archiving.

Space (physical) is now the major constraint in using current size DVD's.
1TB (1000 * 1GB) of pics will require 250 DVD's (approx at 4.7GB per DVD). So in a few years one could have literally 1000's of DVD's.

Cataloguing, organising and storing these is likely to become almost a full time job in itself.
I'm reluctant to consider new DVD formats (BLU-RAY etc) as these are likely to both be incompatable with existing DVD formats and themselves likely to supeceeded by something else.
IMO even a 30GB DVD isn't a sufficient improvement from existing DVD formats to make a significant difference.

I'd consider DVD's again once the capacity nudges above 100GB per disc especially if double sided at 100GB per side - I could tolerate 10 DVD's per TB or even better if double sided 5 DVD's per TB.

Anybody using that stalwart of "Dinosaur Technology" - Tapes. I know the've been around since "Pontius was a Pilot" but high capacity tapes could be useful for archiving. Banks and other organisations requiring huge archives are still using these AFAIK.
I know it's old technology but it DOES seem to offer limitless data storage in a small physical space (even though data retrieval will be more complex).

External Hard disks suffer from the fact that they can be accidentally (or maliciously) erased as well so any archive will have to be duplicated (like any otrher medium).
Who would have thought that just average photographers were having to worry about managing a few TB of storage. (My ist computer had a hard disc of only 20MB - yes MB and at that time I thought that was plenty. I hadn't yet started usining computers for image processing then of course.

Cheers

-K

Julian Love · « **Reply #4 on:** April 19, 2006, 05:50:49 am »

I too find DVDs too small and time consuming for long term storage. I store everything on a series of 250GB external hard disks, one set for primary storage and another set for backup - so everything is stored on two disks for redundancy (poor man's RAID5). As I need more storage I just add more disks. Currently at 1TB. The second set of disks are not at a separate location, which is a weakness at the moment.

I do not yet convert all my RAWs into DNG, as I am not yet convinced of the longevity of the DNG format. Saving both (RAW and DNG) as nniko does is probably best, I may do that soon.

Julian

bruce fraser · « **Reply #5 on:** April 19, 2006, 12:51:20 pm »

Quote

I do not yet convert all my RAWs into DNG, as I am not yet convinced of the longevity of the DNG format. [a href=\"index.php?act=findpost&pid=63053\"][{POST_SNAPBACK}][/a]

The difference between DNG and all other raw formats is that you can download the file format specs for DNG. Armed with that knowledge, any reasonably good programmer can write a DNG converter. With any other format, you're relying on the camera vendor exclusively. Since I have the DNG spec archived with my DNGs, I'm not at all worried about its longevity. I am, however, worried about the longevity of support from camera vendors for older cameras. It's only thanks to Tom Knoll and Camera Raw that I can read my Kodak DCS 460 files on an OS from this millenium. I wonder how much longer Konica/Minolta will support their cameras?

DarkPenguin · « **Reply #6 on:** April 19, 2006, 01:05:07 pm »

Didn't they just release a SDK for DNG just the other day? I haven't downloaded it so I'm not sure what is in it.

bruce fraser · « **Reply #7 on:** April 19, 2006, 04:26:47 pm »

Quote

Didn't they just release a SDK for DNG just the other day? I haven't downloaded it so I'm not sure what is in it.
[{POST_SNAPBACK}][/a]

[a href=\"http://www.adobe.com/support/downloads/dng/dng_sdk.html]http://www.adobe.com/support/downloads/dng/dng_sdk.html[/url]

kjkahn · « **Reply #8 on:** April 20, 2006, 05:45:57 pm »

Quote

The difference between DNG and all other raw formats is that you can download the file format specs for DNG. Armed with that knowledge, any reasonably good programmer can write a DNG converter. With any other format, you're relying on the camera vendor exclusively. Since I have the DNG spec archived with my DNGs, I'm not at all worried about its longevity. I am, however, worried about the longevity of support from camera vendors for older cameras. It's only thanks to Tom Knoll and Camera Raw that I can read my Kodak DCS 460 files on an OS from this millenium. I wonder how much longer Konica/Minolta will support their cameras?
[a href=\"index.php?act=findpost&pid=63082\"][{POST_SNAPBACK}][/a]

As I mentioned in another thread, there's a reason for Canon shooters not to discard CR2 files. I converted some of mine to DNG, and then read messages on RG forums from guys who thought that DPP 2.0 does better conversions than ACR. I tried it and became a convert, but my original RAW files were gone. It is possible to embed the CR2 files in DNG files, but at the cost of increased file size. I really prefer the interface of ACR, but image quality comes first. If Adobe can get ACR to extract as good images from DNG files as DPP does from CR2 files, I'll switch back to ACR in a heartbeat.

Another advantage for ACR is that I have more confidence in what I am doing, thanks to your great book.

Ken

jani · « **Reply #9 on:** April 20, 2006, 05:46:11 pm »

Perhaps we're whining a bit too much about this long term storage problem.

How on earth did film photographers manage to store tens of thousands of rolls of film?

DarkPenguin · « **Reply #10 on:** April 20, 2006, 06:00:17 pm »

You couldn't delete 10,000 rolls of film with 2 keystrokes.

Lisa Nikodym · « **Reply #11 on:** April 20, 2006, 06:17:30 pm »

Quote

You couldn't delete 10,000 rolls of film with 2 keystrokes.

That's what the DVDs are for!

And on the flip side, as long as you can read your digital archive, the images won't have degraded with age. Film, on the other hand, changes color and degrades slowly over the years.

Lisa

kyle · « **Reply #12 on:** April 22, 2006, 05:14:16 am »

Quote

Perhaps we're whining a bit too much about this long term storage problem.

How on earth did film photographers manage to store tens of thousands of rolls of film?
[a href=\"index.php?act=findpost&pid=63197\"][{POST_SNAPBACK}][/a]

I'm sure we shoot MUCH more now with digital. I'm sure in the old film days I'd never have processed several hundred pics over a weekend.

Wiht a 1Ds2 or even a 1D2 it's easy to get several GB's worth of RAW images - it's quite easy to shoot say 300 pics. OK not so many will be keepers but they all need to be reviewed, sorted, stored etc.

Obvious No hopers can be discarded but on a busy PJ shoot you don't always have the time to review images during the shoot.

Cheers

-K

Author Topic: Archiving - Help Also retrieving Old RAW formats (Read 5595 times)

Julian Love

DarkPenguin

DarkPenguin