Pages: [1]   Go Down

Author Topic: Workflow: file storage  (Read 6399 times)

tshort

  • Newbie
  • *
  • Offline Offline
  • Posts: 43
Workflow: file storage
« on: January 02, 2006, 05:23:22 pm »

I've been working on my workflow and am getting it to the point where it is pretty consistent, at least on the front end.  Main steps are:

1. download raw files (only thing I shoot)  from CF to folder, "incoming"
2. quick edit - cull out unwanted shots
3. move into a pre-archive folder on c:\ (naming is yyyy-nn, 2005-01...)
4. make lo-res .jpg copy of all files (I use Irfan), store in same pre-archive folder
5. when pre-archive folder hits 600MB, dual backup to both CD and external HDD
6. leave pre-archive on c:\, but only with lo res .jpg - all raws deleted

What I haven't yet figured out is what to do with the raws that I decide to work on and process further.  These turn into large psd files in a hurry, and I'm wondering what the best thing to do with them is.  Right now I'm keeping them in a folder called My Work, that is growing large.
If I use a similar strategy for these as I do for my raws, I'll end up with two parallel archives systems, the 2005-nn one, and a new one for My Work.  And if I drop the psd files into the pre-archive folder, I may end up with psd files in a different archive from the raw that they are based on.

Any thoughts?

Thanks.
Logged
-T
Wisconsin

Jonathan Wienke

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5829
    • http://visual-vacations.com/
Workflow: file storage
« Reply #1 on: January 02, 2006, 06:45:30 pm »

Here's my workflow (99.9% RAW):

1. Copy all files from all bodies used in the shoot into a single folder named YYYY-MM-DD Event Name (like Office Christmas Party, Alpha Phi Sorority Yearbook, etc. This gives me the ability to find most photos with a simple directory name search.)
2. Quick cull to delete lens cap interiors, etc.
3. Sort files by date/time (all camera clocks are synced periodically)
4. Rename all files YYYY-MM-DD_XXXX where XXXX is a 4-digit counter
5. Create a subfolder called \selects where any PSD work files go
6. Create another subfolder called \jpg for web preview or proof print JPEGs

CDs are too small to be useful. Get a couple of NAS devices or hard drives in USB enclosures instead. You're much more likely to be able to access your data 10 years from now than if backing up to a CD-R; most CD-Rs are NOT at all archival.

I have over 110,000 frames in my archives, and over 2 terabytes of storage. I can find almost anything I've ever shot in a minute or two.
Logged

tshort

  • Newbie
  • *
  • Offline Offline
  • Posts: 43
Workflow: file storage
« Reply #2 on: January 04, 2006, 05:34:22 pm »

Thanks, Jonathan.  Seems like a good approach.  Will have to look into some of the HDD options you mentioned.  My current one makes me nervous - lots of creaks and squeals.
Logged
-T
Wisconsin

Tim Gray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2002
    • http://www.timgrayphotography.com
Workflow: file storage
« Reply #3 on: January 04, 2006, 05:46:36 pm »

I second the wisdom of using a pair of mirrored USB drives as archives.  I've had CD's with unrecoverable files, and the problem is that since they are prone to degradation over time you have to check the whole pile of them periodically.
Logged

monik

  • Newbie
  • *
  • Offline Offline
  • Posts: 37
Workflow: file storage
« Reply #4 on: January 16, 2006, 11:25:14 am »

Thanks Jonathan for this interesting information, what hard drives would you recommend?  So far I only have a PC and save on CD or DVD, I am planning to buy an Epson 2000 to download cards when I am in the field but I know I need extra storage at home for safety.  A laptop is not on the cards at the moment.
Grateful for your opinion,
Monique
Logged

TimothyFarrar

  • Newbie
  • *
  • Offline Offline
  • Posts: 49
Workflow: file storage
« Reply #5 on: January 16, 2006, 07:55:26 pm »

I also use the mirrored drives approach. Except as a Mac user I use Firewire drives because a lot of macs only have USB1 support which is really slow (USB2 is about the same speed as firewire).

If you always keep the files on a pair of drives, this reduces your risk of loss to having both drives go out at the same time. So if both drives are perminately plugged into the same surge protector and it overloads from the building getting hit by lightning, you might lose all your files (this 1/10000000... chance is why I don't keep the drives plugged in).

The drive technology is changing so fast that it is really hard to tell what drives are better than others for reliability. Performance is easy, look at seek time and drive cache size. Since backup performance is rairly the issue, I go with the most economical ($/GB) at the time of purchase. Then phase out older drives over the years as they become too small to be useful.

And most importantly, if you get signs that a drive might fail (a new clicking or grinding noise, odd warning messages, random corrupt files), toss the drive, buy a new one and copy the files off of the other mirrored drive (not the drive that is getting tossed). You don't want the possibility of copying off a corrupt backup of a key file.

- Timothy Farrar : farrarfocus.com
Logged
Timothy Farrar
Farrar Focus Digit

DarkPenguin

  • Guest
Workflow: file storage
« Reply #6 on: January 16, 2006, 07:58:15 pm »

Quote
Thanks Jonathan for this interesting information, what hard drives would you recommend?  So far I only have a PC and save on CD or DVD, I am planning to buy an Epson 2000 to download cards when I am in the field but I know I need extra storage at home for safety.  A laptop is not on the cards at the moment.
Grateful for your opinion,
Monique
[a href=\"index.php?act=findpost&pid=56062\"][{POST_SNAPBACK}][/a]

I think Jonathan has a good month left in basic training.  So it might be a bit before he can respond.
Logged

61Dynamic

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1442
    • http://
Workflow: file storage
« Reply #7 on: January 16, 2006, 09:16:15 pm »

The file management workflow I've developed for myself is almost identical to Jonathan's. It's simple, quick, easy to manage and, as Jonathan's 110K files indicates, expandable.

Quote
I also use the mirrored drives approach. Except as a Mac user I use Firewire drives because a lot of macs only have USB1 support which is really slow (USB2 is about the same speed as firewire).
[{POST_SNAPBACK}][/a]

USB2 in actual use is slower than FireWire. FW has the advantage of maintaining a constant data transfer rate wherein USB2 fluctuates in speed. If you want to have external drives that perform their best, FW is generally the way to go. For PC users, a FW PCI expansion card can be had for a measly $24.

The drive enclosure can have an effect on performance as well so don't buy a cheap POS. Stick with a recognized brand.

Quote
If you always keep the files on a pair of drives, this reduces your risk of loss to having both drives go out at the same time. So if both drives are perminately plugged into the same surge protector and it overloads from the building getting hit by lightning, you might lose all your files (this 1/10000000... chance is why I don't keep the drives plugged in).[a href=\"index.php?act=findpost&pid=56086\"][{POST_SNAPBACK}][/a]

Backups are important. Being a Mac guy, I have my files stored on an internal drive which is automatically cloned to an external FW drive every night via Applescript/rsync. PC users have similar options available, but [a href=\"http://www.karenware.com/powertools/ptreplicator.asp]third party software[/url] is required.

Forget surge protectors. I think battery backup solution (UPS) such as an APS or a Tripp Lite are a must. They do more than a surge protector by protecting from surges, spikes, voltage dips, noise and outages. They're cheep insurance for protecting your computer (they come with $15K to $200K equipment replacement warranties) and drives.

Quote
And most importantly, if you get signs that a drive might fail (a new clicking or grinding noise, odd warning messages, random corrupt files)

- Timothy Farrar : farrarfocus.com
[{POST_SNAPBACK}][/a]
Emphasis added. Those last two points are software failures and can be fixed in most cases with drive recovery software such as [a href=\"http://www.grc.com/sr/spinrite.htm]SpinRite[/url] (stay the hell away from anything made by Norton!).

Quote
hanks Jonathan for this interesting information, what hard drives would you recommend?
I'd primarily recommend Hitachi (formerly IBM) and Seagate.

Many people hate Maxtor claiming high drive failures but I'm skeptical how legit those concerns are (in terms of overall reliability, not wether or not someone actually lost a drive). Depending who you talk to, you can get the same horror story for any of the companies. It's a moot point anymore as Seagate have recently purchased Maxtor. Western Digitals are good with their Raptor line for speed.

Stick to those name-brands and maintain constant backups and you should be golden.
Logged

TimothyFarrar

  • Newbie
  • *
  • Offline Offline
  • Posts: 49
Workflow: file storage
« Reply #8 on: January 17, 2006, 11:23:16 am »

I probably should have been a little more specific in my previous post, as Daniel pointed out on a bunch of my points.

In the case of odd warning messages I ment something like the following,

'Warning, SMART has detected that a hard drive failure may be imminent. Please back up your data immediately.' which would be something seen on booting of a PC.

Basically each harddrive has a pool of spare drive sectors that are remapped in place of sectors that go bad. This happens inside the harddrive itself. If the rate of sectors that are going bad increases or the number of spare sectors approaches zero, then on some drives (SMART drives) you will end up with a message as seen above.

Note that the drive might still pass a drive check because the bad sectors were remapped from the drive, so all data could be on good sectors.

There is a small chance that random corrupt files could be caused by not shutting down the computer properly, however with a modern Journaled filesystem, which I believe is standard on Mac OS X, the journaling system would be used to restore the drive to the un-corrupted state after a dirty power out. In most cases what is happening is the sectors that map to your files are going bad.

From my experience in the past 8 years I have had a few drives go bad, such as the drive in my G4 PowerBook. In each case the drives ran fine up the point where I got a few serious file corruptions, then shortly after things got worse. Sometimes you can "fix" the drive with software (it uses software to remap files off of bad sectors, unlike the hardware approach done inside modern drives), and it will probably work ok for a while, but the drive has given its sign that I might fail soon in the future. I've seen this go both ways, sometimes the drive is good for another year, sometimes a few days later the drive goes.

The point I was tring to make was, sometimes is it better to be safe than sorry.
Logged
Timothy Farrar
Farrar Focus Digit

monik

  • Newbie
  • *
  • Offline Offline
  • Posts: 37
Workflow: file storage
« Reply #9 on: January 17, 2006, 01:12:58 pm »

Thanks Tim, this is thorough and very helpful; I certainly agree that it is better to be safe than sorry.  Will start looking into hard drive purchase. both portable and for home use.  
Regards,
Monique
Logged

photopat

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
    • http://
Workflow: file storage
« Reply #10 on: January 17, 2006, 02:08:13 pm »

I went for a raid 5 array as I think it's the best combo of saftey and storage usage.
75% of total disk usage instead of 50% of total disk usage when setting up a raid 1 (mirrored)
Logged

monik

  • Newbie
  • *
  • Offline Offline
  • Posts: 37
Workflow: file storage
« Reply #11 on: January 18, 2006, 11:31:19 am »

Thanks for the info, I am still quite unclear about some points - how easy is it to fit in an external hard drive, I don't want to have to do anything else apart from plugging in to a usb, I am not very technical...
Where can I read a bit more to understand all these new things?  Is there a book,  a comprehensive website or other?
For holidays is an Epson 2000 with 40/80 gb a good idea or is a portable hard drive possible without a laptop?  Do I need both?
Sorry to display such ignorance and thanks for your help and patience,
Monique
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Workflow: file storage
« Reply #12 on: January 18, 2006, 03:42:43 pm »

Quote
Thanks for the info, I am still quite unclear about some points - how easy is it to fit in an external hard drive, I don't want to have to do anything else apart from plugging in to a usb, I am not very technical...
An external hard drive is very easy to use. You need only an available power outlet and a USB (preferably high-speed USB, often incorrectly referred to as "USB 2") or FireWire (IEEE 1394) connection to your computer.

Recent operating systems will recognize this as an external drive with just a brief waiting time (some seconds).

Quote
Where can I read a bit more to understand all these new things?  Is there a book,  a comprehensive website or other?
Since you're wondering about several things, I suggest you look at HowStuffWorks.com; they have well-written and easy to understand articles on many, many topics.

Quote
For holidays is an Epson 2000 with 40/80 gb a good idea or is a portable hard drive possible without a laptop?  Do I need both?
The Epson P2000 or P4000 connect via USB to your computer, so it doesn't matter if your computer is a laptop or a desktop beast.

The P2000 or P4000 are great for reviewing pictures when you don't have a laptop available.
Logged
Jan

monik

  • Newbie
  • *
  • Offline Offline
  • Posts: 37
Workflow: file storage
« Reply #13 on: January 18, 2006, 05:11:16 pm »

Jani,
thanks very much, for all this precise information, I think I know what to do now and will read the site you mention first of all, then I will choose a hard drive and probably buy the Epson soon as I saved a bit in recent months.  Good luck in your photography.  Regards,
Monik
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
Workflow: file storage
« Reply #14 on: January 18, 2006, 05:34:01 pm »

You're welcome, Monique.
Logged
Jan
Pages: [1]   Go Up