Pages: [1] 2 3   Go Down

Author Topic: How do you back up your massive files?  (Read 10373 times)

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« on: May 04, 2010, 03:00:12 pm »

probably a repost, but always a good topic.

first, I have the structure in my computer(s):
year\year-month number-Month Name\year-month-day - location\band,group,person

so an example
MyPhotos\2010\2010-04-April\2010-04-23 - nemhf day one\mainstage\after_the_burial\after_the_burial001.ccr.jpg
MyPhotos\2010\2010-04-April\2010-04-23 - nemhf day one\mainstage\after_the_burial\after_the_burial002.jpg
...
etc.

I synch them up using beyond compare with
1) a drobo (5TB) that sits at my work
2) a QANP 639pro (12TB) that sits in my house
3) a couple drives that stay turned off at home
4) my desktop
5) my laptop

I keep 1/2-1 year on my desktop and 4 months on my laptop (both are backed up nightly with WHS)
Before I delete them from my photos from my laptop/desktop, I will byte-wise compare the QNAP and the drobo.

I also have my webservers backup to the QNAP which rules.

what is your process?  Someone I know from another forum has 4 4xdrobos and uses time machine back ups from each to each keeping a bunch of giant snap shots.

Steve Hendrix

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1662
    • http://www.captureintegration.com/
How do you back up your massive files?
« Reply #1 on: May 04, 2010, 05:01:22 pm »

Quote from: revaaron
probably a repost, but always a good topic.

first, I have the structure in my computer(s):
year\year-month number-Month Name\year-month-day - location\band,group,person

so an example
MyPhotos\2010\2010-04-April\2010-04-23 - nemhf day one\mainstage\after_the_burial\after_the_burial001.ccr.jpg
MyPhotos\2010\2010-04-April\2010-04-23 - nemhf day one\mainstage\after_the_burial\after_the_burial002.jpg
...
etc.

I synch them up using beyond compare with
1) a drobo (5TB) that sits at my work
2) a QANP 639pro (12TB) that sits in my house
3) a couple drives that stay turned off at home
4) my desktop
5) my laptop

I keep 1/2-1 year on my desktop and 4 months on my laptop (both are backed up nightly with WHS)
Before I delete them from my photos from my laptop/desktop, I will byte-wise compare the QNAP and the drobo.

I also have my webservers backup to the QNAP which rules.

what is your process?  Someone I know from another forum has 4 4xdrobos and uses time machine back ups from each to each keeping a bunch of giant snap shots.


Without getting into naming convention, automation options, etc.

From an equipment and strategy perspective, but putting aside for the moment the brand and type of devices used, the minimum standards for effective backup would be:

*a clarified data protection strategy that encompasses the following:
- redundancy of data
- simplicity of the backup process
- offsite strategy
- consistent execution

I work with photographers that have a very wide range of data capacity requirements. The amount of data that needs protecting is a crucial component of the strategy itself. The biggest problem I see with clients is that they are in a short term, high device buying cycle. Meaning, they fill up storage rapidly and add devices as they go. In many cases this makes an effective backup strategy difficult and complicated. Frequently I have found clients without any redundancy at all!

When I consult with photographers on data protection, some of the important questions are:

*how much data do you currently have?
*how much do you anticipate adding over the next 36 months?
*if the amount of data is high, are you able to discern critical data vs data that is non-critical or will be retired after a certain time period (this helps determine an offsite strategy)

I am a big proponent of RAID because the advantage over individual high capacity hard drives is much higher capacity and read/write speed (many clients use RAID as a working drive). It is very difficult to backup data from 8 or 12 individual hard drives. Right now individual hard drives top out at 2TB capacity. Non server-based RAID solutions start at 4TB's and can scale to at least 32TB's (with the solutions we sell). We try to move content producers into a (roughly) 36 month device buying cycle and they can typically do this with RAID technology because of the capacity.

This makes it much simpler for a photographer to implement a strategy. By projecting out 36 months and buying a solution that with one device/interface provides capacity for all the current and future content over the next 36 months (and duplicating that at least once with a like device for redundancy), the backup process becomes very simple, very easy. As a result, an end user is much more likely to consistently maintain the backup process.

A solution like this costs a bit more than buying individual hard drives, but reliability and performance are way higher. And the backup workflow is no longer a struggle.

I do have opinions about the equipment as well, but first, the strategy is the key.

Revaaron, you have invested the time and money to protect your data with high capacity RAID, an offsite component, and thought on the term of the storage for some of the data.  So you are ahead of the game when compared to many.

My advice - don't be sorry. If you do not have a strategy that you can easily and consistently and sustainably employ, implement one today that changes all that for you and eliminates the precarious position your data is currently in.


Steve Hendrix

Logged
Steve Hendrix • 404-543-8475 www.captureintegration.com (e-mail Me)
Phase One | Leaf | Leica | Alpa | Cambo | Sinar | Arca Swiss

bdp

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 266
    • http://www.bendearnleyphotography.com
How do you back up your massive files?
« Reply #2 on: May 04, 2010, 06:14:10 pm »

my naming is similar - year, month, day, client so it all sorts nicely by default with the last job at the bottom of a list sorted alphabetically. eg 20100505 AGT would be today's job.

I use two Drobos - one in the studio and one at home, and ferry data home every fortnight or so on an external drive. I have 6TB worth of drives in each Drobo which gives me 3.62TB on each for redundant backup. I also use time machine here at the studio for hourly backups in case something goes bad. Worst case is I lose one or two shots (setups, motifs etc) on that day's shoot. However my Drobos are filling up fast - I only have 586GB available after putting 5-6 years worth of digital jobs on it so will either have to put more 2TB drives in them or step up to the big boys that take 8 drives each. Not looking forward to that expense. The great thing about having everything in one place instead of individual hard drives in a cupboard is when a client asks for an image from 4 years ago, and I can easily search for it. Cataloging is also easy and searching for pictures based on file name or metadata is also easier if it's all in one place. I actually just let spotlight catalog it all then can search by client (folder) name or year or filename (usually a description of the shot) or a combination of all of these. It's not as good as something like Portfolio, but much less work.

I suppose it's not realistic to expect to keep all my jobs in one place like this forever, unless we get 100TB SSD drives or something like that in the near future. I may have to put the drives from my drobos on a shelf one day (carefully labelled!) and start again with a bunch of new empty drives and keep going from that point. A bit of a hassle if I want to go back to old data to remove 4 drives and shove in the 4 older ones to get to it.

Ben
« Last Edit: May 04, 2010, 06:34:18 pm by bdp »
Logged

sergio

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 666
    • http://www.sergiobartelsman.com
How do you back up your massive files?
« Reply #3 on: May 04, 2010, 08:38:50 pm »

Mine is very simple. I buy 2 identical HDs and start filling #1 with images that I import in LR and I have in the catalog. I then duplicate it to hd #2 and after that each time I do something to #1 I use carbon copy cloner app to make an incremental backup to #2. When I fill those up I buy other 2 HDs and start over. They are named in a consecutive manner and they live offline in LR, but I can still see what each contains, so all I have to do is plug the one I need and access the images. Pretty simple, cost effective and it works as advertised.

Logged

Kitty

  • Full Member
  • ***
  • Offline Offline
  • Posts: 126
    • http://
How do you back up your massive files?
« Reply #4 on: May 04, 2010, 09:29:09 pm »

After massive backup hardware, how do you catalog all files?
What is the best software on mac to catalog and search these back up files?
Logged

bdp

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 266
    • http://www.bendearnleyphotography.com
How do you back up your massive files?
« Reply #5 on: May 04, 2010, 10:23:13 pm »

Quote from: Kitty
After massive backup hardware, how do you catalog all files?
What is the best software on mac to catalog and search these back up files?

Extensis Portfolio
Logged

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #6 on: May 04, 2010, 11:01:47 pm »

Quote from: sergio
Mine is very simple. I buy 2 identical HDs and start filling #1 with images that I import in LR and I have in the catalog. I then duplicate it to hd #2 and after that each time I do something to #1 I use carbon copy cloner app to make an incremental backup to #2. When I fill those up I buy other 2 HDs and start over. They are named in a consecutive manner and they live offline in LR, but I can still see what each contains, so all I have to do is plug the one I need and access the images. Pretty simple, cost effective and it works as advertised.

I did it that way for YEARS. except I would have 3 hard drives with one sitting at work.  it got too hard to manage with (at that time) all these 500GB drives sitting around.

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #7 on: May 04, 2010, 11:09:00 pm »

Quote from: bdp
I suppose it's not realistic to expect to keep all my jobs in one place like this forever, unless we get 100TB SSD drives or something like that in the near future. I may have to put the drives from my drobos on a shelf one day (carefully labelled!) and start again with a bunch of new empty drives and keep going from that point. A bit of a hassle if I want to go back to old data to remove 4 drives and shove in the 4 older ones to get to it.

cloud storage at home. This is an awesome article about it:
http://blog.backblaze.com/2009/09/01/petab...-cloud-storage/

the only issue I have with it is this "ST31500341AS". That makes me not want to post this article since I bought 6 of those drives and have gone through 13 of them getting them replaced by seagate and I currently have 3 of them sitting here dead.

Guy Mancuso

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1133
    • http://forum.getdpi.com/forum/index.php
How do you back up your massive files?
« Reply #8 on: May 05, 2010, 12:10:54 am »

Mine is simple. Use a Drobo than to back that up I will take a 1 TB drive and put all raws in it and another and put all final Tif files in it date it than repeat as i go along. I put these bare drives off site. Also Drobo has 4 2 tb drives in once filled I store those drives as a group.

Naming Client/job date month, date, year

searstractors05032010 example of course


I try to keep it very simple.

Also any DVD burned for a client a backup is made as well for the client and me.
« Last Edit: May 05, 2010, 12:11:50 am by Guy Mancuso »
Logged
[url=http://forum.getdpi.com/forum/showt

K.C.

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 671
How do you back up your massive files?
« Reply #9 on: May 05, 2010, 01:08:38 am »

Quote from: revaaron
I keep 1/2-1 year on my desktop and ...

Take them off your desktop and put them in a logical location on the drive. No matter what OS you're running you're wasting system memory by having it manage the files from the desktop.
Logged

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #10 on: May 05, 2010, 11:39:07 am »

just an FYI for you guys, 2TB @ new egg this saturday for $120
http://slickdeals.net/forums/showthread.php?threadid=2010708

semillerimages

  • Full Member
  • ***
  • Offline Offline
  • Posts: 163
    • http://www.semillerimages.com
How do you back up your massive files?
« Reply #11 on: May 05, 2010, 12:19:54 pm »


Hey Rev, thanks so much for this link, absolutely fascinating! Love that they list the parts they use too, gives me a better idea about which sata cards are best

Cheers!

*steve

Quote from: revaaron
cloud storage at home. This is an awesome article about it:
http://blog.backblaze.com/2009/09/01/petab...-cloud-storage/

the only issue I have with it is this "ST31500341AS". That makes me not want to post this article since I bought 6 of those drives and have gone through 13 of them getting them replaced by seagate and I currently have 3 of them sitting here dead.
Logged
semillerimages.com

Guy Mancuso

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1133
    • http://forum.getdpi.com/forum/index.php
How do you back up your massive files?
« Reply #12 on: May 05, 2010, 01:53:32 pm »

http://www.newegg.com/Product/Product.aspx...N82E16822136514

I have 4 of these in my Drobo . There quiet, run clean and green. Not the fastest drives but for storage your looking more for reliability than speed.
Logged
[url=http://forum.getdpi.com/forum/showt

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #13 on: May 05, 2010, 02:47:11 pm »

@Guy: those are what I'm swapping through everything in my systems cause of massive failures of my seagate 7200.11 drives. Those are also the ones that are listed as $20 less on saturday.  Also, if you look at the drobo architecture for the 1st, 2nd gens, the hard drives don't matter at all for the speed. The issues with speed of the drobos is that they have a cellphone processor inside dealing with up to 16TB of data (when the 4TB drives come out).

BJNY

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1112
How do you back up your massive files?
« Reply #14 on: May 05, 2010, 03:06:56 pm »

I just helped a friend set up Drobo Pro 16TB with the same WD Green WD20EARS 2TB 64MB Cache hard drives mentioned above.
So far, transfer rates are about 2GB per minute (33MB/s) from LaCie (FW800)
into Drobo Pro connected via iSCSI (Gigabit ethernet).
« Last Edit: May 05, 2010, 03:42:25 pm by BJNY »
Logged
Guillermo

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #15 on: May 05, 2010, 03:12:54 pm »

my 2nd gen drobo gets around 25Mbps connected via firewire, 15Mbps when connected by USB.

I'm either buying a pro or elite for my radio station in 6 months as we transition to more digital storage.

Steve Hendrix

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1662
    • http://www.captureintegration.com/
How do you back up your massive files?
« Reply #16 on: May 05, 2010, 03:16:43 pm »

Quote from: BJNY
I just helped a friend set up Drobo Pro 16TB with the same hard drives mentioned above.
So far, transfer rates are about 2GB per minute (33MB/s) from LaCie (FW800)
into Drobo Pro connected via iSCSI (Gigabit ethernet).


Read or Write?



Steve Hendrix
Logged
Steve Hendrix • 404-543-8475 www.captureintegration.com (e-mail Me)
Phase One | Leaf | Leica | Alpa | Cambo | Sinar | Arca Swiss

BJNY

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1112
How do you back up your massive files?
« Reply #17 on: May 05, 2010, 03:27:24 pm »

Quote from: Steve Hendrix
Read or Write?

Only write speeds matter for the moment
as contents of smaller LaCie hard drives are being copied onto Drobo Pro.

2GB per minute as timed by iPhone stop watch.


For myself, I would be interested in hearing experiences with QNAP units
which cost in the same neighborhood as the Drobo Pro, yet offers faster speed and is stand-alone NAS (with its associated features).
My hesitation being I'm only versed in Mac.
« Last Edit: May 05, 2010, 03:46:44 pm by BJNY »
Logged
Guillermo

Guy Mancuso

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1133
    • http://forum.getdpi.com/forum/index.php
How do you back up your massive files?
« Reply #18 on: May 05, 2010, 03:32:46 pm »

Quote from: revaaron
@Guy: those are what I'm swapping through everything in my systems cause of massive failures of my seagate 7200.11 drives. Those are also the ones that are listed as $20 less on saturday.  Also, if you look at the drobo architecture for the 1st, 2nd gens, the hard drives don't matter at all for the speed. The issues with speed of the drobos is that they have a cellphone processor inside dealing with up to 16TB of data (when the 4TB drives come out).


Yea good choice they are very quiet and efficient. I have I guess the first generation Drobo with 4 drives Firewire 800. I thought about the S unit and go e-sata but now with my new I7 core MBP 15 inch i lost my express port slot . Bums me out but what I do now is keep stuff on a 2nd Drive internally on the laptop a 500gb 7200 drive just for about 1 month backup so if I need access it is a little faster than going to Drobo. I use Drobo as my safe if you will, more for storage than working with it. Guess i should point out my OS drive is SSD with the new OWC Extreme 200gb drive which is very fast .

I have some space left for awhile until 4gb drives start hitting the street than just update the 2TB and than use the 2tb to off load storage. Get to recycle these drives as they keep increasing the TB on them. Which is kind of nice.


It would be interesting to find out read and write time with all the Drobo units with the same type of drives. Kind of confusing.
« Last Edit: May 05, 2010, 03:35:28 pm by Guy Mancuso »
Logged
[url=http://forum.getdpi.com/forum/showt

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #19 on: May 05, 2010, 09:45:24 pm »

Quote from: BJNY
For myself, I would be interested in hearing experiences with QNAP units
which cost in the same neighborhood as the Drobo Pro, yet offers faster speed and is stand-alone NAS (with its associated features).
My hesitation being I'm only versed in Mac.

I LOVE my QNAP and would give up 2 drobos for one QNAP in a heart beat.  I can back up 3 different computers at around 35Mbps each at the same time. I would say if you want to do "other things" with the server and are familiar with linux, get a QNAP over the drobo.  If you just want storage and no extras, get a drobo.  I don't use iSCSI since my WHS had issues re-connecting to the iSCSI device after a reboot.  I don't like how you can't connect to a iSCSI drive other than through the iSCSI target so I don't trust it.
Pages: [1] 2 3   Go Up