Pages: 1 [2] 3   Go Down

Author Topic: How do you back up your massive files?  (Read 10374 times)

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #20 on: May 05, 2010, 09:47:38 pm »

Quote from: Guy Mancuso
I have some space left for awhile until 4gb drives start hitting the street than just update the 2TB and than use the 2tb to off load storage. Get to recycle these drives as they keep increasing the TB on them. Which is kind of nice.
4TB, I assume. Hitachi said in 2008 that their 4TB drives will be out in 2011.

shutay

  • Full Member
  • ***
  • Offline Offline
  • Posts: 203
    • http://www.asiaphotohub.com/Jason/
How do you back up your massive files?
« Reply #21 on: May 05, 2010, 10:35:24 pm »

I use a homebuilt NAS with Gigabit Ethernet and RAID 5. Current available capacity is 2TB which is fine for me, but the rig'll go up to 14TB in terms of space to install hard disks and it boots Linux from a SSD. Whole house is wired for Ethernet, since backup via WiFi is sloooow. It recently had one drive fail, which it survived without any problems. Check which drive failed, power down, remove the busted drive, pop in the new drive and off you go.

IMPORTANT TIP: One tip I would give to those considering a self-built NAS or storage appliance like the Backblaze tip (also something they don't mention) - make sure you label each drive with it's serial number in a location where you easily see it, because I've learned that the drives don't necessarily get enumerated in the OS in the order in which they are installed or connected, although this also depends on the OS. Basically, you happily build your NAS with 12 drives stacked together, and then bam! One drive fails. Which one was it? Windows/Linux shows it as the nth drive, so since I installed the drives in order, it should be the nth one up, shouldn't it? Not always. Unless you're using a fancy schmancy NAS enclosure which has a dedicated LED to indicate which drive failed, you won't know which one it was. So you check in the OS, and you get the serial number of the drive that failed, but which one was it? They're all stacked together so you can't see the manufacturer's label to find the serial number. If you only have 3 or 4 drives stacked together, although it's troublesome enough to look through, you can do it, but any more than that, and it's a hassle, downtime, far too much disruption, etc. So make sure you label each drive with it's own serial number as you install them in a location that you can easily see the label when you open up the hood (see attached pic).

Ensure your cooling is adequate, especially the disk drives. Decide on your airflow, ensuring that it all flows in a single direction (i.e., front to back for example). The motherboard will have temperature sensors on it, and each drive also has a temperature sensor. You can use utilities like HWMonitor (Windows users, or the lm_sensors package in Linux) to check the temperature of the CPU, the ambient temperature inside the casing (motherboard sensor) and the temperature of each drive. If you find your base setup is hotter than you'd like it, make one change to the cooling setup at a time and then check temperatures again over a period of time (e.g., an hour, with simulated use). If you make too many changes at once you risk having a system that doesn't seem to be making any improvement in temperatures and you won't know why. Keeping your drives cool will help them last longer.

I've heard that it's also a good idea to ensure that each drive comes out of a different production batch. If you have 45 drives, this may not be possible, but if you have a smaller number of drives, it's quite doable. Just get the supplier to indulge you a little! Supposedly, this reduces the risk that a particular manufacturing defect or tolerance will affect too many of the disks at once.

For the really paranoid, choose a NAS solution that is capable of mirroring itself to another NAS and perform an automatic failover if one array should fail altogether. Don't forget to use a good UPS. RAID 6 affords tolerance of a simultaneous 2 drive failure.
« Last Edit: May 06, 2010, 04:01:39 am by shutay »
Logged

nicolaasdb

  • Full Member
  • ***
  • Offline Offline
  • Posts: 213
    • http://
How do you back up your massive files?
« Reply #22 on: May 06, 2010, 03:21:30 am »

I have 12 x 2 TB..... use them as single drives (No raid) the first 6 are main drives the 2nd 6 are backup and I have a lot of old drives as 2nd backup.

I have the 12 bay eSATA storage unit online 24/7.

I know this system is seriously flawed, but don't have the energy to change it and don't feel like spending thousands on something better.

Active work sits on 2 of my computer hard drives (there are 4 in the mac towers..whatever you call them) so if something goes wrong I have about 3 backups and when done (after a LONG time) I move them to the external drives to never be looked at again.

I think that I have a total of 40TB in online and offline storage... it is too much... where are the days of boxes and boxes full of contact sheets??
« Last Edit: May 06, 2010, 03:35:53 am by nicolaasdb »
Logged

nicolaasdb

  • Full Member
  • ***
  • Offline Offline
  • Posts: 213
    • http://
How do you back up your massive files?
« Reply #23 on: May 06, 2010, 03:22:38 am »

oeps double post
« Last Edit: May 06, 2010, 03:28:17 am by nicolaasdb »
Logged

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #24 on: May 06, 2010, 03:53:19 am »

I have 1 3x2tB RAID 5 and I was converting the RAID 1 2x2TB to another 3x2tB RAID 5 and something didn't finish.
I just had to reboot my QNAP and I will try again once it is done building.

Christoph C. Feldhaim

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2509
  • There is no rule! No - wait ...
How do you back up your massive files?
« Reply #25 on: May 06, 2010, 06:09:40 am »

The "rsync" command is my friend!
Can be used from any platform with a little research.

Link: http://www.mikerubel.org/computers/rsync_snapshots/

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #26 on: May 06, 2010, 09:35:17 am »

Quote from: ChristophC
The "rsync" command is my friend!
Can be used from any platform with a little research.

Link: http://www.mikerubel.org/computers/rsync_snapshots/
I use this on my linux systems, but when I tried to get it to work on windows, it was a colossal failure. I was unable to get it to work over SSH.  Also, I did find a way to do bitwise comparison.

Another thing is that sometimes my file times were messed up using rsync.

mtomalty

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 541
    • http://www.marktomalty.com
How do you back up your massive files?
« Reply #27 on: May 06, 2010, 12:24:23 pm »


Not directly related to each of out individual needs but interesting, nevertheless.


http://magazine.creativecow.net/article/th...-archive-system
Logged

narikin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1376
How do you back up your massive files?
« Reply #28 on: May 06, 2010, 07:05:13 pm »

Quote from: revaaron
cloud storage at home. This is an awesome article about it:
http://blog.backblaze.com/2009/09/01/petab...-cloud-storage/

4gb file limit means this (otherwise great) service is useless for photographers.

Logged

revaaron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 333
How do you back up your massive files?
« Reply #29 on: May 07, 2010, 02:31:53 am »

Quote from: narikin
4gb file limit means this (otherwise great) service is useless for photographers.
you have files that are over 4GB?  a 190MP TIFF is 1.1GB.

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
How do you back up your massive files?
« Reply #30 on: May 07, 2010, 11:07:05 pm »

I'm very impressed with the 'ultra-portable' pocket drives. I have a few of these, bought over the years, ranging from 160GB to 1TB. None of them have failed yet.

A very small bookshelf could hold an enormous amount of data in the form of 'WD, My Passport, 1TB' pocket drives which are powered through the USB connection.

This seems to me an ideal form of storage, considering that the 1TB versions are now so affordable.

Does anyone reading this thread have any negative comments about the reliability of 'WD My Passport' drives?
Logged

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
How do you back up your massive files?
« Reply #31 on: May 08, 2010, 03:48:44 am »

I have just read this thread for the first time and I am staggered. I had long realised that digital storage presents problems for the active photographer simply by considering my own current, tiny shooting rate as an ex-pro, but it seems to me to be a very expensive, time-consuming nightmare.

I have no idea how long any of you have been in practice, but when I consider my own situation, I am happy to have lived out my working life in film!

All it took was a register - a book - that carried columns with job number, film number, date, client name, location, subject; a few rows of negative bags and a metal filing cabinet containing transparencies on racks. It was all so easy, accessible and compact too.

Of course I can't know for sure how it affects others, but I imagine much of the problem with digital storage is also due to the relative ease/temptation of overshooting even the most simple subject.

Was a time one did the shot, handed over the print or transparency, went home and forgot about it until the cheque came in. Or didn't, but that was a whole other battle.

You truly have my sympathy.

Rob C

fredjeang

  • Guest
How do you back up your massive files?
« Reply #32 on: May 08, 2010, 04:42:12 am »

Quote from: Rob C
...Of course I can't know for sure how it affects others, but I imagine much of the problem with digital storage is also due to the relative ease/temptation of overshooting even the most simple subject...

Rob C

I join your opinion here.

I guess that overshooting is part of the digital in itself.

The irony is that in fashion for example, where the shooting rate is very high, do you see better pics now from the gurus than 15 years ago? I don't honestly.
And that, in any area. There are no better street photography despite the incredible increment in volume.

But yes, what's new is a total dependance and slavery on electronic solutions for any purpose.
It's a much bigger market, a huge industry, hungry of our money.

There are very few people that really use and exploit digital as a completly new tool, most is a more flexible variation and interpretation of the traditional photography.
In fact, what we see is new technology for old practises. Here I'd say that I always like better the originals.

Backups are facing the problems of overshooting (over-volume of datas) plus, the problem of classification (bad folder manes)  plus, the problem of reliability (unstable solutions).







Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
How do you back up your massive files?
« Reply #33 on: May 08, 2010, 09:27:12 am »

Quote from: Rob C
I have just read this thread for the first time and I am staggered. I had long realised that digital storage presents problems for the active photographer simply by considering my own current, tiny shooting rate as an ex-pro, but it seems to me to be a very expensive, time-consuming nightmare.

I have no idea how long any of you have been in practice, but when I consider my own situation, I am happy to have lived out my working life in film!

All it took was a register - a book - that carried columns with job number, film number, date, client name, location, subject; a few rows of negative bags and a metal filing cabinet containing transparencies on racks. It was all so easy, accessible and compact too.

Of course I can't know for sure how it affects others, but I imagine much of the problem with digital storage is also due to the relative ease/temptation of overshooting even the most simple subject.

Was a time one did the shot, handed over the print or transparency, went home and forgot about it until the cheque came in. Or didn't, but that was a whole other battle.

You truly have my sympathy.

Rob C

Strange perspective, Rob. Digital storage is easy and cheap. What's problematic is the preservation and storage of film. It has to be scanned, and that's expensive and time-consuming. It's easier to transfer the contents of one 8GB flash card to an external hard drive than it is to scan and save one frame of film and contend with the issues of getting the colors and tonality right and removing the scratches.

As an amateur shooting film many years ago, I wasn't nearly as prolific as I've been since I started using DSLRs. However, the freedom from concern about the cost and hassles of film development every time I press the shutter is quite liberating.
Logged

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
How do you back up your massive files?
« Reply #34 on: May 08, 2010, 11:13:00 am »

Quote from: Ray
Strange perspective, Rob. Digital storage is easy and cheap. What's problematic is the preservation and storage of film. It has to be scanned, and that's expensive and time-consuming. It's easier to transfer the contents of one 8GB flash card to an external hard drive than it is to scan and save one frame of film and contend with the issues of getting the colors and tonality right and removing the scratches.

As an amateur shooting film many years ago, I wasn't nearly as prolific as I've been since I started using DSLRs. However, the freedom from concern about the cost and hassles of film development every time I press the shutter is quite liberating.



Haven't found it that way, Ray. I have both Ektachrome 120 (few, because I hardly ever shot 120 after around the early '80s) and Kodachrome and they are still in pretty good condition. Yes, a lot of my 35mm slides were effed, but I received them that way after getting them back from my stock library when we parted company. I have written them off as too much work.

But, with commissioned work, the essential thing was that once the client had approved the shots and they had them in their hands, I was free to forget about them and look for new work or just lotus eat. Didn't have to store them! Transparencies that didn't leave my control are still okay. I find no expense with scanning: I do my own. And it saves time too, since I can edit on a lightbox far more quickly and surely than taking the time to put files through the system and then run them up together on the monitor to compare. Mabe it's just the years that make it natural for me.

Of all the professional stages, it was watching pages run off a huge litho printer that gave most angst: those mothers never matched the client-approved proofs!

As for digital being liberating - well yes, you can click to sick cheaply, but does it bring better results than holding back and preserving your powder until you see the whites of their eyes?

I repeat again what I have said quite recently: but that I had my 500 series stuff back and a 120 scanner! Expensive as those scanners may be, I am sure I have wasted far more chasing two digital formats to reach the stratus quo, and without really being totally happy having got there. The other thing is, as I get older, I feel more inclined towards considering what I shoot than I used to - slow suits me better now. ;-(

Rob C

fredjeang

  • Guest
How do you back up your massive files?
« Reply #35 on: May 08, 2010, 03:05:17 pm »

Quote from: KLaban
Rob, I don't get this, why would the capture medium make a difference? Why would I shoot more images when using digital capture and less using film? If anything I shoot less images using digital capture simply because I have more confidence in the results and confirmation of what I've shot. This knowledge has made me more relaxed and liable to shoot all the slower. There was a time that I could only dream of the kind of hit rate that I can now achieve when using digital capture.
Klaban,
What you say is true, but I do think that you are in the very few who shoot less with digi. I tend to take this path also, in other words, less I shoot, more I like digital. More I shoot, more I hate it.
But you come from painting, so do I.
We are used of slowness.
Logged

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
How do you back up your massive files?
« Reply #36 on: May 08, 2010, 05:06:20 pm »

Hi Keith

Yes, for you I am sure it holds true - you never were into the machine gun school because of your subject matter (if your site is representative of your thinking over the years). As Fred suggests, you are in the minority that way. Also, you are a pro and know more or less what you are looking for; many do not. In my own case, I used to use a cassette of 35mm on a shot if I was using that format, or usually a roll on 120 if on that. Why? Because the two cameras promoted different ways of doing the job and almost suggested themselves depending on the work.

With the Nikons it was a case of building up the mood until it started to degenerate and go stale; on the Hass I would be using it for shots that didn't demand much model reaction - greater than that was the need for detail/shape of outfit and room for the art department to drop in text. You got that pretty soon, and doing more than a dozen on any setup was overkill.

I suspect that doing exactly the same work today, with digi, even I would have been tempted to shoot way more than made productive sense.

That's why I can honestly declare that, today, the 'blad/scanner would be it for me - I no longer have the model work and my subjects probably don't move intentionally... But, one took the decisions as one did and that's a deal that won't permit a re-run.

Really, it's the horses for courses thing again. (There has got to be something there to add about one trick ponies; I can't quite think of it, but it's still too good an opportunity to miss!)

;-)

Rob C

PS Fred, if you are reading this, can you give me again that link to the photographer using the weebly system? I put it into Favourites for reference, but I can't remember his name and there are dozens of guys listed there!
« Last Edit: May 08, 2010, 05:10:34 pm by Rob C »
Logged

rueyloon

  • Full Member
  • ***
  • Offline Offline
  • Posts: 221
    • http://www.36frames.com
How do you back up your massive files?
« Reply #37 on: May 08, 2010, 09:35:28 pm »

The best way is still the easiest way...... or is it the easiest way is the best way ?

After calculating the cost of a Drobo, I'm convinced that using a dual HDD system is the best.

Dual HDD (back ups, NOT RAID !) all the way, from your working computer to the final storage.
It works out cheaper and easier.

You can keep one of the HDD in a safe location and put the other in a external HDD (I use a 4 tier) rack
to access your past years' files.

You can even get one of those slot in type of HDD casings to have easier access to your past years' files.
The whole thing looks and works like a giant CF card reader.
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
How do you back up your massive files?
« Reply #38 on: May 09, 2010, 01:39:48 am »

Quote from: fredjeang
....... but I do think that you are in the very few who shoot less with digi. I tend to take this path also, in other words, less I shoot, more I like digital. More I shoot, more I hate it.
But you come from painting, so do I.
We are used of slowness.

This doesn't make much sense to me, Fred. But I'm an amateur.

As a matter of fact, many years ago I was amazed when I first learned that some professionals would shoot off a whole roll of film in order to get the one good shot that would satisfy the client. I was far more frugal. A roll of film could last me a month.

When I went trekking in Nepal for a couple of months about 46 years ago with my recently purchased Pentax Spotmatic, I took only 3 rolls of 36 exposure Kodachrome 64 and a couple of rolls of B&W.

That oversight probably saved my life. At that time (in 1964) the Tibetans were still putting up a resistance to the Chinese invasion. Lots of them continued to flee into Nepal where they were given shelter in U.N.-run camps. I visited them at one camp in Trisuli, and took lots of photos (ie. about 20).

Here's a couple of examples: [attachment=21887:Mounds_of_rice.jpg]  [attachment=21888:Tib_monk_A3_print.jpg]

When I learned that many of these refugees would cross back and forth at will, through the high mountain passes into Tibet and back again, I got very excited about the prospects of some real adventure. I would join one of these groups going back into Tibet, photograph their struggles and way of life, and perhaps even photograph some war conflict between the Chinese and Tibetan resistance groups.

I had National Geographic in mind. I'd become famous. (I coulda bin somebody   ) Of course, I was young and foolhardy. In retrospect I now realise I'd probably have got caught and thrown into a Chinese jail, if not killed.

Fortunately, perhaps (perhaps not), I was down to my last roll of film which was already half-used.

My main concern was not so much the physical risk but the frustration of coming across amazing events to photograph but not having any film left. So I regretfully decided against the plan.

I just mention this little off-topic aside to make the point that I've always followed the principle that one only photographs something that interests one, that catches one's attention, that 'moves' one in some way, whether the camera is film or digital.

There's a strong tendency to take more shots when using a digital camera simply because of the greater potential of the tool. As a result of taking multiple shots of a particular scene you may avoid the mistake of an exposure which is not ideal; you may increase dynamic range (by merging to HDR); you may turn your humble 35mm camera into a tool which surpasses an 8x10 field camera (by stitching numerous shots), and you may also increase your chances of capturing an unplanned, fortuitous, accidental masterpiece.

As George Bernard Shaw commented, when asked what he thought of the proliferation of affordable 35mm cameras bought by the masses, "The greater the total number of photos taken, the greater the number of 'good' photos taken."

Logged

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
How do you back up your massive files?
« Reply #39 on: May 09, 2010, 04:25:09 am »

Quote from: Ray
As George Bernard Shaw commented, when asked what he thought of the proliferation of affordable 35mm cameras bought by the masses, "The greater the total number of photos taken, the greater the number of 'good' photos taken."



Unfortunately, an even greater, unavoidable volume of shit comes packaged with that deal.

Rob C
Pages: 1 [2] 3   Go Up