Pages: 1 [2] 3   Go Down

Author Topic: LR backup software  (Read 26765 times)

ButchM

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 749
Re: LR backup software
« Reply #20 on: November 20, 2015, 09:30:56 am »


It can't. If you think it can, you don't understand the concept of a Unix hard link.

Jeremy

Perhaps ... I don't understand. Once again ... explain how, for example that the machine I am typing from at this very moment, the TM backup for the startup drive dates back to Sept. 20, 2015. The startup drive is only a 500GB HD. The TM backup drive is likewise a 500GB HD.

Yet, If I were to go back and restore all available files that are available from my TM backup since Sept20, the total volume of data would certainly exceed 500GB. Especially when you consider all the shared files that have come and cone in my workgroup Dropbox folder over the past two months far exceeds the maximum uncompressed storage volume available to this computer.

If there is no 'compression' occurring ... then how is it possible to retain this amount of data? Not to mention multiple iterations of the same file can cumulatively measure above and beyond the space required to store the most recent version of said file.

So please, explain away, I'm always willing to learn.
« Last Edit: November 20, 2015, 09:59:41 am by ButchM »
Logged

Hans Kruse

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2106
    • Hans Kruse Photography
Re: LR backup software
« Reply #21 on: November 20, 2015, 10:20:02 am »

New to LR after Aperture.
I am dedicating a large drive to LR, I will be adding identical drives for backup. Which software for backing up the drives are preferred? I was using the Aperture vaults, very simple so not easy to get it wrong.
I'm hoping there is something easy and simple. Suggestions please, thanks.

Kevin,

Let me explain my total backup procedure so we are not just addressing part of the problem.

My system setup is as follows: One machine which is a MacBook Pro (MBP) with 1TB internal drive. On that drive are my documents, e-mails, applications. The Lightroom catalog and previews are stored here as well. The most recent photos from shoot are there as well. Older photos as well as documents etc. is stored on a single external LaCie Thunderbolt connected drive (3TB in total and mirrored (aka. Raid-1)). I regularly make Lightroom backups which is not really backup as such, but more a consistent copy of the catalog provided that testing integrity before backup is checked. I delete all but the latest backup from the backup folder. The backups are backed up on the TM backups.

I'm travelling a lot doing my photo workshops and the backup strategy is taking this into consideration. The backups are split between the MBP (and the external drive.

The MBP is backed up in my office on two external USB 3.0 drives (2 and 3 TB capacity) using Time Machine (TM). I daily (or sometimes every second day) attach the drives to my machine and run the TM backups. There are alternatives to TM. I have used TM since 2009 without a problem so continue to use it.

The external LaCie drive is backed up on two USB 3.0 drives (not the same as the TM drives). I use ChronoSync http://www.econtechnologies.com to synchronize folders on the LaCie drive to each of the two other drives and I have templates (synchronizer documents) for doing this. There are other solutions available that does the same thing.

For disaster backup (a backup independent of all other backups and which take in consideration physical disasters like a fire and theft in my home) I use Backblaze https://www.backblaze.com. Here my MBP and the external LaCie drive is backed up. There are other solutions available. When I return from a trip it takes about a day before all is backed up again. I have a 100Mbps symmetrical internet connection based on fiber into the building. Backblaze can use the entire bandwidth with 10 backup threads configured with no throttling.

For travel I bring two 2TB portable drives for TM backup. During my trip I will every day do the TM backup on the two drives. During my travel the online backup is disabled as typically the WiFi systems in hotels does not have capacity to backup the possibly more than 100GB that I shoot during a trip.

If during a trip I loose everything, I can go back to the TM backups in my office and get a new MBP and restore everything as it was when I left for the travel. If I loose the MBP only I can restore one of the two portable TM backups. The two backups should be kept separate so not both are lost in case of theft.

Why two of every backup? The reason is simple: I do not like having only one copy left of the primary copy is lost. If only one copy is left after failure this one could also fail before restore on a new drive or MBP.

Clearly it is possible to do something similar with different products and also using Windows, of course.

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: LR backup software
« Reply #22 on: November 20, 2015, 10:32:33 am »

If there is no 'compression' occurring ... then how is it possible to retain this amount of data? Not to mention multiple iterations of the same file can cumulatively measure above and beyond the space required to store the most recent version of said file.



http://icanhelpyourmac.com/tip_archive/understanding_apples_time_machine/index.html
Quote
An important fact to remember is that a file is not "backed up" unless it is stored on two hard drives at the same time. If you delete the original file off your computer's internal HD it is no longer backed up. Yes, the file does still exist on the Time Machine HD, but if the Time Machine HD would fail in any manner there is no longer a copy on your Internal HD. In this way, your Time Machine HD is also a Storage HD, but don't rely on it as such.With the above sentences in mind it is important to remember that since Time Machine will never delete the final copy of any file this is created on your Time Machine HD, your Time Machine HD may fill up very quickly unless steps are taken to reduce the number or size of files that are created, renamed or moved. For instance... if you create a file called "A.doc" on your desktop Time Machine will create a file called "A.doc" on your Time Machine HD. If you simply rename this file to "A1.doc" Time Machine will now permanently store a copy called "A.doc" and another file called "A1.doc".
This is not a problem when it comes to Word files or any text file in general as their sizes are small enough you likely would not notice the backup HD space being affected. Conversely, if you are working with an Adobe Photoshop file (.psd) that is 300 MB in size you Time Machine HD will likely fill up very quickly. Each hour you work on the file and alter it Time Machine will back up that hour's version of the 300 MB file. Thusly if it takes 4 hours to complete the file, Time Machine could have created well over 1 GB worth of backups. My suggestion is to use a folder called "In Progress" and another folder called "Archive". When you work on files save them only into the "In Progress" folder. Only once complete should the file be moved to the "Archive" folder. The reason this can solve the problem is because you can then simply delete the "In Progress" folder from your Time Machine HD. Doing so will delete all files and all copies of files from within the "In Progress" folder and then the next hour Time Machine will recreate the "In Progress" folder with it's new contents.



Here's why using TM on image files might not be such a good idea.
« Last Edit: November 20, 2015, 10:37:30 am by digitaldog »
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Hans Kruse

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2106
    • Hans Kruse Photography
Re: LR backup software
« Reply #23 on: November 20, 2015, 11:04:07 am »




http://icanhelpyourmac.com/tip_archive/understanding_apples_time_machine/index.html


Here's why using TM on image files might not be such a good idea.

Yes, but only a problem if the TM drive is constantly connected. Or just have a drive big enough so it's not really a problem. If the amount of data being backed up by TM is say 500GB, a 2-3TB TM drive would not fill up fast even with PSD files being updated and hourly backups are consolidated after 24 hours, daily backups for a month and weekly backups older than a month until the volume runs full and the oldest backup gets deleted. But, of course, as you point out, one need to be aware of this.

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: LR backup software
« Reply #24 on: November 20, 2015, 11:57:40 am »

Yes, but only a problem if the TM drive is constantly connected.
Sure but not having it connected kind of defeats it's purpose of 'backing up' all kinds of data while you work. I've got email going all the time, TM is really slick for recovering them if I delete some mail and change my mind.


Quote
Or just have a drive big enough so it's not really a problem.

An easy answer but not always possible. And with TM, eventually something has to get deleted.
Quote
If the amount of data being backed up by TM is say 500GB, a 2-3TB TM drive would not fill up fast even with PSD files being updated and hourly backups are consolidated after 24 hours, daily backups for a month and weekly backups older than a month until the volume runs full and the oldest backup gets deleted. But, of course, as you point out, one need to be aware of this.
Maybe not but the point I attempted to make is to 'dismiss' the idea that TM is just like other backup's, it's not and that TM is adequate for all backup needs, it's not. The URL points out how many of the files copied are not a backup any longer as only one copy exists.
I love TM, I encourage people to use it. It's not the same as a daily clone using the products discussed. Use both systems or more, but understand that TM and cloning are not remotely the same schema.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Hans Kruse

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2106
    • Hans Kruse Photography
Re: LR backup software
« Reply #25 on: November 20, 2015, 12:13:00 pm »

Sure but not having it connected kind of defeats it's purpose of 'backing up' all kinds of data while you work. I've got email going all the time, TM is really slick for recovering them if I delete some mail and change my mind.

 
An easy answer but not always possible. And with TM, eventually something has to get deleted.  Maybe not but the point I attempted to make is to 'dismiss' the idea that TM is just like other backup's, it's not and that TM is adequate for all backup needs, it's not. The URL points out how many of the files copied are not a backup any longer as only one copy exists.
I love TM, I encourage people to use it. It's not the same as a daily clone using the products discussed. Use both systems or more, but understand that TM and cloning are not remotely the same schema.

The recommendation not to use TM because of frequently updated large files is my point. Either you don't have the TM connected all the time or you have a large enough TM backup drive and have two of them is my recommendation. Even if you had 20 1GB PSD files being updated every hour for a working day 10 hours you would have at the most 10*20GB=200GB of data which gets down to 20GB after 24 hours. This should be clear for those who use TM and get this disrecommendation of it's use.

I do agree it's important to understand it's use and limitations. The thing about deleted files goes away with two TM drives connected which would be a good idea anyway. Never have only one copy of your data.

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: LR backup software
« Reply #26 on: November 20, 2015, 12:17:28 pm »

The recommendation not to use TM because of frequently updated large files is my point. Either you don't have the TM connected all the time or you have a large enough TM backup drive and have two of them is my recommendation.
3rd option, what I use: storing all my image data on a separate drive and tell TM not to touch it. Then tell the cloning software (and for me CrashPlan) to back it up every night. TM is dedicated to my boot disk and all those important files. Large image files are not touched but backed up at least once a day. As such, my 2TB TM drive has files going back 2 years. IF I included big images files, that wouldn't be the case. Now one can ask, do you need to go back 2+ years to find some old file? I've gone back many months but no, not two years. But that data is available if I had to. It wouldn't if I included big image files. But whatever works for you.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Hans Kruse

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2106
    • Hans Kruse Photography
Re: LR backup software
« Reply #27 on: November 20, 2015, 01:08:40 pm »

3rd option, what I use: storing all my image data on a separate drive and tell TM not to touch it. Then tell the cloning software (and for me CrashPlan) to back it up every night. TM is dedicated to my boot disk and all those important files. Large image files are not touched but backed up at least once a day. As such, my 2TB TM drive has files going back 2 years. IF I included big images files, that wouldn't be the case. Now one can ask, do you need to go back 2+ years to find some old file? I've gone back many months but no, not two years. But that data is available if I had to. It wouldn't if I included big image files. But whatever works for you.

Well, one can argue about that, but the consolidation to one file per day after 24 hours avoids most if not all of the problem. Anyway one could just buy a bigger drive. But whatever works is the important part and know how it works.

Manoli

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2296
Re: LR backup software
« Reply #28 on: November 20, 2015, 01:08:58 pm »

I'm hoping there is something easy and simple ...

A couple of pages on and me thinks the OP has done a runner ...  :)

As we've touched upon RAID's I'd point out that for those on OS X, Apple has stripped RAID support from their disk utility on El Capitan. OWC have SoftRAID , which they've now produced in a LITE version ($34). Well worth the investment for two reasons: (1) they're is constant real-time monitoring of all drives on the system in the menu bar (via an LED style bulb) which will alert you to any persistent errors in writing to disk - usually ample warning before a disk fails, and (2) real time logging of hours each individual disk's hours of use, i/o errors and reallocated sectors.

Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: LR backup software
« Reply #29 on: November 20, 2015, 01:17:22 pm »

Well, one can argue about that, but the consolidation to one file per day after 24 hours avoids most if not all of the problem. Anyway one could just buy a bigger drive. But whatever works is the important part and know how it works.
If an argument is about the consolidation of all image files, LR database etc on one drive, I'll point out some of the advantages:


These are probably the largest files you'll create so as you continue to shoot and edit iterations, having the biggest drive you can find and consolidating them in one place makes sense. Why clog up disk space with MS word files?


Cloning this one drive to as many drives as you desire means you can pick one up, take it on location and have all your data with you. As you add images from location, it's a snap to clone the original drive, backing it up and ensuring the desktop now has the new files.


With the LR catalog on this drive, anyone with a copy of the application, Mac or Windows can access all the photo's as they also reside on this drive.


As I stated, it's super easy to turn off TM for this one drive.


Multiple cloned drives can be rotated. Off site, in a fireproof safe, on your desktop machine and for location work.


Cloning with the applications mentioned is fast, only new or altered data is written.


Off hand, none of the above is possible or easy IF the backup schema is solely TM.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

ButchM

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 749
Re: LR backup software
« Reply #30 on: November 20, 2015, 01:37:53 pm »

A couple of pages on and me thinks the OP has done a runner ...  :)

As we've touched upon RAID's I'd point out that for those on OS X, Apple has stripped RAID support from their disk utility on El Capitan.

There are mixed views as to whether Apple 'stripped ' RAID support from DU ... or if it hasn't been added yet. Disk Utility was completely rewritten for El Cap and the RAID portion may come back at a later date as that project moves along.

I reference this point that the RAID function still exists in El Cap as it can be invoked via Terminal ... it just isn't available in the Disk Utility UI.
Logged

KevinA

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 979
    • Tree Without a Bird
Re: LR backup software
« Reply #31 on: November 20, 2015, 01:44:24 pm »

Thanks all. At present I'm running Aperture and LR while I make the transition.
I already back up my computers HD with TM, I didn't fancy using TM for the library. I don't store much on the laptop, not even the LR catalog as its a flash drive and space is not great, I wouldn't buy with such little space again (250gb).
My LR dedicated thunderbolt drive also holds the catalog, I will use one of the mentioned programs to make two clones of the LR drive. One backup is never enough and if I do get a major problem, I NEVER plug in a backup until I know what the problem is.  I'm thinking if the catalog is on the same drive as the images whichever computer I use the catalog is staying consistent .
Also a large selection of my work gets uploaded to my site as high res jpg, so its 2 1/2 backups in reality I have.
Thanks again.
Logged
Kevin.

KevinA

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 979
    • Tree Without a Bird
Re: LR backup software
« Reply #32 on: November 20, 2015, 01:47:27 pm »

Also I have my library insured for quite a lot incase for some reason everything vanished....in fact if that happened I would take the money and run :-)
Logged
Kevin.

Jeremy Roussak

  • Administrator
  • Sr. Member
  • *****
  • Offline Offline
  • Posts: 8961
    • site
Re: LR backup software
« Reply #33 on: November 20, 2015, 03:15:53 pm »

Perhaps ... I don't understand. Once again ... explain how, for example that the machine I am typing from at this very moment, the TM backup for the startup drive dates back to Sept. 20, 2015. The startup drive is only a 500GB HD. The TM backup drive is likewise a 500GB HD.

Yet, If I were to go back and restore all available files that are available from my TM backup since Sept20, the total volume of data would certainly exceed 500GB. Especially when you consider all the shared files that have come and cone in my workgroup Dropbox folder over the past two months far exceeds the maximum uncompressed storage volume available to this computer.

If there is no 'compression' occurring ... then how is it possible to retain this amount of data? Not to mention multiple iterations of the same file can cumulatively measure above and beyond the space required to store the most recent version of said file.

So please, explain away, I'm always willing to learn.

The link that Andrew posted is quite helpful, if a bit superficial. The crucial thing is that when it makes what is apparently an image of the entire disk, what it is actually doing is duplicating only the directory structure of the disk and those files which have changed.

Every file has an entry in at least one directory which points at it. It also has a reference count, which is the number of directories in which it appears. For an existing, unchanged file, all Time Machine has to do is to create an entry in a new directory and increment the reference count. However large the file, that takes only a few bytes. Some changes had to be made to the Unix file system to allow hard links to point at directories, and a good deal of care was needed to avoid circular directory references.

When a file is deleted, its reference count is decremented and if it is now zero, the space the file occupies is returned to the free list.

That also a superficial explanation, but I hope it helps. I used to have a link to a very comprehensive article, but I can't find it now.

Jeremy
Logged

Eric Brody

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 487
    • http://www.ericbrodyphoto.com
Re: LR backup software
« Reply #34 on: November 20, 2015, 06:24:26 pm »

I may have missed something as I read all the interesting and useful posts in this thread. The OP said he wanted to back up Lightroom. As he is coming from Aperture, about which I know absolutely nothing, I suspect but am not certain he recognizes the difference between the LR catalog (database) and the actual image files which can be located anywhere one wishes. It is critical to back up the image files as well as the catalog. I keep my LR catalog on the SSD of my Mac Pro but keep the image files, the actual photographs are on a different drive in an OWC Thunderbay array. I use TM and CCC for the SSD and use CCC for the image files, including an off site backup. Lots of great information about backup strategies in this thread, thanks to all.
Logged

ButchM

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 749
Re: LR backup software
« Reply #35 on: November 20, 2015, 09:49:27 pm »

The link that Andrew posted is quite helpful, if a bit superficial. The crucial thing is that when it makes what is apparently an image of the entire disk, what it is actually doing is duplicating only the directory structure of the disk and those files which have changed.

Every file has an entry in at least one directory which points at it. It also has a reference count, which is the number of directories in which it appears. For an existing, unchanged file, all Time Machine has to do is to create an entry in a new directory and increment the reference count. However large the file, that takes only a few bytes. Some changes had to be made to the Unix file system to allow hard links to point at directories, and a good deal of care was needed to avoid circular directory references.

When a file is deleted, its reference count is decremented and if it is now zero, the space the file occupies is returned to the free list.

That also a superficial explanation, but I hope it helps. I used to have a link to a very comprehensive article, but I can't find it now.

Jeremy

That doesn't explain how my TM backup actually contains more data than the drive can store ... even though my backup TM drive is the same size as the drive being backed up.
« Last Edit: November 20, 2015, 10:02:49 pm by ButchM »
Logged

Jeremy Roussak

  • Administrator
  • Sr. Member
  • *****
  • Offline Offline
  • Posts: 8961
    • site
Re: LR backup software
« Reply #36 on: November 21, 2015, 03:45:23 am »

That doesn't explain how my TM backup actually contains more data than the drive can store ... even though my backup TM drive is the same size as the drive being backed up.

It doesn't. It can't. Why do you think that it does?

Jeremy
Logged

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website
Re: LR backup software
« Reply #37 on: November 21, 2015, 04:29:05 am »

Time Machine looks to me to be very similar if not identical to the standard rsnapshot/rsync mechanism available on many Linux and UNIX systems.  As is so often the case, Apple seems to have taken an open-source tool, modified it to work within Apple's version of the UNIX operating system, and made it easier to use.

(I use rsnapshot on a pair of Linux systems as my second-level back-up, after Time Machine, for my images and Lightroom catalogs.  I have a script that copies any changes in the image and catalog hierarchies to a general-purpose Linux file server once a day—the location I use for LR imports is updated once a minute—and rsnapshot backs up the file server to a dedicated Linux back-up server once an hour.)

KevinA

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 979
    • Tree Without a Bird
Re: LR backup software
« Reply #38 on: November 21, 2015, 06:40:17 am »

Eric, I'm well aware of LR stupid none baqckup solution. If they can create an option to backup the catalog why can't they come up with an integrated back up for the important things.
There are many things in LR I prefer the dam side of it isn't one. Clunky to say the least.
Logged
Kevin.

Manoli

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2296
Re: LR backup software
« Reply #39 on: November 21, 2015, 07:00:11 am »

If they can create an option to backup the catalog why can't they come up with an integrated back up for the important things.

Such as ?

1- Create a catalog backup directory
2- Create a sub-directory 'lr-config' ( for all your configuration files, dcp, presets, etc etc , of which there will be plenty)
3- Dig deep and invest in donation ware :

http://www.photographers-toolbox.com/products/mdawson/tpglrbackup/
http://thephotogeek.com/quick-easy-lightroom-backup/

All your Lr program files will be in one place - up to you how and where you back up your image files.
« Last Edit: November 21, 2015, 07:51:09 am by Manoli »
Logged
Pages: 1 [2] 3   Go Up