Luminous Landscape Forum
Raw & Post Processing, Printing => Adobe Lightroom Q&A => Topic started by: Bill Koenig on November 28, 2011, 07:28:12 pm
-
I'm about to make the move to LightRoom 3 and I have a few questions.
Just to let you know, I'm not a working pro, just a serious amateur. In the 5 years with my DSLR I have about 1 TB of data, all backed up to multiple drives both on site /off site. At this point I'm not going to import any of this into LR, but from this point on, plan to save all of my future work in LR.
I use a app called Second Copy 8 to back up all of my data from my internal data drive to my external drive, that external drive gets backed up to another that I keep in my desk. Now that drive is rotated off site with yet another drive weekly.
This method backs up everything to 4 different drives minis the weekly rotation.
First question, with the lightroom catalog file, I've read that all files in a catalog must now be moved within LR, this isn't going to work with Second Copy 8
Is there a way I can make LR work with my currant backup strategy?
I work with raw files exclusively, at what point in the LR work flow does the raw file get converted/processed? I assume that a raw file saved in the catalog is still just the raw file with the editing saved as a metadata tag that rides with the file in the catalog.
There is also a cache file created as well, I assume that rides within the catalog as well, but I see that the cache can be saved in another location to increase performance, so how does LR know where the cache is if I move it to a different partition, and how do I back all of this up?
Thanks for any help,
-
Hi Bill,
It sounds like you have a slight misconception about the Lightroom import process that once cleared up might help you figure out the impact on your backups.
When you "import" iimages into Lightroom from your camera's memory card, Lightroom copies those images to a location on disk of your choice, then makes a catalog (database) entry for that disk location so it can associate your image file with its preview file and any edits (all of which are paramatric) you make to the image. Thoese parametric edits are kept in the catalog database, and optionally writtedn out to .xmp files next to your image files, you raw files are not "touched" (one exception I'll not go into here). Your raw files are not "in" the catalog, they are "pointed to" by the catalog.
So this meas that your normal backup process should pick up new raw files as usual, and the catalog file is just an additional .lrcat file to back up, and the Lightroom Settings are in a folder that should also be backed up. You would probably exclude the preview cache files from backup as they are automatically re-creatable from the raw files and the catalog - the previews are neatly kept in a xxxPreviews.lrdata folder.
Now the business about "I've read that all files in a catalog must now be moved within LR" is probably referring to the fact that Lightroom's database is keeping a "pointer" to where each image exists on disk, so if you want to reorganize the folder structure of you images for whatever reason, then Lightroom will lose track of the images. This can be handled in 2 ways - the easiest is to do the folder structure movements inside Lightroom - this moves the files on disk as well as keeping those catalog pointers up-to-date. You could also just go ahead and move the folders on disk and when you run Lightroom again the thumbnails for the moved images will show a question mark - and Lightroom then allows you to "find" the moved master image to re-establish the link in the catalog.
Either way, the effect is that the files move on disk, and Lightroom catalog is kept in sunc with the moves. Either way the files HAVE moved and your backup software will notice the changes and back them up.
"I work with raw files exclusively, at what point in the LR work flow does the raw file get converted/processed?" When you export, the parametric edits are applied and the resulting jpeg, tif, or whatever is written out to a folder of your choice. When you print, the edits are applied and the result is printed. When you create a web page, the edits are applied to the generated jpegs for the web page. When you are looking at the photos on the screen, the edits have been applied when the screen image is generated.
Hope this helps,
Kevin
-
Hi Bill,
I seems like Kevin has done a greg job of answering your questions. I have a couple of other points that might be helpful to you...
First of all, don't be confused by Lightroom's "Back-up" feature in the Catalog Settings. It backs up your Catalog only, not your original image files. Furthermore, it only backs up your Catalog to the same hard drive - a bit useless if you are trying to protect yourself from a hard drive failure! Remember, the Catalog has your metadata, a preview and your Develop/Print/Web and Slideshow settings for each image and Collection but it does not have your original image files.
If you are already backing up your hard drive on a regular basis, then there is no real need to back-up the LR Catalog separately.
I have found it helpful to keep all my original image files in a folder called LRPhotoLibrary. I advocate doing this so that photos don't just get dumped into "My Pictures" - it keeps things "tidy". You most likely already have a file structure that works for you, but if not, this is what I do... Within that LRPhotoLibrary folder I have sub-folders for each year: 2011, 2010, etc. Within each Year folder are the image folders for each shoot based on date: YYYYMMDD-DescriptiveFileName. While Lightroom does an excellent job of keeping track of the location of files, every once in a while one goes astray which I can easily find because of the file structure I've set up. At the same time, once your Catalog is built within Lightroom, it is important to remember to move files and folders from within Lightroom so that it will keep track of the changes.
Since you already have your originals on a hard drive, when you commit to creating your Lightroom Catalog via "Import", you will be "Add" them to Lightroom, not "Copy". Choosing "Add" does not move the photos, but references them to their present location. Your TB of photos will take a long time to import; you might consider doing a year at a time.
When importing, you have the option of specifying what size of Previews Lightroom will create for each image and store in the Catalog (see Catalog Settings > File handling). I would recommend something smaller than the 2048 pixel maximum. When you need it, Lightroom automatically creates a 1:1 preview (e.g. for looking at details, sharpening, etc.).
Good luck with this and don't hesitate to ask more questions!
-
Hi luxborealis,
"Furthermore, it only backs up your Catalog to the same hard drive - a bit useless if you are trying to protect yourself from a hard drive failure! "
At least in Lightroom 3 you may select your catalog backup to go anywhere you choose (the "Choose..." button in the screenshot below).
-
Why do you say that moving files within LR won't work with your backup application? Where/how you move the files on the disk shouldn't have any impact. The idea, of course, is to develop a sound file structure so that images don't have to be moved much at all.
In addition to the two ways mentioned of moving and relocating files in LR, you can also use the Sync Folder option. If you move files outside of LR, right click on the affected folders and select Sync Folder. LR will remove the files from the original location and add them back in the new location.
-
Yes - thanks for clarifying. The "Choose" button comes up only after you have selected a back-up option and that option executes (upon closing or once-a-week, what ever you've chosen). It's rather bass-ackwards to normal workflows as one should be able to select the back-up destination when the option to back-up is first selected - one of the reasons I have stayed away from LR's rather weak back-up option.
-
For the last 5 years working with Photoshop, bridge, and ACR, I've used the following structure to organize my photos.
My currant internal data drive (2TB) has 3 partitions, each partition represents old hard drives that have filled up over time. Right now 2 of these partitions hold all of my old files and the third (1TB) which is my currant backup location, and was created first so as to be on the fastest part of the drive. All of these are backed up to 1TB drives as explained in my original post.
Here is how I organized all of my data.
First I create a main folder that holds everything on a given drive, then sub folders with date and key word, then I create 3 more sub folders within, with the following names RAW, MASTERS, PRINTS. This has worked very well.
Second Copy 8 is set to back up daily, it checks for new files and backs them up and then verifies the backup. All of this runs in the background.
Once a week I pull the drive I have in my desk (its just a bare drive that I keep in a plastic antistatic HD storage box) I plug in my hard drive dock (Toaster) to a USB port and run Second Copy 8, it backs up and verifies all the new files, then I take that drive to my off site location swap that drive with the other bring it home and run a backup on that drive.
I also have a collection of older smaller retired drives off site as well that still have old files on them.
OK, so now you know how I've organized my photos for the last 5 years.
My currant partition has about 700 GB of free space. I could create a new partition for Lr, but I think just a new folder would be easier.
What I gather from all of the reply's so far, I'm still not sure what Lr needs, do I need to change my backup, or will it work with Lr?
Of all the good things I've read about Lr, the catalog/date base backup issue seems to be biggest problems people have.
How should I proceed from here?
-
Bill, that's what many backup utilities do. It's an incremental backup. It backs up new and changed files. Depending on how the software works you may also be able to have files deleted from the origin also deleted in the backup. This should have no impact on how you move files on the origin drive; from within LR or using Windows Explorer.
Lux, whether you choose the location to place the LR backup at one point or another really is irrelevant. There's nothing essentially wrong with the built in LR catalogue backup process.
-
Bob,
Yes, I have Second Copy 8 setup to do incremental backups, I really like the fact that it will verify the backup. I also use Ghost 15 to make images of my OS as well, I also have a clone of the fresh install of my OS stashed away on a small drive as well. Data storage is dirt cheap, and as you can tell, I'm pretty anal about backups. (saved my butt more than once)
I see that I can set Lr to backup the catalog as often as I chose, so, if I tell Lr to backup the catalog to my internal drive, the same location as my images, but a different folder, then Second Copy 8 will backup the internal data drive to my external drive at the end of the day, and all is well, Yes?
This brings up another question, where does Lr create the catalog, C: drive? Or maybe a better question, where should I create this file?
Sorry for all of the questions, I've already spent the last week watching video tutorials on Linda.com
I just received my copy of LightRoom 3 yesterday, so tonight I'm installing it on my Win7 64 PC. Getting up and running should help.
-
On an internal drive.
Have you seen this http://kb2.adobe.com/cps/400/kb400808.html ?
John
-
LR does have a default location for the catalogue. Off the top of my head I don't recall where that is, but it may be the My Pictures folder. It's largely irrelevant; however, because you can tell LR to put the catalogue anywhere you want. The recommended course is to put it on a drive other than the drive the OS and the program sit on (i.e., the C:\ drive). I have 4 drives in my computer and have the LR catalogue on a drive that is otherwise dedicated for nothing else but the LR catalogue and to be the scratch disk.
There's nothing that says you have to use LR to back up the catalogue. You can forego the LR process and have your back up software do it. If you have LR back up its catalogue to an internal drive then have your back up software do its thing, you're essentially getting two back ups of the catalogue. I have LR create the back up of the catalogue but then I explicitly exclude the LR catalogue from my normal Windows back up routine. That way it only gets backed up once. Otherwise it's just eating up space on the back up drive(s) unnecessarily.
-
If you rely on backup software and target your actual Lightroom catalogue you are taking the risk that the catalogue may be open when the scheduled backup kicks in. There's a long history of such backups being unusable.
You also have to fiddle around with your backup software so it doesn't include the potentially-huge but non-critical previews folders.
So it's generally best to let Lightroom make its backups to another drive, and then point your backup software at this location. Notice too that Lightroom's backup routine also includes a catalogue integrity check.
John
-
I just installed Lr 3 last night, updated to 3.5
One question I have, there is a folder call "Lightroom" that's created in the "My Picture folder" what is the purpose of this folder? It can be copied to another folder, but not moved. This is also the default location for the catalog file.
I copied that Lightroom folder that's in the "My Picture folder" over to my internal drive and made a sub folder for the catalog. Not sure if I needed to copy that folder, but I imported my first SD card it all worked. For now I'm just feeling my way around.
John, good point about catalog integrity check, I was just reading about that.
In regards to the backup, I don't think that Second Copy 8 will back any files that are in use buy a application, but there is a box that I can check that tells SC8 to try and backup these files, I have it unchecked.
I'm going experiment with my backups until I'm convinced they are going do what I want them to, and just try to get a good understanding of how file handling in Lr works now that I have it up and running.
<You also have to fiddle around with your backup software so it doesn't include the <potentially-huge but non-critical previews folders.
John, another good point.
The one question I haven't asked is, how do the rest of you deal with Lr backups?
Like said before, I'm really anal about backing up everything, and so far, I've never been burned, but that's just me.
If you rely on backup software and target your actual Lightroom catalogue you are taking the risk that the catalogue may be open when the scheduled backup kicks in. There's a long history of such backups being unusable.
You also have to fiddle around with your backup software so it doesn't include the potentially-huge but non-critical previews folders.
So it's generally best to let Lightroom make its backups to another drive, and then point your backup software at this location. Notice too that Lightroom's backup routine also includes a catalogue integrity check.
John
-
Cut & Paste using right click should work fine.
If for some reason that doesn't work, copy the folder to the location you want it, delete the folder in the original location, find the .lrcat file in the new location, double click to open it and you have your catalogue moved.
-
Make sure Lightroom is closed before you cut and paste the folder.
Bob's point about double clicking is important because if you don't do it, LR may try to create its default catalogue again. I've encountered lots of people who've overlooked this and blame Lightroom for losing their photos and work, when they don't realise they've accidentally created a new catalogue. Be as anal as you like about knowing which lrcat file is your catalogue.
-
Bob,
I'll try that tonight. So what your saying is, the Lightroom file that's created by default in the "My Picture Folder" should only be in one location? Right now, I have in two places, but on two different drives.
Then I assume that the catalog's live inside the Lightroom file, and the Lightroom file can only be in one place, yes?
-
Make sure Lightroom is closed before you cut and paste the folder.
Bob's point about double clicking is important because if you don't do it, LR may try to create its default catalogue again. I've encountered lots of people who've overlooked this and blame Lightroom for losing their photos and work, when they don't realise they've accidentally created a new catalogue. Be as anal as you like about knowing which lrcat file is your catalogue.
Thanks for clearing that up John.
In all of the tutorials I've watched on Lr, I never would have had clue, if it weren't for this forum.
-
Yes, the catalogue should only be in one place. By opening the catalogue via the .lrcat file you're telling Lightroom to reassociate itself with that catalogue file. That prevents the problems John mentioned.
-
There is a General Preference that allows you to specified that Lightroom is to open the last catalog used or to prompt you for the catalog to open. If you use the former on your new catalog, LR will never, I believe, look at the default catalog. I also believe that LR has no memory of prior catalogs.
Alan
-
In all of the tutorials I've watched on Lr, I never would have had clue, if it weren't for this forum.
So, what vid tuts have you seen? Not for nothing but I might suggest the video tutorial Mike and I did on LR 3...I would also suggest Martin Evening's The Lightroom 3 Book (http://www.amazon.com/Adobe-Photoshop-Lightroom-Book-Photographers/dp/0321680707) Note, he is a friend so I might be bias...I would also suggest George Jardine's (http://mulita.com/blog/) Lightroom tutorials as well as Seth Resnick's D-65 (http://www.d-65.com/) and Victoria Bampton's books (http://www.lightroomqueen.com/).
From reading your questions and answers so far, you are a long way from really knowing how to take advantage of Lightroom and before you jump in feet first, it's optimal to learn about what you are planning on doing. The fact is, starting LR only with newly shot images isn't what LR is all about. It's for organizing ALL your digital photography...if you don't do that, then the change over from ACR/Bridge & Photoshop to Lightroom isn't gonna really pay off for you.
-
I believe that's the reason to double click on the moved catalogue file. That's what makes the connection that it's the 'last used catalogue'.
-
From reading your questions and answers so far, you are a long way from really knowing how to take advantage of Lightroom and before you jump in feet first, it's optimal to learn about what you are planning on doing. The fact is, starting LR only with newly shot images isn't what LR is all about.
Though the kind of questions you're asking are the right ones. I don't think it's a bad idea to start off with new images, but pretty soon I think you're going to be at the point where you do understand what your catalogue is and will then want to bring all your pictures under Lightroom's control.
-
OK, now that I have Lr up and running, last night I successfully moved the catalog to my internal drive, and deleted it from the "My Pictures folder"
I now understand what the Lightroom file is all about. I understand Bob's point about double clicking the .lrcat file in the new location, and doing so completes the transfer to my internal drive. I understand how make new catalogs to the location of my choice.
I imported my first SD card, gave it date and location and imported that into my new catalog, then I worked a few files in the develop module before calling it a night.
Looking back, I really should waited until I had Lr installed up and running, work with it, get comfortable with it, get a understanding of the catalog and how it works before I started asking questions about backups.
I'll just let my backup software run as is for now, I'll still be covered.
Jeff, I'm going buy the "Guide To Lightroom 3" I do plan to import my old files, just not right away, when I feel I'm ready.
Thanks for all the help everyone, I'm sure I'll have more questions down the road, but I'm moving foreword now.
-
Just remember when planning your backups, LR doesn't normally change your original files so one backup with a main copy and a redundent 'backup of the backup' copy in a separate location is sufficient. The catalog is where all the data regarding your edits is stored as well as the pointers and other metadata about the files. If you use an incremental backup strategy in order to save copies of the files at various stages of your workflow, the original files are not what are changing but rather it is the data stored in the catalog about those files. Backing up the image files themselves more than once serves no purpose, but backing up the CATALOG at regular intervals is is important because that's where all the work you've done on your images in LR is kept.