Topic: Lightroom Library 'Missing Files' (Read 9434 times)

Josh-H · « **on:** May 04, 2008, 10:26:31 pm »

Has anyone else noticed that in the Library pane if you synchronise a folder you get a collection titled 'Missing Files'?

As an example - if you have a folder with 100 files in it. You work on a file in CS3, then save it under a new name - you will now have 101 files in that folder.

If you then synchronise the folder in Lightroom 'Library / Synchronise Folder' Lightroom will find the new file and import it to stay in sync and will now show 101 files as it should.

However - Lightroom ALWAYS creates a collection under 'Previous Import' in the left tab called 'missing files 'Which is the other 100 files'

Very annoying.

Anyone else noticed this or know how to stop it?

CatOne · « **Reply #1 on:** May 05, 2008, 12:02:03 am »

Mine says I have 6,289 missing files.

But I don't think they're missing.

I have been unable to fix the issue, and I've been unable to get any real acknowledgment of what the issue is. Really rather annoying.

And yes, synchronizing the folders caused the issue. I haven't resolved it, I've just resigned myself to living with it. Actually, I've been using Aperture more again lately.

Josh-H · « **Reply #2 on:** May 05, 2008, 12:58:03 am »

Quote

Mine says I have 6,814 missing files.

But I don't think they're missing.

I have been unable to fix the issue, and I've been unable to get any real acknowledgment of what the issue is. Really rather annoying.

And yes, synchronizing the folders caused the issue. I haven't resolved it, I've just resigned myself to living with it. Actually, I've been using Aperture more again lately.
[a href=\"index.php?act=findpost&pid=193513\"][{POST_SNAPBACK}][/a]

Thanks for Posting. I think you are right - the files are not actually missing as they are clearly there in lightroom right where they should be. Lightroom is just creating a collection for some reason of files it is labelling as 'missing'.

Matt Koslowskit just posted in the last few days a video tip on his website 'Adobe Lightroom Killer Tips' on Synchronising folders - it shows up also in his video 'Missing Files' when he demonstrates synchronising folders - but he makes no mention of it.

I left a comment on his site - but he hasnt responded. I also emailed Adobe with no response thus far.

Since it clearly an issue across the board and not specific to my install I would really like to hear from someone who has an explanation for why it occurs and how to 'clean it up'.

DavidB · « **Reply #3 on:** May 05, 2008, 09:17:06 am »

I haven't been able to reproduce this behaviour with either 1.4.1 or 2.0.

Quote

Matt Koslowskit just posted in the last few days a video tip on his website 'Adobe Lightroom Killer Tips' on Synchronising folders - it shows up also in his video 'Missing Files' when he demonstrates synchronising folders - but he makes no mention of it.

No.
His catalog has a bunch of Missing Files, but that probably has something to do with the missing folders visible in his Folders panel... We don't actually see the Missing Files increment during the video.

Quote

Since it clearly an issue across the board and not specific to my install I would really like to hear from someone who has an explanation for why it occurs and how to 'clean it up'.[a href=\"index.php?act=findpost&pid=193519\"][{POST_SNAPBACK}][/a]

Is it really an issue "across the board"?

CatOne · « **Reply #4 on:** May 05, 2008, 11:17:00 am »

Quote

I haven't been able to reproduce this behaviour with either 1.4.1 or 2.0.

I certainly have. And I can't fix it; nobody has been able to tell me how to fix it. I've seen this posted on forums here, I've seen it posted on Adobe's forums, and I've seen it posted on lightroom-forums.com. Never answered.

Quote

Is it really an issue "across the board"?
[a href=\"index.php?act=findpost&pid=193555\"][{POST_SNAPBACK}][/a]

Maybe not. But it sure would be nice if SOMEONE had an idea how to fix this. Will it eventually be all my files? Should "synchronize files" be removed from the product if it's broken in this way?

Josh-H · « **Reply #5 on:** May 05, 2008, 10:24:13 pm »

Quote

I certainly have. And I can't fix it; nobody has been able to tell me how to fix it. I've seen this posted on forums here, I've seen it posted on Adobe's forums, and I've seen it posted on lightroom-forums.com. Never answered.

Same - Ive seen it posted before elsewhere as well - still waiting for an answer.

Whilst it may not effect everyone - its certainly seems fairly prevalent.

Adobe are u out there? Are you listening?.....

DavidB · « **Reply #6 on:** May 06, 2008, 03:24:10 am »

Can you describe how to repeat the problem? The key to getting a fix from Adobe will be to reproduce it.
Josh's description at the start of this thread doesn't do it for me. Is it a Windows-only thing? (my LR machines are all Mac) Is it related to the filesystem? (are the files on FAT32/HFS/NTFS disks?)
If someone can produce a minimal catalog that exhibits the problem then that could be provided as a sample/test case.

Josh-H · « **Reply #7 on:** May 06, 2008, 06:29:14 am »

Quote

Can you describe how to repeat the problem? The key to getting a fix from Adobe will be to reproduce it.
Josh's description at the start of this thread doesn't do it for me. Is it a Windows-only thing? (my LR machines are all Mac) Is it related to the filesystem? (are the files on FAT32/HFS/NTFS disks?)
If someone can produce a minimal catalog that exhibits the problem then that could be provided as a sample/test case.
[a href=\"index.php?act=findpost&pid=193730\"][{POST_SNAPBACK}][/a]

Ill try again to describe the process and what happens.

Lets say you have a folder with 100 images in it you have been working on in Lightroom. You select one of these images and choose 'edit in Photoshop'. Make sure you choose to create a Tiff for this example and stack it with the original so that you now have 101 files in this folder.

Now when in CS3 with this Tiff file open choose 'Save As' and save it under a new name in the same folder. Now you will have 102 files in that folder, but Lightroom will currently only see 101 as the new saved file has not yet been imported.

Now in Lightroom choose to Synchronise the folder. Lightroom will find the new file you just saved [the 102nd file] and import it. It will then create a collection called 'Missing Files' [as in the jpeg posted above] which will be similar to a quick collection of the other 101 files in this folder. This is just a link to those files. The files are not actually missing as far as I can tell. But.. its impossible to remove this 'Missing Files' collection.

This is using Lightroom on a Very fast Windows Vista 32 bit machine. My Library is on a USB attached 2 Terrabyte NAS drive.

aduke · « **Reply #8 on:** May 06, 2008, 12:44:46 pm »

Quote

Ill try again to describe the process and what happens.

Lets say you have a folder with 100 images in it you have been working on in Lightroom. You select one of these images and choose 'edit in Photoshop'. Make sure you choose to create a Tiff for this example and stack it with the original so that you now have 101 files in this folder.

Now when in CS3 with this Tiff file open choose 'Save As' and save it under a new name in the same folder. Now you will have 102 files in that folder, but Lightroom will currently only see 101 as the new saved file has not yet been imported.

Now in Lightroom choose to Synchronise the folder. Lightroom will find the new file you just saved [the 102nd file] and import it. It will then create a collection called 'Missing Files' [as in the jpeg posted above] which will be similar to a quick collection of the other 101 files in this folder. This is just a link to those files. The files are not actually missing as far as I can tell. But.. its impossible to remove this 'Missing Files' collection.

This is using Lightroom on a Very fast Windows Vista 32 bit machine. My Library is on a USB attached 2 Terrabyte NAS drive.
[a href=\"index.php?act=findpost&pid=193745\"][{POST_SNAPBACK}][/a]

I just tried this on a laptop running Vista. Starting with a folder of 24 images, exported one to CS3, saved as a new TIFF, synchronized the folder.

LR now reports 26 images in the folder and no missing files.

Perhaps LR is synchronizing all folders and has found files missing in others? The menu item does say "Synchronize folder...", maybe it means all folders?

Alan

DavidB · « **Reply #9 on:** May 06, 2008, 07:58:18 pm »

Quote

Ill try again to describe the process and what happens.

Lets say you have a folder with 100 images in it you have been working on in Lightroom. You select one of these images and choose 'edit in Photoshop'. Make sure you choose to create a Tiff for this example and stack it with the original so that you now have 101 files in this folder.

Now when in CS3 with this Tiff file open choose 'Save As' and save it under a new name in the same folder. Now you will have 102 files in that folder, but Lightroom will currently only see 101 as the new saved file has not yet been imported.

Now in Lightroom choose to Synchronise the folder. Lightroom will find the new file you just saved [the 102nd file] and import it. It will then create a collection called 'Missing Files' [as in the jpeg posted above] which will be similar to a quick collection of the other 101 files in this folder. This is just a link to those files. The files are not actually missing as far as I can tell. But.. its impossible to remove this 'Missing Files' collection.[a href=\"index.php?act=findpost&pid=193745\"][{POST_SNAPBACK}][/a]

I've followed these instructions exactly in 1.4.1, to no avail. Everything works as it should.
This is with the library and the files on local disk on OS X, and again with the files across the network on a ReadyNAS (the library's always on local disk).

Quote

This is using Lightroom on a Very fast Windows Vista 32 bit machine. My Library is on a USB attached 2 Terrabyte NAS drive.

Is it NAS (Network-Attached Storage: i.e. connected via Ethernet) or is it a USB drive?

By the way, the "Edit in" functions have changed slightly in LR 2, so the problem might be gone by the time the "real" 2.0 (or another beta) comes out. In any case I can't reproduce your problem on 2.0 either.

Josh-H · « **Reply #10 on:** May 07, 2008, 07:05:33 am »

Quote

I've followed these instructions exactly in 1.4.1, to no avail. Everything works as it should.
This is with the library and the files on local disk on OS X, and again with the files across the network on a ReadyNAS (the library's always on local disk).
Is it NAS (Network-Attached Storage: i.e. connected via Ethernet) or is it a USB drive?

By the way, the "Edit in" functions have changed slightly in LR 2, so the problem might be gone by the time the "real" 2.0 (or another beta) comes out. In any case I can't reproduce your problem on 2.0 either.
[a href=\"index.php?act=findpost&pid=193981\"][{POST_SNAPBACK}][/a]

Well thank you for trying - but I definately still have this issue.

My NAS is attached via USB 2.0

john beardsworth · « **Reply #11 on:** May 07, 2008, 07:42:04 am »

deleted

Nat Coalson · « **Reply #12 on:** May 07, 2008, 11:11:04 am »

The records in the database are somehow screwed up for those images. Could have happened at many points in the workflow, but the problem is bad data.

The fix: remove them from the catalog and re-import.

1. Show the Missing Files source
2. Press Cmd-A/Ctrl-A to select all the images
3. Press Cmd-S/Ctrl-S to save out the metadata to all the images*

*Very Important - Make sure to save out your Metadata before removing files from the catalog.

4. Press Delete; in the alert box choose Remove
5. Synchronize the folders with the affected images (or do another Import).**

**Very Important - make sure to show the Import dialog box and make sure NO settings are being applied to the newly imported images!!

You should end up with the complete set of images for the folder with all your previous work intact.

And the Missing Files source will be gone.

Test, backup and optimize your database regularly!

CatOne · « **Reply #13 on:** May 07, 2008, 02:32:28 pm »

Quote

The records in the database are somehow screwed up for those images. Could have happened at many points in the workflow, but the problem is bad data.
...

Test, backup and optimize your database regularly!
[a href=\"index.php?act=findpost&pid=194138\"][{POST_SNAPBACK}][/a]

Well I'm 98% sure that using "synchronize files" is what caused it. I'm on a Mac Pro, running 10.5.x. I had gone for a couple month with no issues with LR, but after I did a couple "synchronize files" operations on some folders (to try to snare about 150 files that I had deleted from the LR library but not from disk -- so I could delete them), this issue flared up.

That's a pretty ugly workaround, a full export/delete from the library and a re-import, especially considering there are a few things (discussed by Andrew Rodney) that are stored in the Library but which will not be stored in the XMP sidecars (or DNG representation of same).

DavidB · « **Reply #14 on:** May 07, 2008, 07:06:31 pm »

Quote

That's a pretty ugly workaround, a full export/delete from the library and a re-import, especially considering there are a few things (discussed by Andrew Rodney) that are stored in the Library but which will not be stored in the XMP sidecars (or DNG representation of same). [a href=\"index.php?act=findpost&pid=194195\"][{POST_SNAPBACK}][/a]

Agreed, but it's pretty much guaranteed to remove corruption of those records in the .lrcat.

An intermediate step you could try is to export those images into a temporary catalog. That should pass over the things that aren't stored in XMP (Collections, Virtual Copies, etc). Then remove the images from the original catalog, test it for integrity, optimise it, etc. Then import the images back from the temporary catalog.
This *might* preserve any "screwed-up-ness" of the records, but it's worth a try.

Nat Coalson · « **Reply #15 on:** May 07, 2008, 07:35:19 pm »

Ugly as it may be, using Lightroom as the core of the workflow frequently requires adding and removing images from the database. An efficient workflow provides for this and ensures that *most* of your work will remain intact.

Maybe Optimizing will fix some of those corrupt records.

In any case, based on my past experience with various databases, if you've got bad data in there you often have to recreate those records one way or another; there may be no way to "fix" it. But the files themselves should be safe if you've saved everything along the way.

Admittedly, if you have 9 images showing in the Missing Files collection, it's probably not a big deal to remove them and re-import. If you have thousands images showing as Missing, you've got bigger problems and the database really needs attention.

Maintaining a clean LR catalog takes persistent work. I take it for granted that I might have to redo a few things along the way. Ugly workarounds are sometimes the prettiest solution in a production environment.

Josh-H · « **Reply #16 on:** May 08, 2008, 06:31:00 am »

Quote

The records in the database are somehow screwed up for those images. Could have happened at many points in the workflow, but the problem is bad data.

The fix: remove them from the catalog and re-import.

1. Show the Missing Files source
2. Press Cmd-A/Ctrl-A to select all the images
3. Press Cmd-S/Ctrl-S to save out the metadata to all the images*

*Very Important - Make sure to save out your Metadata before removing files from the catalog.

4. Press Delete; in the alert box choose Remove
5. Synchronize the folders with the affected images (or do another Import).**

**Very Important - make sure to show the Import dialog box and make sure NO settings are being applied to the newly imported images!!

You should end up with the complete set of images for the folder with all your previous work intact.

And the Missing Files source will be gone.

Test, backup and optimize your database regularly!
[a href=\"index.php?act=findpost&pid=194138\"][{POST_SNAPBACK}][/a]

Thank you - I did this and it has solved the problem of the missing files.

However - it seems LR has not saved my star ratings and images that were flagged no longer appear flagged. Likewise, images that were stacked are now unstacked. Something also appears a little screwy with the keywords - some files imported without their keywords.

Is there a way to get this information back?

Edit - oh and virtual copies were lost as well.

I can easily re-keyword, re flag and re-star, but virtual copies lost is a bit of a bug bear.

The good news is: I synchronised another folder without the problem re-occuring.

john beardsworth · « **Reply #17 on:** May 08, 2008, 06:38:32 am »

The flags are lost - they are "local" and exist only in the context of the folder or collection where they were created. The same for stacks. Both are deficiencies of the ugly workaround - one reason why I wouldn't have recommended it at this stage (you'd be better off following the Export as Catalog route). However, the star ratings and keywords should have been saved out in the xmp and read upon import, and I've not seen any cases of their not doing unless they were applied to virtual copies or the user forget to select all before writing out the xmp files. Did you keep your faulty catalogue as a backup?

John

Josh-H · « **Reply #18 on:** May 08, 2008, 06:49:45 am »

Quote

The flags are lost - they are "local" and exist only in the context of the folder or collection where they were created. The same for stacks. Both are deficiencies of the ugly workaround - one reason why I wouldn't have recommended it at this stage (you'd be better off following the Export as Catalog route). However, the star ratings and keywords should have been saved out in the xmp and read upon import, and I've not seen any cases of their not doing unless they were applied to virtual copies or the user forget to select all before writing out the xmp files. Did you keep your faulty catalogue as a backup?

John
[a href=\"index.php?act=findpost&pid=194340\"][{POST_SNAPBACK}][/a]

Yes - kepy the faulty catalogue as a back-up.

It looks like LR only wrote out XMP files for images that had develop settings applied. In other words, it kept the keywords for those files, but not the others. Weird...

My virtual copies were also wipped out.

CatOne · « **Reply #19 on:** May 08, 2008, 07:02:25 am »

Quote

Yes - kepy the faulty catalogue as a back-up.

It looks like LR only wrote out XMP files for images that had develop settings applied. In other words, it kept the keywords for those files, but not the others. Weird...

My virtual copies were also wipped out.
[a href=\"index.php?act=findpost&pid=194341\"][{POST_SNAPBACK}][/a]

All this from using the 'synchronize files' command to try and pick up a few files I deleted from the library but clicked the wrong dialog box when it asked if I wanted to delete the files off disk.

Maybe Aperture's managed library isn't such a bad idea after all

Author Topic: Lightroom Library 'Missing Files' (Read 9434 times)