Avoiding Importing Duplicates

[drralph]

I am having a hard time making sure all my images have been imported to Lr, while avoiding duplicate imports.  I check the box on the right in the Import panel which says "Don't Import Suspected Duplicates."  Despite this, images I have imported previously appear, checked for import. 

I suspect the problem may lie in the metadata status.  Some of the previously imported images are labeled Metadata Status "Has been changed."  I don't know what was changed.  Clicking on that opens a dialog box offering to save the changes to disk.

How should I manage this issue?

[drralph]

I may have answered my own question.  It appears that if an image in the catalog has had keywords applied, it will be seen as unique, and the original on the card will be seen as new, and not a duplicate.  Annoying.

[john beardsworth]

I may have answered my own question.  It appears that if an image in the catalog has had keywords applied, it will be seen as unique, and the original on the card will be seen as new, and not a duplicate.  Annoying.

I don't think that is true and have never seen evidence of it. I suggest you continue looking for patterns because this duplicates check is pretty reliable in my experience.

[Rhossydd]

I suggest you continue looking for patterns because this duplicates check is pretty reliable in my experience.

I regularly get issues with this. Mainly with imports from my Sony RX100.
I've never bothered looking too deeply at the problem. I doubt anything I can do will change it's behaviour without breaking my workflow, so I've just treated it as another Adobe weirdness and try to spot the problem before it causes annoyance.

[drralph]

Both of the cameras I use currently are Sony as well: RX100V and a7RII.  The issue might have something to do with a Sony idiosyncrasy.

[Alan Klein]

This happened with my RX100iv.  Then the last time, I walked away from the computer for awhile to do something else.  When I came back, all the ones that were checked but had been duplicates, disappeared from the screen.  It's as if it needed time to figure it out.  I haven't re-checked this situation.  But if you try, report back here  whether this worked for you too.

[rdonson]

Alan may be on to something.

I don’t know how the duplicate search by Lr works but there may be some things that slow it down.

How big is your catalog?  Lots and lots of photos? 

Is the catalog on a fast drive?

How powerful is your computer?  4 cores? 6 cores?  8 cores? 

How much RAM in your computer?

[Alan Klein]

Off hand,  I think I have the I7 processor with 24gm memory and 256mb ssd.  Don't know how big my catalog is.   One thing.   Some of the filess are short video clips as well as stills that were"duplicates" .  I wonder if videos slow it down.

[Chris Kern]

Adobe has documented how Lightroom determines which imports are duplicates:

Lightroom Classic determines a photo is a duplicate of another file in the catalog if it has the same, original filename; the same Exif capture date and time; and the same file size.

Apparently all three conditions must be met before a file is recognized as having previously been imported.

[Rhossydd]

Adobe has documented how Lightroom determines which imports are duplicates:
Apparently all three conditions must be met before a file is recognized as having previously been imported.

"Lightroom Classic determines a photo is a duplicate of another file in the catalog if it has the same, original filename; the same Exif capture date and time; and the same file size. You can instruct Lightroom Classic to disregard duplicate files when importing."

As you would expect, but it doesn't always work.

[rdonson]

that might suggest that the database has some sort of corruption.  I've never had this issue but I regularly "optimize" the catalog and back it up.

[rdonson]

Lightroom uses SQLite as its relational database unless something has changed.  I'm not a DBA so I'm not going to get out any of the SQLite tools and have a look.

[Alan Klein]

that might suggest that the database has some sort of corruption.  I've never had this issue but I regularly "optimize" the catalog and back it up.

How do you optimize the catalog?

[john beardsworth]

File > Optimize Catalog

[Alan Klein]

What does Optimizing actually do?

[digitaldog]

What does Optimizing actually do?

See reply #2:
https://www.lightroomqueen.com/community/threads/process-of-optimizing-catalog-what-exactly-is-done.24606/
If it's a Sony issue, it's not all. I haven't seen this issue with my RX10 IV

[Alan Klein]

See reply #2:
https://www.lightroomqueen.com/community/threads/process-of-optimizing-catalog-what-exactly-is-done.24606/
If it's a Sony issue, it's not all. I haven't seen this issue with my RX10 IV

Thanks.  How would someone know if Optimizing should be done?  I always fear doing things like that as something gets screwed up in the process.  If it's not going to make much of a difference, I'd rather not do it.  Are there tell-tale signs that suggest doing Optimizing?

[digitaldog]

I don't know there is any cut and dry time to Optimize except perhaps if you feel the database like operations are "sluggish".

[Alan Klein]

I don't know there is any cut and dry time to Optimize except perhaps if you feel the database like operations are "sluggish".

I don't think that's the situation in my case beside the aforementioned "duplicate" issue.  The only "slow" thing is transfering the images which takes awhile.  But that seems to be just normal hardware transfer rates from the camera to the computer.  Otherwise, I find LR and my Adobe Premiere Elements (video) to operate pretty fast.

[Chris Kern]

How would someone know if Optimizing should be done?  I always fear doing things like that as something gets screwed up in the process.

I've never known it to create a problem.  I check the Optimize catalog after backing up option (see attached) every time I back up my catalog . . . and I back up my catalog after every editing session—and during an extensive session, I may make several incremental back-ups.  Since the unoptimized catalog has already been safely stored, I know I can revert to it if anything goes wrong on the next launch of Lightroom, although that has never been necessary.