Pages: [1] 2   Go Down

Author Topic: Avoiding Importing Duplicates  (Read 753 times)

drralph

  • Full Member
  • ***
  • Offline Offline
  • Posts: 162
    • Ralph's Instagram
Avoiding Importing Duplicates
« on: June 29, 2019, 09:34:29 am »

I am having a hard time making sure all my images have been imported to Lr, while avoiding duplicate imports.  I check the box on the right in the Import panel which says "Don't Import Suspected Duplicates."  Despite this, images I have imported previously appear, checked for import. 

I suspect the problem may lie in the metadata status.  Some of the previously imported images are labeled Metadata Status "Has been changed."  I don't know what was changed.  Clicking on that opens a dialog box offering to save the changes to disk.

How should I manage this issue?

drralph

  • Full Member
  • ***
  • Offline Offline
  • Posts: 162
    • Ralph's Instagram
Re: Avoiding Importing Duplicates
« Reply #1 on: June 29, 2019, 06:19:19 pm »

I may have answered my own question.  It appears that if an image in the catalog has had keywords applied, it will be seen as unique, and the original on the card will be seen as new, and not a duplicate.  Annoying.

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4592
    • My photography site
Re: Avoiding Importing Duplicates
« Reply #2 on: June 30, 2019, 01:16:51 am »

I may have answered my own question.  It appears that if an image in the catalog has had keywords applied, it will be seen as unique, and the original on the card will be seen as new, and not a duplicate.  Annoying.

I don't think that is true and have never seen evidence of it. I suggest you continue looking for patterns because this duplicates check is pretty reliable in my experience.

Rhossydd

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2954
    • http://www.paulholman.com
Re: Avoiding Importing Duplicates
« Reply #3 on: June 30, 2019, 03:25:49 am »

I suggest you continue looking for patterns because this duplicates check is pretty reliable in my experience.
I regularly get issues with this. Mainly with imports from my Sony RX100.
I've never bothered looking too deeply at the problem. I doubt anything I can do will change it's behaviour without breaking my workflow, so I've just treated it as another Adobe weirdness and try to spot the problem before it causes annoyance.
Logged

drralph

  • Full Member
  • ***
  • Offline Offline
  • Posts: 162
    • Ralph's Instagram
Re: Avoiding Importing Duplicates
« Reply #4 on: June 30, 2019, 11:47:32 pm »

Both of the cameras I use currently are Sony as well: RX100V and a7RII.  The issue might have something to do with a Sony idiosyncrasy.

Alan Klein

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • Flicker photos
Re: Avoiding Importing Duplicates
« Reply #5 on: June 30, 2019, 11:57:06 pm »

This happened with my RX100iv.  Then the last time, I walked away from the computer for awhile to do something else.  When I came back, all the ones that were checked but had been duplicates, disappeared from the screen.  It's as if it needed time to figure it out.  I haven't re-checked this situation.  But if you try, report back here  whether this worked for you too.

rdonson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3009
Re: Avoiding Importing Duplicates
« Reply #6 on: July 01, 2019, 08:55:34 am »

Alan may be on to something.

I don’t know how the duplicate search by Lr works but there may be some things that slow it down.

How big is your catalog?  Lots and lots of photos? 

Is the catalog on a fast drive?

How powerful is your computer?  4 cores? 6 cores?  8 cores? 

How much RAM in your computer?
Logged
Regards,
Ron

Alan Klein

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • Flicker photos
Re: Avoiding Importing Duplicates
« Reply #7 on: July 01, 2019, 11:17:21 am »

Off hand,  I think I have the I7 processor with 24gm memory and 256mb ssd.  Don't know how big my catalog is.   One thing.   Some of the filess are short video clips as well as stills that were"duplicates" .  I wonder if videos slow it down.

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 869
    • Chris Kern's Eponymous Website
Re: Avoiding Importing Duplicates
« Reply #8 on: July 01, 2019, 03:50:56 pm »

Adobe has documented how Lightroom determines which imports are duplicates:

Quote
Lightroom Classic determines a photo is a duplicate of another file in the catalog if it has the same, original filename; the same Exif capture date and time; and the same file size.

Apparently all three conditions must be met before a file is recognized as having previously been imported.

Rhossydd

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2954
    • http://www.paulholman.com
Re: Avoiding Importing Duplicates
« Reply #9 on: July 01, 2019, 04:02:28 pm »

Adobe has documented how Lightroom determines which imports are duplicates:
Apparently all three conditions must be met before a file is recognized as having previously been imported.
"Lightroom Classic determines a photo is a duplicate of another file in the catalog if it has the same, original filename; the same Exif capture date and time; and the same file size. You can instruct Lightroom Classic to disregard duplicate files when importing."

As you would expect, but it doesn't always work.
Logged

rdonson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3009
Re: Avoiding Importing Duplicates
« Reply #10 on: July 01, 2019, 04:29:18 pm »

that might suggest that the database has some sort of corruption.  I've never had this issue but I regularly "optimize" the catalog and back it up.
Logged
Regards,
Ron

rdonson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3009
Re: Avoiding Importing Duplicates
« Reply #11 on: July 01, 2019, 04:36:10 pm »

Lightroom uses SQLite as its relational database unless something has changed.  I'm not a DBA so I'm not going to get out any of the SQLite tools and have a look.
Logged
Regards,
Ron

Alan Klein

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • Flicker photos
Re: Avoiding Importing Duplicates
« Reply #12 on: July 01, 2019, 04:39:53 pm »

that might suggest that the database has some sort of corruption.  I've never had this issue but I regularly "optimize" the catalog and back it up.

How do you optimize the catalog?

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4592
    • My photography site
Re: Avoiding Importing Duplicates
« Reply #13 on: July 01, 2019, 05:16:56 pm »

File > Optimize Catalog

Alan Klein

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • Flicker photos
Re: Avoiding Importing Duplicates
« Reply #14 on: July 01, 2019, 06:07:06 pm »

What does Optimizing actually do?

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 15552
    • http://www.digitaldog.net/
Re: Avoiding Importing Duplicates
« Reply #15 on: July 01, 2019, 06:47:45 pm »

What does Optimizing actually do?
See reply #2:
https://www.lightroomqueen.com/community/threads/process-of-optimizing-catalog-what-exactly-is-done.24606/
If it's a Sony issue, it's not all. I haven't seen this issue with my RX10 IV
Logged
Andrew Rodney
Author “Color Management for Photographers"

Alan Klein

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • Flicker photos
Re: Avoiding Importing Duplicates
« Reply #16 on: July 01, 2019, 07:04:05 pm »

See reply #2:
https://www.lightroomqueen.com/community/threads/process-of-optimizing-catalog-what-exactly-is-done.24606/
If it's a Sony issue, it's not all. I haven't seen this issue with my RX10 IV

Thanks.  How would someone know if Optimizing should be done?  I always fear doing things like that as something gets screwed up in the process.  If it's not going to make much of a difference, I'd rather not do it.  Are there tell-tale signs that suggest doing Optimizing?

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 15552
    • http://www.digitaldog.net/
Re: Avoiding Importing Duplicates
« Reply #17 on: July 01, 2019, 07:08:35 pm »

I don't know there is any cut and dry time to Optimize except perhaps if you feel the database like operations are "sluggish".
Logged
Andrew Rodney
Author “Color Management for Photographers"

Alan Klein

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • Flicker photos
Re: Avoiding Importing Duplicates
« Reply #18 on: July 01, 2019, 07:13:50 pm »

I don't know there is any cut and dry time to Optimize except perhaps if you feel the database like operations are "sluggish".

I don't think that's the situation in my case beside the aforementioned "duplicate" issue.  The only "slow" thing is transfering the images which takes awhile.  But that seems to be just normal hardware transfer rates from the camera to the computer.  Otherwise, I find LR and my Adobe Premiere Elements (video) to operate pretty fast.

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 869
    • Chris Kern's Eponymous Website
Re: Avoiding Importing Duplicates
« Reply #19 on: July 01, 2019, 07:37:26 pm »

How would someone know if Optimizing should be done?  I always fear doing things like that as something gets screwed up in the process.

I've never known it to create a problem.  I check the Optimize catalog after backing up option (see attached) every time I back up my catalog . . . and I back up my catalog after every editing session—and during an extensive session, I may make several incremental back-ups.  Since the unoptimized catalog has already been safely stored, I know I can revert to it if anything goes wrong on the next launch of Lightroom, although that has never been necessary.
Pages: [1] 2   Go Up