Pages: [1]   Go Down

Author Topic: Using the catalog with filters is slow and inconsistent  (Read 2007 times)

scrollop

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Using the catalog with filters is slow and inconsistent
« on: June 10, 2018, 09:59:11 am »

I've imported 220 000 (aprox 4.4 tb) photos/videos into a catalog (system: win 10 x64, 24gigs, 4th gen i7, GTX 970, C1 11.1.1 catalog on an SSD). That took a few hours to import and 24 hours to make 900x previews.

There are major issues when attempting to use the catalog.

- It's inconsistent. When selecting a filter of f1.4, for example, where alongside the filter it states there are 15 000 images, perhaps 2000 images appear (slowly). Switching to other filters does not show the correct amount of photos in the browser vs the amount alongside the filter. Sometimes, when changing from one filter to another, nothing happens.

- It's slow. It takes a long time to import photos, and after you import them, the system appears to review every photo when populating the filter figures.

- When selecting one filter eg. 35mm (though the system doesn't show the correct amount of photos) and another filter eg f1.4 (using CTRL), the second filter selection does not refine the results ie. nothing happens.

On paper the filter system appears powerful. In practice, IMHO, it is inconsistent and slow and thus unusable.

If you have any recommendations on how to improve performance, I'd appreciate it.


Logged

scrollop

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #1 on: June 10, 2018, 12:23:48 pm »

Also, when selecting f1.4, for example, which apparently has 15000 photos, why does it only show 2 photos?
After waiting 30 seconds or so, slowly photos are added.
I try another aperture - nothing happens for 30 seconds. Then it starts at 30 photos and slowly adds them (maybe 10 - 20 photos per second, though it pauses every 2 -5 seconds, then adds more). Apparently the new aperture has 10 000+ photos. How long will it take to show all the photos that I have selected?

This is after it spent 24+ hrs importing the photos in the first place.
It's unusable. Perhaps it is due to the size of the catalog. In which case, it's unusable with catalogs above a certain size on moderately powerful systems.


Oh, and as it's searching the disk and populating the filter I chose, it's now using 3 gigs in memory and 20% CPU. LR would have accomplished this within 10 seconds.
Logged

Hoggy

  • Full Member
  • ***
  • Offline Offline
  • Posts: 207
  • Never take life, or anything in it, too seriously.
Re: Using the catalog with filters is slow and inconsistent
« Reply #2 on: June 12, 2018, 01:03:31 am »

Yep.

That's one of the biggest reasons I wouldn't switch from LR - as I'm totally a catalog person.  Not only is that CO behavior happen with filters - but also any smart albums as well.  Simply... Just.. Unusable.  Period.

It's beyond me how they still haven't fixed such a basic usability flaw after so many years.  I just have no words to explain it anymore......
Logged
Cams: Pentax K-3, K-30 & Canon G7X, S100
Firm supporter of DNG, throwing away originals.
It's the hash, man..  That good hash!

scrollop

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #3 on: June 13, 2018, 08:57:31 am »

I read somewhere that the problem could be that I selected 900px previews and C1 was scaling them for the screen size (4k).

It was recommended to use previews the size of your monitor or larger. I selected larger previews, let it alone for 2 days to regenerate them and have come back to see how it's going.

C1 was using 14 gigs of RAM and pretty much at a standstill. It had used the remaining 150 gb of storage space on my second SSD and had generated 40% of the previews.

Perhaps I am doing something wrong here. Perhaps you need large SSDswith 400gigs of space just for the previews and the latest hardware.

Or, perhaps C1 is not designed for large catalogs, which is my conclusion. Am happy to be proven wrong.
Logged

UnfamiliarLight

  • Newbie
  • *
  • Offline Offline
  • Posts: 30
    • Unfamiliar Light
Re: Using the catalog with filters is slow and inconsistent
« Reply #4 on: June 13, 2018, 11:16:11 am »

Or, perhaps C1 is not designed for large catalogs, which is my conclusion. Am happy to be proven wrong.
Well it is supposed to be designed for large catalogues. Looks like some proof is needed :-) What does support say on the matter?
Logged

chadchat

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #5 on: June 25, 2018, 04:02:10 pm »

I agree the time it takes to load a smart album is not good.
Logged

mrenters

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 80
    • http://www.teckelworks.com/
Re: Using the catalog with filters is slow and inconsistent
« Reply #6 on: August 03, 2018, 04:49:35 pm »

Catalogs don't perform very well (or reliably for that matter) in Capture One. I've imported my old Lightroom catalogs into Capture One and have approximately 30,000 images in the catalog now. Both the images and catalog live on a NAS drive and the performance is terrible in Capture One while being reasonable in other software line ON1 Photo Raw.

As a software developer I've tried to figure out why this is happening and have found that Capture One seems to love to frequently traverse all the image files in a directory, presumably to check whether it has previews or whether meta data is up-to-date. This can tie up Capture One for extremely long periods of time and locks up basic filtering. The issue doesn't seem to be that the underlying SQL database is unable to answer the question of how many files meet the criteria, but that Capture One is too busy doing other things to get the answer.

Filtering all IIQ files from a folder containing 3400 files can take minutes, although this could be answered in at most a second by the database.

I *really* hope that Phase One gets their act together and dramatically improves performance in the next version. I haven't come across any other software that is this slow at doing basic cataloging functions. On top of that there are a number of bugs where it shows a preview for an entirely different image until you make a small adjustment and then it gets it right.

Martin
Logged

Dave Rosser

  • Full Member
  • ***
  • Offline Offline
  • Posts: 228
    • My Website
Re: Using the catalog with filters is slow and inconsistent
« Reply #7 on: August 04, 2018, 09:33:20 am »

Catalogs don't perform very well (or reliably for that matter) in Capture One. I've imported my old Lightroom catalogs into Capture One and have approximately 30,000 images in the catalog now. Both the images and catalog live on a NAS drive and the performance is terrible in Capture One while being reasonable in other software line ON1 Photo Raw.

As a software developer I've tried to figure out why this is happening and have found that Capture One seems to love to frequently traverse all the image files in a directory, presumably to check whether it has previews or whether meta data is up-to-date. This can tie up Capture One for extremely long periods of time and locks up basic filtering. The issue doesn't seem to be that the underlying SQL database is unable to answer the question of how many files meet the criteria, but that Capture One is too busy doing other things to get the answer.

Filtering all IIQ files from a folder containing 3400 files can take minutes, although this could be answered in at most a second by the database.

I *really* hope that Phase One gets their act together and dramatically improves performance in the next version. I haven't come across any other software that is this slow at doing basic cataloging functions. On top of that there are a number of bugs where it shows a preview for an entirely different image until you make a small adjustment and then it gets it right.

Martin
+1 to all that.  I have given the catalogue function in Capture One many tries and have at last given up on it, it is just too slow and flaky.  I have reverted to using lightroom for cataloguing and run of the mill editing and use Capture one in session mode for images that need Capture One's particular strengths.  In session mode I simply brows to the picture I want to edit in the Library/System Folders - I make no use at all of Session Folders or Session Albums.  Capture One stores all my edits etc in a sidecar file in the directory in which the RAW file resides.
Logged

Doug Peterson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • http://www.doug-peterson.com
Re: Using the catalog with filters is slow and inconsistent
« Reply #8 on: August 04, 2018, 11:26:57 am »

Based on 11 years of teaching Capture One classes and consulting with photographers and companies, the majority of C1 users that are currently using catalogs are doing so because they used catalogs in LightRoom (the only option in LR) and would choose to primarily use sessions if they had the pros and cons fully explained to them. As a consequence we wrote this article: Capture One Session vs Catalogs to help users decide which is right for them.

That's not to say that I know for sure you specifically would be better of with sessions. Just that it's definitely worth your time and consideration.

IanSeward

  • Full Member
  • ***
  • Offline Offline
  • Posts: 108
Re: Using the catalog with filters is slow and inconsistent
« Reply #9 on: August 06, 2018, 05:23:16 pm »

I used LR since V1 and therefore only thought in catalogues. :-)

I now use sessions and much prefer it. In LR I always "found" my images by going to the folder in LR.  I realise others "live or die by the catalogue" for whatever reason. I import the sessions into a catalogue at the end of a year out of habit, but rarely actually use it.

I just opened up an old "Master Catalogue" imported form LR and then added to.  It opened in around 15 sec and filtered 1650 red images in a few seconds.

I am using a 5 year old desktop computer with 16Gb ram, I73820 @ 4.3GHz and a Nvidia 1060 6Gb video card.

What I have noticed with C1Pro is that if you have any corrupt images where the Event viewer says "cannot build preview" then C1 Pro is slow. It appears to check every image in the catalogue each time it starts to make sure the images are OK.  After you have removed the faulty images it still appears to check each file for the next 2 start ups and then it is fine.

It might be worth checking the Event viewer to see if it flags up any issues.

The other suggestion is to export the catalogue to a new catalogue, as this can sometimes "sort out" unexplained issues with the catalogues integrity. You need to recreate the previews but this can be an overnight job.

Hope this helps.

Ian
Logged

Hoggy

  • Full Member
  • ***
  • Offline Offline
  • Posts: 207
  • Never take life, or anything in it, too seriously.
Re: Using the catalog with filters is slow and inconsistent
« Reply #10 on: August 12, 2018, 12:05:29 am »

What I have noticed with C1Pro is that if you have any corrupt images where the Event viewer says "cannot build preview" then C1 Pro is slow. It appears to check every image in the catalogue each time it starts to make sure the images are OK.  After you have removed the faulty images it still appears to check each file for the next 2 start ups and then it is fine.

It might be worth checking the Event viewer to see if it flags up any issues.

The other suggestion is to export the catalogue to a new catalogue, as this can sometimes "sort out" unexplained issues with the catalogues integrity. You need to recreate the previews but this can be an overnight job.

I did this on my circa 2012 4-core AMD A8-3500M APU w/8GB-maxed-ram laptop before moving to my new supercomputer.  In particular, I removed all the LR-merged dng files from the C1 catalog.  (btw, I'm a complete and total catalog person here.)
It was a bit faster, but still dog slow when moving between collections - with catalog and previews being on a SATA SSD, of course.  I didn't try the export to new catalog though - maybe I'll try that on that laptop.

Now on my new supercomputer, C1 is finally not super slow - but still MUCH slower than LR on this same machine.  This is with a reasonably all-core overclocked AMD Threadripper 16/32-core with 128GB of 2933Mhz RAM and an 8GB Nvidia Quadro P4000 video card, with catalog and previews on a ~1TB Intel Optane 905P and images on a ~1TB Samsung 970 Pro SSD (both NvMe PCIE x4 SSD's).

However, the fact remains that one should never **NEED** a $10,000 PC just to say a certain desktop software package is finally not super slow.  They really ought to get catalog speeds fixed one of these decades, as opposed to continually looking at it as a benefit of sessions.
(Hey, I went 8 years with $0 income, so I might finally kinda sorta deserve a bit of bragging rights.  :P ;D )
Logged
Cams: Pentax K-3, K-30 & Canon G7X, S100
Firm supporter of DNG, throwing away originals.
It's the hash, man..  That good hash!

chadchat

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #11 on: January 22, 2019, 03:45:17 pm »

(Sorry to resurrect an old post)

I would presume that for some people the issue with using sessions is how to use them in a way that allows you to browse through your images over a period of months or longer, and how to define what goes in them.
If I wait until my card is full as I often do, and on that card are some shots I took for a project I'm working on from time to time, a mountain of images of my daughter, and some images from some outings that I think might become a new project... am I supposed to create 3 sessions, one for each theme? Should I create a session for each month or each card? How would I then see across the months to cross reference or group images?
Perhaps I'm just missing the point, but if sessions are the answer, I'd love to understand how to use them within my own workflow.
It was a lot simpler in Lightroom where you had potentially all the images you'd ever taken that you could quickly browse through, and then the collections you'd created.
Logged

myotis

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 262
Re: Using the catalog with filters is slow and inconsistent
« Reply #12 on: January 22, 2019, 05:33:09 pm »

(Sorry to resurrect an old post)
It was a lot simpler in Lightroom where you had potentially all the images you'd ever taken that you could quickly browse through, and then the collections you'd created.

Lots of people do exactly this with C1, you don't "have" to use sessions, but there are advantages of doing so, as explained in the guidance on sessions vs catalogues that was linked to earlier in the post.

Session are ideal for work that involves discrete projects and are probably more suited to professional work where work is often split into clearly identified projects e,g. The Jones's wedding 2017,  or the Sarah Williams portrait June 2015. Or you might have a project that you work on over several years e.g. London Street Photography project 2014 -2018.  Or at a hobby level, you may set up a project/session for things like "holiday 2017" or a project/session for all photographs of your daughter.  Each project gets it's own session, which makes it easy to find, easy to archive and back up, and easy to move between computers because all files related to that project are store within a single file hierarchy within a single folder.

If, on the other hand you are something like a bird photographer, and you have thousands of bird photographs taken over several years and at several locations, a catalogue will work better as you may well want to search for every picture you have ever taken of a specific bird species, in a specific habitat, or/and showing a specific behaviour.

What you choose, depends on how you work. Some people have never used a catalogue( or sessions)and rely on labelling system folders with appropriate names, which they use to find files to open directly into Photoshop, or other editing software. Others have complex search needs and a catalogue is important. As an ex-professional photographer, sessions make a lot of sense for the project based professional work that I used to do.

For the average amateur photographer, who doesn't have a tight structure to their work, I suspect a catalogue is easier to manage than sessions. Many photographs are likely to fall into multiple categories. e.g photographs of your daughter taken during your summer holiday 2017.  Do you put those photographs into a session for "daughter growing up project" or a session for your "summer holiday 2017 project".

C1 gives a lot of flexibility in letting you manage your files in a way that matches your working practices and needs, but it can take a bit of time to get your head around the options. The guidance on sessions mentioned earlier is good, as are the many webinars that Phase One have on the topic.

But, you certainly don't "have to" use sessions if a catalogue is a better match to your photography.  If you do want to use them, than I think you need to think about how you might split your photography up into logical projects, where this would make it easier to find files, and not more difficult.

Cheers,

Graham



Logged

IanSeward

  • Full Member
  • ***
  • Offline Offline
  • Posts: 108
Re: Using the catalog with filters is slow and inconsistent
« Reply #13 on: January 23, 2019, 03:01:53 am »

(Sorry to resurrect an old post)

I would presume that for some people the issue with using sessions is how to use them in a way that allows you to browse through your images over a period of months or longer, and how to define what goes in them.
If I wait until my card is full as I often do, and on that card are some shots I took for a project I'm working on from time to time, a mountain of images of my daughter, and some images from some outings that I think might become a new project... am I supposed to create 3 sessions, one for each theme? Should I create a session for each month or each card? How would I then see across the months to cross reference or group images?
Perhaps I'm just missing the point, but if sessions are the answer, I'd love to understand how to use them within my own workflow.
It was a lot simpler in Lightroom where you had potentially all the images you'd ever taken that you could quickly browse through, and then the collections you'd created.

The way your post reads is that you don't transfer your images to the computer until your SD card is full. This is a risky strategy as a SD card is a relatively fragile and you do not have a backup until they are on your hard disk and subject to whatever backup strategy you employ.

Ian
Logged

chadchat

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #14 on: January 23, 2019, 04:19:37 am »

To be honest I don't usually wait until my card is full but this time around I did because my pc was out for repairs for a few months, hence jumping back onto an old post.

I believe I do have a grasp of how catalogs vs sessions function in C1, but like others using a NAS, catalog performance, particularly when using filters, often isn't up to it. I guess I find myself returning to this topic because I really want it to work smoothly, and not to have to return to Lightroom.
Logged

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 165
Re: Using the catalog with filters is slow and inconsistent
« Reply #15 on: January 23, 2019, 05:09:10 am »

To hijack this thread even more, I am finding Lightroom extremely slow, and I am considering alternatives. I take it C1 is even slower??
Logged

chadchat

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #16 on: January 23, 2019, 09:59:31 am »

My complaint is just filtering 'collections' you've defined from within a large catalog. Over the network on my NAS at least it has me waiting more than I would like.
The general workflow with images though is fast and excellent IMHO.
Logged

Doug Peterson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3983
    • http://www.doug-peterson.com
Re: Using the catalog with filters is slow and inconsistent
« Reply #17 on: January 23, 2019, 02:34:30 pm »

To hijack this thread even more, I am finding Lightroom extremely slow, and I am considering alternatives. I take it C1 is even slower??

C1 is super fast, assuming a good computer (fast SSD and good GPU) and the use of sessions.

With very large catalogs C1 is not super fast, especially if the catalog and/or assets are stored externally, but to a lesser extent even if they aren't.

But regardless, most users are best served with sessions anyway (see also: C1 Sessions Vs Catalogs)

chadchat

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
Re: Using the catalog with filters is slow and inconsistent
« Reply #18 on: February 06, 2019, 02:05:09 am »

I want to follow up by saying that I bought an SSD to work locally from my PC rather than on mechanical drives over the local network on my NAS, and the performance of filtering through catalogs etc is greatly improved, and is around about what I'd hoped for. This works for me.
Logged
Pages: [1]   Go Up