Pages: [1] 2   Go Down

Author Topic: Lightroom performance with large files  (Read 6655 times)

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4237
    • waynefox.com
Lightroom performance with large files
« on: May 05, 2012, 01:53:00 am »

Just wondering if this is normal. I've got a pretty decent system and I don't understand why moving from one file to another in Lightroom and then back again takes so long to load each image in the develop module.  12 core MacPro, 10.7.3,  PCI/SSD for startup/application drive, lightroom library and data on 4 7200 drives raid 0.  64 gigs of Ram.  Tried moving the library to the SSD, doesn't help.

When I select an IQ180 file I get the "loading" message for 12 to 14 seconds.  If I select another image, same loading message.  If I then immediately go back to the first image, it takes the same amount of time, even though all of that was just load a few seconds before.

Works a little better in the Library module, but there are often times I'm moving between two different files trying to tweak them for an HDR stack in photoshop.  Seems like I have enough ram and speed things would at least be a little quicker the second time.  Seems I get a full render from scratch whenever I selected an image in the develop module.

Normal or should I start looking for issues?
Logged

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: Lightroom performance with large files
« Reply #1 on: May 05, 2012, 02:12:50 am »

When I select an IQ180 file I get the "loading" message for 12 to 14 seconds.  If I select another image, same loading message.  If I then immediately go back to the first image, it takes the same amount of time, even though all of that was just load a few seconds before.

For an IQ 180 that LR 4.1 RC2 has never seen, it takes about 8-9 seconds to go from Library to Develop. If I go back to Library and then back to Develop, that same image doesn't show Loading...if I go from the 1st image to the next image it takes 8-9 seconds for Loading to disappear. If I go back to the first image, I see Loading for 7-8 seconds–slightly faster (but not real fast).

Just so you know, I keep my Camera Raw Cache set to 200GB on one of my fastest drives (a 15K SAS 2 drive stripped array).

The IQ 180 files are large...

FYI, when I open multiple IQ 180 files in ACR 7.x (ok, I'm using a beta of 7.1, don't tell anybody) it takes about 9-10 seconds before the warning goes away...so it's similar to LR 4.1 RC2.
Logged

Robert-Peter Westphal

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 402
    • Nature-Photography Westphal
Re: Lightroom performance with large files
« Reply #2 on: May 05, 2012, 03:17:39 am »

Hi,

The explanation ( imho) ist pretty simple :

When you are in Library mode, Lr reads the little previews which are located on your harddisk. (-> jpegs)

When you are in developement mode, it reads the original big file, then bakes in all changes done which are inside the database and then shows te image with all changes done to it. When going back and forth inside the dev-mode, it doesn't reecognized that it rendered the picture 2 seconds ago without any changes applied to the image, it starts from scratch to render the image the way described above.

This debts due to the fact that Lr shows the images much more exactly when in dev mode than in library mode.

I hope I understood your problem right and found the right explanation to it.

Robert
Logged
'visit my completly renewed gallery at http://www.naturfotografie-westphal.com '

dreed

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1715
Re: Lightroom performance with large files
« Reply #3 on: May 05, 2012, 03:25:10 am »

Since I doubt that an image or two is filling the 200GB cache, there's probably more going on here ... for example, LR probably writes out your current image into the cache and then reads the next one in. So maybe 5 seconds to write the current one out and 3 seconds to read the next one in? An IQ180 is probably a 400-500MB TIFF file (uncompressed), so the delay you're experiencing would be about right if the size of the cached data is similar to that for a TIFF file.

But with 64GB of RAM, you'd expect that LR would be able to hold an image or two or three in RAM as a first layer cache...
Or maybe photographers in front of computers just don't operate in a manner that makes that worthwhile?

I think Jeff's case is interesting because it suggests that there is very little performance benefit from the cache - maybe one or two seconds.

The raw cache directory would be more important on systems where the CPU was slow enough to be the bottleneck during the raw conversion.

Jeff, I'd be interested to know if setting the cache size to (say) 2GB changed the times at all.

I suppose a further point is, if CPUs become fast enough and images large enough that raw conversion is not a significant factor in the image load time, is there any point in keeping a raw cache?

Similarly, I'm curious as to if Wayne used the SSD for the cache if it made any real difference as the times from moving between images suggest that it is disk read/write that is the limiting factor and supposedly the SSD should provide better throughput. Update: that is contrary to testing that has showed negligible performance benefit from using an SSD with Lightroom - see Can LR benefit from SSD?.
« Last Edit: May 05, 2012, 03:27:51 am by dreed »
Logged

Robert-Peter Westphal

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 402
    • Nature-Photography Westphal
Re: Lightroom performance with large files
« Reply #4 on: May 05, 2012, 03:32:32 am »

Yes, but how should Lr know if the cache is dirty ( change to the image has been made) or not ?


If you do a subtle change to the image, the cache is dirty and has to be dropped and recreated. And becasue of this, the image is rendered every time you load an image, regardless it was changed prior or not.

Furthermore, I'm not sure if Lr can drop a single image from the cache or if it must drop the complete cache and recreate it in complete. This is pretty complicated.

As soon as you go nto dev, Lr expects you to do changes to your images. If you just want to watch tem, it expects you to enter lib.

Robert
Logged
'visit my completly renewed gallery at http://www.naturfotografie-westphal.com '

dreed

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1715
Re: Lightroom performance with large files
« Reply #5 on: May 05, 2012, 03:39:47 am »

Furthermore, I'm not sure if Lr can drop a single image from the cache or if it must drop the complete cache and recreate it in complete. This is pretty complicated.

If you look in the cache, you'll notice that files were created at various times (assuming it is large enough for more than one image.)

This suggests that it isn't dropping the entire cache and has some idea of what is there.

Update:
Having said that, whatever the cache is used for, it is not a cache for raw data or anything of that ilk as working with 20MP image files resulted in files on the order of 700k to 800k being written into the cache directory. So scrub the above idea about the delay being due to reading/writing data into the cache. I've got to wonder just what the purpose of the cache is. But the image to image delay is thus likely the render time from one to the other. The small performance gain when using a file that you've recently looked at is more than likely the result of the raw file being cached by the operating system and thus it doesn't need to read it from disk when LR asks for it. That is more inline with the results that prove that SSDs do not help LR performance. A way to test this would be to copy the raw IQ180 file that has never been used by LR before from one place on the disk to another and then see what the opening time was for the original.
« Last Edit: May 05, 2012, 04:04:17 am by dreed »
Logged

John Cothron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 349
    • Cothron Photography
Re: Lightroom performance with large files
« Reply #6 on: May 05, 2012, 07:07:21 am »

My understanding is it keeps some of the basic rendering, but none of the adjustment stuff.  Things like de-mosaicing for instance.  The rest of it is performed each time to account for any adjustments that have been made.

It does seem it might be interesting if here were a way for it to "re-member" unchanged adjustments and keep those too.  I think that would speed it up a bit for sure.  That would also be sort of messy to do.  You would need to check the dirty/not dirty property of each setting when opening the preview again.
Logged
John
Flickr

John Cothron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 349
    • Cothron Photography
Re: Lightroom performance with large files
« Reply #7 on: May 05, 2012, 08:17:26 am »

As a follow-up, I did some playing around with some hi-res medium format film scans I have.  They're 500mb + in size.  I'm not seeing the kind of delays you guys are mentioning.  In fact it seems nearly the same as smaller size files.  For me, it is around 1 sec. at most in which it is "loading" in the develop module.  It makes me wonder what the real bottle-neck may be and I believe it has a LOT to do with the processor itself.  My cache size is only set to 50GB, and is on a normal 7200 rpm drive, albeit a separate one.

I'm running 12GB of triple channel memory at a speed of ~1550, with an i7-930 processor.  I AM over-clocking however, and my processor speed is a little over 4Ghz.  I have to believe the lag is related to computing power since the image has to render a large portion of the file each time it is loaded in the develop module.  On the other hand, it seems some of you guys are running some pretty powerful set-ups so maybe there is something else we're all missing.

Incidentally, I'm running a GeForce 9800 GTX video card bridged to another GTS 250 card, with two 24" wide screen monitors.
Logged
John
Flickr

Sheldon N

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 828
Re: Lightroom performance with large files
« Reply #8 on: May 05, 2012, 12:13:33 pm »

It's basically a CPU power issue.

In the develop module LR will pull some of the preliminary de-mosaicing info from the ACR cache but the rest of the image is rendered on the fly by the CPU. As Jeff pointed out enlarging the size of the ACR cache will help somewhat, but not a huge amount.

I don't know how effective LR is at using all of your CPU cores when doing things in the develop module. I think it uses multiple cores but is not super efficient at maximizing your full CPU power. For example a 4 core CPU with a faster clock speed might actually do better than a 12 core CPU with a lower clock speed at some tasks. You could look at your activity monitor as you scroll through images in the develop module to see if this is true. I'd bet that the CPU is working the entire time, but that it isn't maxing out all 12 cores.

Fundamentally though the issue is that IQ files are really big. The best solution IMHO is to browse in the Loupe view, and only pop into Develop when you have a specific image that you want to work on.
Logged
Sheldon Nalos
[url=http://www.flickr.com

Sheldon N

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 828
Re: Lightroom performance with large files
« Reply #9 on: May 05, 2012, 12:15:36 pm »

As a follow-up, I did some playing around with some hi-res medium format film scans I have.  They're 500mb + in size.  I'm not seeing the kind of delays you guys are mentioning.  In fact it seems nearly the same as smaller size files.  For me, it is around 1 sec. at most in which it is "loading" in the develop module. 

I think there's less processing involved in a medium format film scan than a medium format RAW file. Part of the CPU work is the debayering/demosaicing of the RAW file, which is not involved when working with a film scan.

Agree that it's a CPU issue though.
Logged
Sheldon Nalos
[url=http://www.flickr.com

John Cothron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 349
    • Cothron Photography
Re: Lightroom performance with large files
« Reply #10 on: May 05, 2012, 12:26:17 pm »

I think there's less processing involved in a medium format film scan than a medium format RAW file. Part of the CPU work is the debayering/demosaicing of the RAW file, which is not involved when working with a film scan.

Agree that it's a CPU issue though.

Makes sense, but my understanding is that once a cache preview is generated most of that step is already done.  They are experiencing the lag even after it is generated the first time based on what I'm reading.

FWIW is does not appear that Lr is fully using the processor as you suggested.  My usage jumps about 40% from base level when moving between previews, but never gets above 50% overall.  Granted I'm not getting the huge time lag either.
Logged
John
Flickr

dreed

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1715
Re: Lightroom performance with large files
« Reply #11 on: May 05, 2012, 11:05:17 pm »

Makes sense, but my understanding is that once a cache preview is generated most of that step is already done.  They are experiencing the lag even after it is generated the first time based on what I'm reading.

FWIW is does not appear that Lr is fully using the processor as you suggested.  My usage jumps about 40% from base level when moving between previews, but never gets above 50% overall.  Granted I'm not getting the huge time lag either.

I wouldn't be surprised if LR prepares for Develop by rendering the image from raw, in memory, so that it is ready to go as soon as you move a slider. Otherwise, if it just showed you the cached image, the first time you made a change to anything in Develop, it would then need to open the raw image and execute the entire develop process then for the first time. (Render would include the demosaic step.)

So when you're working with big images, use Library to browse and compare images and leave Develop until you actually need to make changes?
Logged

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4237
    • waynefox.com
Re: Lightroom performance with large files
« Reply #12 on: May 05, 2012, 11:45:21 pm »

I appreciate all the comments.  I guess I don't remember things being this sluggish from Lightroom 3, maybe I just never paid attention. so that's why the questions.  I also compare this to CaptureOne, and while it has it's own issues with speed in some things (such as local adjustments), moving from image to image is more in the order of a couple of seconds or less.

CaptureOne seems to be threading things differently as I watch the processor and utilizations ... when moving from one image to another I see it almost max out all 24 cores, and hitting close to 2000% of the CPU %, where as Lightroom seems to go through a pretty regular process of using quite a bit of the first core, then a little better, then suddenly all the non virtual cores seems to kick in then for about a second things ramp up to about 1000%.  I suppose it makes sense the CaptureOne team spends some real effort with this since it's trying to support it's own high end backs, Adobe's team is terrific and certainly I appreciate the support LR has for my IQ180 but I know I'm a minuscule part of the market for them and with pretty much any other camera files things are nearly instant.

I've upped the cache a little, and yes the cache is located on my SSD.  (SSD has OS, all applications and user library, all other user folders are mapped to the 4 drive raid 0).

I assumed LR would make better use of preview files, such as tracking the develop state based on the last rendered state of a preview file so it could know whether it needs to render a new preview or could indeed use the last generated one. Seems it's logic is probably a little different so it has to render a develop preview on the fly ... I'll trust the engineers are doing what they can for performance so there's some logical reason it works the way it does.
Logged

dreed

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1715
Re: Lightroom performance with large files
« Reply #13 on: May 06, 2012, 02:02:24 am »

...
I assumed LR would make better use of preview files, such as tracking the develop state based on the last rendered state of a preview file so it could know whether it needs to render a new preview or could indeed use the last generated one. Seems it's logic is probably a little different so it has to render a develop preview on the fly ... I'll trust the engineers are doing what they can for performance so there's some logical reason it works the way it does.

If you look in the preferences for previews, you'll see that the maximum size is 2048 pixels wide for high quality previews so quite obviously the preview cache does not hold full size rendered images.
Logged

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: Lightroom performance with large files
« Reply #14 on: May 06, 2012, 02:21:04 am »

If you look in the preferences for previews, you'll see that the maximum size is 2048 pixels wide for high quality previews so quite obviously the preview cache does not hold full size rendered images.

Ignore entirely the Library "Previews" because that has ZERO to do with the Develop "Loading..." dialog. The Library previews and the Develop previews are not related...Develop previews are rendered from the original raw data and not from the Library previews–two different animals...

Wayne, just so you know, I tested LR 3.6 with IQ 180 files and going from Library to Develop Loading finished repeatedly hits about 10-11 seconds. Compare that to LR4's 9 seconds and I would say that LR 4 is "slightly" faster than 3.6...and considering the image adaptive CPU penalty of PV 2012, I would say that LR4 is the same speed or slightly faster than LR 3.6 with IQ 180 files...

I think it's a matter of IQ 180 images being really large and slow to render in Develop. And comparing C1 previews isn't fair unless you render 100% previews in C1. Apples/oranges unless you are at 100% previews in Develop and C1.
Logged

Electromen

  • Newbie
  • *
  • Offline Offline
  • Posts: 32
Re: Lightroom performance with large files
« Reply #15 on: May 06, 2012, 06:23:32 am »

Since it's a Mac Pro, you probably have several internal HDDs.
1. Do you have the Lightroom software, photos and Cache on three different drives?
2. How large is you Cache?
3. Have you Optimized the Catalog?
4. Are you rendering full size previews?
Logged

dreed

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1715
Re: Lightroom performance with large files
« Reply #16 on: May 06, 2012, 08:07:55 am »

Since it's a Mac Pro, you probably have several internal HDDs.
1. Do you have the Lightroom software, photos and Cache on three different drives?
2. How large is you Cache?
3. Have you Optimized the Catalog?
4. Are you rendering full size previews?

As per Jeff's comments, none of these matter with respect to the "Loading..." time seen in Develop.

And given how the cache is used, I'm becoming quite curious as to just how small it can be as it most definitely seems to offer little benefit.
Logged

John Cothron

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 349
    • Cothron Photography
Re: Lightroom performance with large files
« Reply #17 on: May 06, 2012, 08:44:35 am »



And given how the cache is used, I'm becoming quite curious as to just how small it can be as it most definitely seems to offer little benefit.

Agreed, it seems as if the cache preview doesn't save anything outside of a de-mosaicing, and all other adjustments are rendered again upon viewing in the develop module.  There may be valid reasons for not doing so, but I would think a simple dirty/not dirty check on the database record for that image would allow you to save all adjustments and preview that....unless the image has been adjusted (making the record dirty) you wouldn't have to render again each time.

I'm not personally have a large delay, but I'm working with (primarily) 23mp files, not 80.
Logged
John
Flickr

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Re: Lightroom performance with large files
« Reply #18 on: May 06, 2012, 03:25:49 pm »

This might or might not be related.. I have a fairly modest workstation, i7-950, 12gb RAM, SSD, etc.. and I don't have the redraw issues and my smaller 5d2/1ds3 files open quickly once all the previews have been built and LR isn't otherwise tasked..  but has anyone with the speed issues tried disabling their plug-ins or web gallery templates?  I was just reminded that some of mine were made for LR2 or LR3 and hadn't been updated.  These weren't affecting my use, but maybe it could be affecting someone else's?

As we update LR and carry over our web galleries, templates, plug-ins, preferences.. they're not necessarily compatible with the new version.  It might be worth the time to rename the plug-in and web gallery folders or disabling them.. and giving it a try.

In the same way we know deleting the prefs file can solve issues maybe if having a performance issue we should try optimizing the catalog (something I do as habit every few weeks) or even making a new/small catalog and switching to it for a test?   You don't need to delete your own catalog, you can switch back and forth between several.

I know this is all basic stuff everyone knows about, but sometimes we get comfortable with these types of things if they're not giving us an overt problem.

I admit to being curious.  While there's a definite trend showing a higher spec system performs much better among these threads, there's still a few getting through which are higher spec and still have the issues which leads me to believe there could be other issues at play for some people depending on how your machine is setup. 

I've always kept a "new build" of my OS/programs on a separate hard drive I can pop in (my case has 8 hot bays so they exchange with ease) to help troubleshoot performance or other issues.  The concept is easy to duplicate on an existing build by disabling certain things and/or booting raw..

There's enough of these 'LR is slow' threads to choke a horse.. so it's obvious LR is at least demanding of hardware and proper setup, but the best way to troubleshoot the issue(s) isn't necessarily to keep trying different settings or even throwing hardware at the problem.. first.  It would help understand more if we could narrow down the bottleneck with certainty first.
Logged
----------------------------------------------
http://www.BangkokImages.com

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4237
    • waynefox.com
Re: Lightroom performance with large files
« Reply #19 on: May 07, 2012, 01:11:56 pm »

I think it's a matter of IQ 180 images being really large and slow to render in Develop. And comparing C1 previews isn't fair unless you render 100% previews in C1. Apples/oranges unless you are at 100% previews in Develop and C1.
C1 does do something differently - seems the preview files are more actively used.  Once it has generated the previews for a folder it takes only about a second to render the screen image, and if you zoom to 100% it only takes a couple of seconds to resolve that.  If you move to a different image and then back to that image, its pretty much instant. However, once you are in 100% LR seems to have the full rendered image available so if you move around with the navigator things are always sharp, where C1 appears to only render what's available in the current window, and if you move around with the navigator it takes a second to render the sharp image each time.  Not that it matters, I know they are very different programs with different design philosophies.

The main reason for my initial questions is my puzzlement over the lack of performance of Lightroom on my MacPro as compared to my Macbook Pro, at least in rendering the initial image in develop.  In fact, it seems my MacBook Pro (2.5ghz i7 quad with 16gigs of ram) connected to a firewire raid 0 beats the Mac Pro consistently by about 1 second.  That's what has be concerned there might be something wrong with the MacPro. Puzzling since it is a brand new built from scratch OS drive with minimal software.

Any chance that Lightroom in it's attempt to thread out the tasks to so many cores it actually slows it down?  I may disable a bunch of the cores on my MacPro and see what happens.
Logged
Pages: [1] 2   Go Up