Pages: [1] 2   Go Down

Author Topic: Lightroom performance question: will supersizing RAM make for a faster workflow?  (Read 15548 times)

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

here's my current situation: I am working with a late 2009 2.66Ghz i5 iMac.  My OS and Applications use an OWC Mercury Elite 6G SSD I installed in place of the internal optical drive. and I had  what I thought until very recently was the max amount of RAM the computer can hold - 16GB.  The original 1TB HDD  has been repalced with a 7200 RPM 3TB HDD which is partitioned three ways: internal boot drive backup, "hot projects", and scratch disk for PsCC.  My photo and video library resides on a Drobo 5D with fast drives connected via USB 2.0 (While Drobo 5D has Thunderbolt and USB 3.0 output, this model of iMac was the last prior to Thunderbolt and USB 3 being added.  Even so I am getting faster throughput than I did with USB 2 to USB 2 connectivity.)


While I am not planning on replacing the entire computer in the near future as I am  waiting for Adobe to improve/implement better multi-core support, I now know that I can up the RAM capacity to 32GB. Price is about $400 (http://eshop.macsales.com/item/Other%20World%20Computing/8566DDR3S32S/).

My question is: How much extra speed  when working in  Lightroom and Photoshop CC will 32GB RAM buy me? 5%, 10%, 25% 50%? In other words: will the possible cost to performance boost be worth the cash investment?
Logged

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com

More ram will help Photoshop if you are working on large images or many images open at the same time. Adding ram prolly won't do much for Lightroom unless you are the type of user that has many applications open at the same time. Lightroom, Photoshop, Bridge, iTunes, Safari and other apps will combine to make your ram pretty fragmented until you close down all the apps. If that's the way you work, then yes, the extra 16gbs will help.

Help you by how much? There really isn't a way to quantify that. Do you get a lot of spinning beach balls? That's a sign that the apps and system are starving for ram. The spinning beach ball means the system is paging memory to disk to clear it up for apps and the system. Having more ram means seeing the beach ball a lot less.
Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

Thanks Jeff, you answered my question perfectly. I'm not seeingthe spinning beachball much at all so it must be primarily a throughput issue. I don't think I will see a real advantage in increasing my RAM  until  I move up to a computer with faster processors + TB + USB 3 , which will  be a 2013/14 MacPro possibly later this year.
Logged

alifatemi

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 278
    • ali fatemi photography

And now the question is if upgrading to MacPro from iMac is beneficial for LR users? Since LR and PS, can't benefit from dual GPU and more than 6 cores, so what's the point upgrading? higher CPU frequency or more core does any good for us?
Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

And now the question is if upgrading to MacPro from iMac is beneficial for LR users? Since LR and PS, can't benefit from dual GPU and more than 6 cores, so what's the point upgrading? higher CPU frequency or more core does any good for us?

As I said in my original question:"I am  waiting for Adobe to improve/implement better multi-core support." I have difficulty imagining that such support is not forthcoming, but my Quija board is being helpful in projecting a timeline for that.
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography

Jeff is certainly more of an expert than I am on all things Adobe but here's the LINK to the specification optimization for Photoshop.  Only certain manipulations in PS require lots of processing power and memory.  With LR if you have a decent system you are good to go and it's not as dependent on RAM, GPU, or CPU at this point in time.  Perhaps as more PS features get added that might change but again with the movement to CC those of us who stick with LR/PS perpetual license are likely not to see much in the way of differences.  I run a fast i-5 CPU with 16GB of fast RAM and don't see any issues at all with LR.

Alan
Logged

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website

Adding ram prolly won't do much for Lightroom unless you are the type of user that has many applications open at the same time. Lightroom, Photoshop, Bridge, iTunes, Safari and other apps will combine to make your ram pretty fragmented until you close down all the apps. If that's the way you work, then yes, the extra 16gbs will help.

And if you are the type of user who tends to keep a lot of other applications running all the time but want to free up physical memory when you're using Lightroom, keep in mind that when you click on the little red button at the top left of an Apple OS X application window, you are not terminating the underlying process so that it will release its memory allocation and stop making requests to the CPU scheduler (unlike the analogous action in Microsoft Windows).  All you are doing is releasing some monitor real estate while the application process continues to execute in the background.  If you really want to make those machine resources available to Lightroom, you should explicitly issue a "Quit" command from the pull-down menu under the name of the application in the menu bar.

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website

here's the LINK to the specification optimization for Photoshop.

Thanks for posting this URL.  Very interesting.

My empirical experience is that on OS X, at least, Lightroom is most likely to exhibit perceptible delays due to demand-paging when Photoshop is invoked from within a LR session (i.e., with the "Edit In" menu pick) and the Adobe link you posted explains why.  Photoshop is designed to request a large allocation of physical memory—by default, 70 percent of what is available in the host machine—which makes sense for a pixel editor, given the size of the image files most of us are working with these days.  If Lightroom and Photoshop are executing concurrently on a machine with a modest amount of physical memory, the operating system will only be able to honor the PS memory-allocation request by paging out some of the physical memory previously allocated to LR.  When you move your input focus back to LR (i.e., making it the foreground process), the memory LR needs will need to be paged back in.  If your swap device is a spinning disk, that's likely to result in OS X displaying the dreaded "beachball" and you twiddling your thumbs.

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography

Thanks for posting this URL.  Very interesting.

If Lightroom and Photoshop are executing concurrently on a machine with a modest amount of physical memory, the operating system will only be able to honor the PS memory-allocation request by paging out some of the physical memory previously allocated to LR.  When you move your input focus back to LR (i.e., making it the foreground process), the memory LR needs will need to be paged back in.  If your swap device is a spinning disk, that's likely to result in OS X displaying the dreaded "beachball" and you twiddling your thumbs.
I think this is where a SSD scratch disk in Windows will markedly help things out as PS will use the scratch disk to overcome any RAM limitations.  I don't use PS all that much any more with all the LR advancements so it's a non-issue for me.
Logged

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website

I think this is where a SSD scratch disk in Windows will markedly help things out as PS will use the scratch disk to overcome any RAM limitations.

It's important to distinguish the Photoshop scratch disk (selectable within Photoshop) from the operating system's swap device, which is a reserved area of mass storage—i.e., not available to application software—that is typically established at the time the operating system is installed.

I don't have any knowledge of Photoshop's internals, but I presume that during the course of a session Photoshop will use the user-selected scratch disk as a repository for temporary files explicitly created by Photoshop (which application programmers call "scratch files," hence the use of the name to designate the disk).

But demand-paging is an operating system function, invisible to application programs, that occurs when a process needs more physical memory than is currently allocated to it.  The process requests a memory allocation from the operating system and, if surplus physical memory is not available, the operating system reclaims some from idle processes or those running at lower priority than the requesting process ("paging out" some of their physical memory) and making it available to the requesting process ("paging in").

This is the phenomenon that causes Lightroom to block temporarily on both OS X and MS Windows when there is insufficient physical memory in the machine.  While I'm sure designating a solid-state device as a Photoshop scratch disk will speed up operations during a Photoshop session, if you want to accelerate the demand-paging process you will need to reconfigure the operating system to use a faster swap device.  (Which is feasible, but you need to know exactly what you're doing or you can hose your computer.)

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4237
    • waynefox.com

My photo and video library resides on a Drobo 5D with fast drives connected via USB 2.0 (While Drobo 5D has

I think LR benefits much more from drive access speed rather than more RAM.  Seems a USB 2 system, no matter how fast the drives, is pretty slow.

I just bought the new MacPro, upgraded to 64gb of ram, and am using the LaCie thunderbolt  Big 5 striped as a raid 0, and LR is more responsive, and that’s comparing to a previous system with an internal raid 0 on 4 drives (2010 Mac Pro).  But how much faster than your system, hard to say. 

Photoshop screams though (fully launches in less than 2 seconds).
Logged

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

It's important to distinguish the Photoshop scratch disk (selectable within Photoshop) from the operating system's swap device, which is a reserved area of mass storage—i.e., not available to application software—that is typically established at the time the operating system is installed.

I don't have any knowledge of Photoshop's internals, but I presume that during the course of a session Photoshop will use the user-selected scratch disk as a repository for temporary files explicitly created by Photoshop (which application programmers call "scratch files," hence the use of the name to designate the disk).

But demand-paging is an operating system function, invisible to application programs, that occurs when a process needs more physical memory than is currently allocated to it.  The process requests a memory allocation from the operating system and, if surplus physical memory is not available, the operating system reclaims some from idle processes or those running at lower priority than the requesting process ("paging out" some of their physical memory) and making it available to the requesting process ("paging in").

This is the phenomenon that causes Lightroom to block temporarily on both OS X and MS Windows when there is insufficient physical memory in the machine.  While I'm sure designating a solid-state device as a Photoshop scratch disk will speed up operations during a Photoshop session, if you want to accelerate the demand-paging process you will need to reconfigure the operating system to use a faster swap device.  (Which is feasible, but you need to know exactly what you're doing or you can hose your computer.)

Thank you Chris. But do you know if this "he operating system's swap device, which is a reserved area of mass storage—i.e., not available to application software—that is typically established at the time the operating system is installed." - applies to Mac OS X as well? And can its location ever be modified?

Wayne: With my OWC SSD I can also fully launch Photoshop CC within two seconds on my machine.  and i agree about USB 2.0  and also FW800 being the real bottlenecks.
Logged

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website

Thank you Chris. But do you know if this "[t]he operating system's swap device, which is a reserved area of mass storage—i.e., not available to application software—that is typically established at the time the operating system is installed." - applies to Mac OS X as well? And can its location ever be modified?

Yes and yes.  But it's a bit tricky, and you need to be exceedingly careful or you could wind up staring at a monitor connected to a brick.

I'm not going to post instructions on changing the swap device because (1) I've never done it myself on OS X (although, in a past life, I often did so on another flavor of UNIX) and (2) I don't want to feel responsible for somebody else's system going belly-up.  If you're sufficiently motivated, Google "change swap device OS X."

Unless you have a lot of confidence in your system administrative abilities, my advice would be to add more physical memory to your current computer if you are seeing evidence of memory exhaustion.  Or even buy a new one.  Like Wayne Fox, I'm running a 2013 Mac Pro (6-core) with 64 GB of physical memory and I haven't been able to find a way to overwhelm it yet.  I've had Lightroom, Photoshop, and DxO Optics Pro all running simultaneously, and the system didn't skip a beat.

However, as Jeff Schewe hinted (at least, that's how I interpreted his earlier post), if you don't need to run LR concurrently with Photoshop or other memory-intensive applications, you may want to try shutting down whatever else is executing when you're doing post-processing—with the "Quit" command since just clicking on the red button at the top left of the application window(s) won't release any machine resources—before you invest in augmenting your hardware.  Won't cost you a cent, and it may well do the trick.

Ellis Vener

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2151
    • http://www.ellisvener.com

Thank you again Chris. While I am not totally confident in my sys admin skills to try this right now I feel that, with your help, Tom's, and Jeff's,  at least I am getting to a point of learn to ask the right questions.
Logged

PeterAit

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4559
    • Peter Aitken Photographs

I assume that the Mac has a utility like Windows' Task manager that can show, in real time, the amount of free and used RAM. Keep this open while working and you'll get an idea of whether your apps and workflow are using all or most of your RAM.
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography

It's important to distinguish the Photoshop scratch disk (selectable within Photoshop) from the operating system's swap device, which is a reserved area of mass storage—i.e., not available to application software—that is typically established at the time the operating system is installed.

While I'm sure designating a solid-state device as a Photoshop scratch disk will speed up operations during a Photoshop session, if you want to accelerate the demand-paging process you will need to reconfigure the operating system to use a faster swap device.  (Which is feasible, but you need to know exactly what you're doing or you can hose your computer.)
I'm talking about having a dedicated 'separate' drive for the scratch disk and not using the drive that the OS is on.   I have my system and programs on one SSD, data on a fast HDD, and a separate SSD for other purposes including the rare time I need it for PS.  there is no issue in terms of impacting the OS adversely in this type of setup.  there's a lot on the Adobe website on how to set this up and I'm only speaking as a Win7 user.
Logged

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website

I assume that the Mac has a utility like Windows' Task manager that can show, in real time, the amount of free and used RAM. Keep this open while working and you'll get an idea of whether your apps and workflow are using all or most of your RAM.

The OS X utility is called Activity Monitor.  However, unlike the MS Windows Resource Monitor, in the current version of OS X it does not display the operating system kernel counters for paging activity.  It has a "Swap Used" metric, but that's a very blunt instrument.  (OS X does include a sophisticated utility called DTrace which allows the user to probe the kernel counters.  However, it has a moderately steep learning curve.)

I personally don't put much stock in either operating system's low-resolution monitoring utilities, but at least the Microsoft one offers information that is directly relevant to determining whether more memory would be useful.  The problem is that it's difficult to interpret what the display means unless you have so much physical memory installed that some of it is always free.  Assuming you are using all the physical memory in the machine, in a Microsoft environment it is possible to monitor the memory consumption of the process whose performance you want to optimize.  The "commit charge" shows how much virtual memory—either physical memory or backing store—the process is using.  (In other words, the aggregate memory the process has requested.)  The "shareable working set" displays the amount of virtual memory that can be paged-out if another process makes a demand for physical memory on the operating system.  The "hard faults/sec" metric shows how often the monitored process needs to request that a block be pulled off the swap storage device and placed in physical memory.

But the hard part, whether you use the Microsoft Resource Monitor or probe the OS X kernel with the DTrace utility, is to figure out how much demand-paging is too much for a particular process.  Most modern application programs spawn multiple coactive threads of control (in effect, mini-processes), some of which are born and die rather quickly and some of which persist for a long time.  They all share the same virtual memory by default (although the programmer can explicitly protect a thread's memory from other threads).  Some threads probably have more influence on whether an application seems slow than others.  Unless you are thoroughly familiar with the application's internals, I'm not sure any technical performance monitoring will be a more useful guide than experimenting with the things you can directly control—namely how many applications are running at the same time.

Jeff Schewe tells us that Lightroom should be happy as long as it has access to 16 GB of physical memory (minus operating system overhead).  So my feeling is that the best way to determine whether Lightroom would perform better with more RAM would be something along these lines:

  • If you always have some free memory when Lightroom is running, you don't need more.
  • Otherwise, if LR seems sluggish, try terminating other applications that may be competing with it for physical memory.
  • If LR still seems sluggish after terminating inessential applications, add more memory and hope that the improvement will be sufficient.

I've done a fair amount of performance-tuning over the years, but that was mostly on large multiuser computers that were required to execute several (or many) application programs simultaneously and where I had tools at my disposal to modify the behavior of both the system and the application programs.  For a single-user desktop or laptop, I just don't have any confidence that a technical approach will yield a better result than the simple algorithm above.
« Last Edit: April 25, 2014, 01:07:23 pm by Chris Kern »
Logged

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4237
    • waynefox.com

The OS X utility is called Activity Monitor.  However, unlike the MS Windows Resource Monitor, in the current version of OS X it does not display the operating system kernel counters for paging activity.  It has a "Swap Used" metric, but that's a very blunt instrument.  (OS X does include a sophisticated utility called DTrace which allows the user to probe the kernel counters.  However, it has a moderately steep learning curve.)

I personally don't put much stock in either operating system's low-resolution monitoring utilities,
Activity Monitor in Mavericks has seen some changes. Unlike previous versions, it has a graph called Memory Pressure, which shows color bars related to memory usage.  A red bar indicates the OS is using the storage device.  As far as swaps, it simply says Swap used and a total size.

In my case when using LR on my 16gb Macbook Pro, I never saw any red bars, and my Swap Used never was higher than 256k.  I did several functions in LR without this changing.  I had 9 applications running, including PS, and opening a file in PS CC, changing to smart object, applying the ACR filter on the object, and back and forth between LR and PS this didn’t change.

I have no clue if this is meaningful :)  Perhaps with Mavericks,  memory usage is pretty efficient and VM usage might be minimal, and 16 GB is adequate -
Logged

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2034
    • Chris Kern's Eponymous Website

Perhaps with Mavericks,  memory usage is pretty efficient and VM usage might be minimal, and 16 GB is adequate -

Unfortunately, I suspect the fact that Apple implemented the new "memory compression" feature in v. 10.9 means that the OS X kernel developers (assuming there currently are any) haven't tackled the longstanding problems with memory management.  I did some informal testing a while back on a dual-core Mac Mini that I knew had insufficient memory (8 GB) and it did seem that memory compression helped Lightroom's performance a bit by reducing demand-paging.  But I couldn't meaningfully determine how much.

And Apple's low-resolution Activity Monitor, like Microsoft's Resource Monitor, only updates once a second.  That's way too slow for metrics like "Memory Used" or "Swap Used" to be useful in my experience unless they are continuously reporting that you have more physical memory than you need to handle the current process load (i.e., substantially less than all "Memory Used" and no "Swap Used").

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4237
    • waynefox.com

Unfortunately, I suspect the fact that Apple implemented the new "memory compression" feature in v. 10.9 means that the OS X kernel developers (assuming there currently are any) haven't tackled the longstanding problems with memory management.  I did some informal testing a while back on a dual-core Mac Mini that I knew had insufficient memory (8 GB) and it did seem that memory compression helped Lightroom's performance a bit by reducing demand-paging.  But I couldn't meaningfully determine how much.

And Apple's low-resolution Activity Monitor, like Microsoft's Resource Monitor, only updates once a second.  That's way too slow for metrics like "Memory Used" or "Swap Used" to be useful in my experience unless they are continuously reporting that you have more physical memory than you need to handle the current process load (i.e., substantially less than all "Memory Used" and no "Swap Used").
I'll have to take your word for it.  But it seems that even if it samples low resolution activity, eventually you would see an indication of swap or pages to the hard drive - in a couple of hours I have seen nothing on my Mac Pro (although 64 gigs of ram it seems very unlikely the system wouldn't have enough physical ram).  The Swap file size doesn't appear very dynamic, I believe as the system creates and then expands the swap file size, it doesn't necessarily reduce that file size immediately when the memory is no longer needed.  On my MacBook Pro, once it created the 256k swap file, that remained even when all programs where closed and available free ram was several gig.  It just doesn't seem like the OS isn't needing to do much, if any, paging to the hard drive, even with just 16 gigs of ram.  My Mac Pro which has been running with restarting for quite some time still shows the swap used at 0.

I guess from my perspective it appears for LR, 16 gigs of ram seems pretty adequate, so the main bottlenecks will be CPU performance and storage access speeds.
« Last Edit: April 26, 2014, 10:42:09 pm by Wayne Fox »
Logged
Pages: [1] 2   Go Up