Speeding up merging HDR photos and panoramas in Lightroom CC

[Wawe]

Hi!

As an enthusiast travel photographer I do a lot of HDR, panoramas and HDR panoramas. I merge the photos as an HDR dng file in Lightroom and also stitch the panoramas from the Nikon D800 raw files or from the dng files (in case of HDR panoramas) in Lightroom.

The problem currently is that the process takes a lot of time. Of course, the speed depends on how many files I’m merging as an HDR file or stitching as a panorama, but typically we are talking about anything from a couple of minutes to 15 minutes (including both creating the preview and the actual end result).

From what I’ve understood, both HDR merging and stitching panoramas are memory intensive processes. I run Lightroom from my SSD drive, on which I only have about 30Gb free space. Sometimes, when stitching, say, more than 7 images as a panorama, I run out of space on the SSD and Lightroom is unable to complete the final image. From this, I’ve concluded that perhaps the bottle neck currently is the speed of the scratch disk.

What I described above has lead to me to two possible solutions:

1) By increasing the amount of RAM from my current 8Gb to, say, 32Gb I could speed up these processes significantly, assuming that the scratch disk wouldn’t have to be used at all. Does this logic apply?

2) If the operating system (Windows 10) or Lightroom (CC) isn’t for some reason able to benefit from that much of RAM, would a possible workaround be installing a super fast SSD drive, such as the new Samsung 950 Pro M.2 NVMe, so that the by using that as a scratch disk the writing speed would simply be multiple times faster than with my current 4-year-old SSD drive?

Later this year, I’m building a new PC mainly for post-processing, so if something completely else than what I thought above could work, please feel free to suggest other options as well.

Thanks a lot for any help in advance!

[DaveRichardson]

It is hard to tell which component gives the best speed increase without changing individual components.
However my own experience is that stitches of 6-15 RAW images take between 15 seconds and 1.5 minutes on a Windows 10 system with 64GB RAM, an i7-3930K processor an SSD main drive but ordinary spinning disks for other drives. GPU is enabled on AMD HD7900 - but I'm not sure whether that is used in stitching.
Hopefully if others chip in you may get a feel for which components have the biggest impact.

Dave

[Wawe]

Hmm, sounds like your computer is doing something significantly faster than mine.

Perhaps it would help to know a bit more about my PC, so here's some other possible relevant specs:

RAM: Kingston 8GB (2 x 4GB) HyperX Beast, DDR3 1600MHz

Motherboard: AsRock Fatal1ty H87 Performance

SSD: PNY 120GB XLR8 SSD SATA III 6GB/S

Graphics card: NVIDIA Zotac GeForce GTX 760 AMP! 2GB DDR5

Processor: Intel Core I5-4670K 3,4GHz

[SZRitter]

If you are running out of room, it's flat out a capacity issue, not speed on the scratch disk. You need a larger drive.

Beyond that, the faster or larger you make the SSD and RAM will both help. More RAM, by my understanding,  should increase the amount of in RAM information before it writes to scratch disk, then the scratch disk will be a little less relevant. Also, keep in mind that your RAW image file sizes could be playing a role, 5 16mp images will go faster than 5 50mp images.

[Wawe]

Yes, of course, but running out of room isn't the issue here. I just pointed that out to explain why I've understood that the scratch disk matters in the first place; because it being full can cause Lightroom to be unable to merge the files, the scratch disk seems to be playing a role here - and hence also its speed(?).

[TonyW]

Hmm, sounds like your computer is doing something significantly faster than mine.

Perhaps it would help to know a bit more about my PC, so here's some other possible relevant specs:

RAM: Kingston 8GB (2 x 4GB) HyperX Beast, DDR3 1600MHz

Motherboard: AsRock Fatal1ty H87 Performance

SSD: PNY 120GB XLR8 SSD SATA III 6GB/S

Graphics card: NVIDIA Zotac GeForce GTX 760 AMP! 2GB DDR5

Processor: Intel Core I5-4670K 3,4GHz

Not sure exactly how much has changed in LR CC from my current version of LR 5.7 or even if the slowdown is a result of a setting in your current LR install, but generally there are a few things that you can do regarding hardware that should have an effect on performance.

1.  As much RAM as you can fit will help with speed.  LR will make use of it all

2.  A fast HDD or SSD

3.  Coupled to the above the Disk should have ideally 50% free space.  Having less than 25% of disk space available is likely to slow down most applications, LR particularly.

4.  You are running LR as 64 bit?

Some general comments from what you have described so far.

a.  AFAIK, LR does not use scratch disk this is only for PS - if you have dedicated a large scratch disk for PS then this may have an impact on performance.  Therefore you may want to consider a separate SSD for PS scratch disk duties

b.  SSD:  Check free space - 50% free good - less than 25% less than optimal

c.  Check graphics card drivers are latest - and also check for net chatter about the latest drivers and any known issues!

d.  If all else fails try disabling GPU in LR Preferences.

Some info. on optimising LR application
https://helpx.adobe.com/lightroom/kb/optimize-performance-lightroom.html

[DaveRichardson]

Hmm, sounds like your computer is doing something significantly faster than mine.

Perhaps it would help to know a bit more about my PC, so here's some other possible relevant specs:

RAM: Kingston 8GB (2 x 4GB) HyperX Beast, DDR3 1600MHz

Motherboard: AsRock Fatal1ty H87 Performance

SSD: PNY 120GB XLR8 SSD SATA III 6GB/S

Graphics card: NVIDIA Zotac GeForce GTX 760 AMP! 2GB DDR5

Processor: Intel Core I5-4670K 3,4GHz

Biggest difference I can see there is RAM - I am using 64GB of RAM

Edit to add : My Specs
Processor i7-3930k at 3.2 GHz
Memory Kingston 64GB at 1800MHz
Motherboard ASUS p9-x79Pro
500GB SSD system drive + 3x2TB drives
Graphics card AMDHD7970 3GB  (driving 3 HP monitors)
Windows 10 64-bit

Dave

[wmchauncey]

This was built by my geek kid for your very reasons...

[tonyrom]

Lots of good advice but don't guess.  Use Task Manager to see what resources the app is using the most (doesn't show GPU). Start a merge/stich and look at CPU, and I/O.  If you go under the Performance, Tab, select LR in the combo box.  Then there is a link at the bottom of the performance tab that says "Open Resource Monitor".  From there you can look at what the app is consuming.  If the I/O or CPU is pegged, there is probably a memory constraint.  What happen is the memory manager uses both physical ram and Disk space to make Virtual ram.  If it runs critically low in physical ram, things get swapped to the disk, hence a bottle neck.

-tony

[Paul2660]

A few more thoughts, some repeated from above.

1.  8 GB of total ram needs to be much more like 16 even better 32.  LR takes advantage of some system ram, but IMO is much more processor dependent however adding more ram will all the OS to handle other background requests better and thus free up system resources.

2.  You mention scratch disc for LR, are you referring to the location of the cache?  I have in the past moved the location of the cache away from the same drive as the catalog, but recently move the cache location back to the same as the catalog, seemed to give a bit of improvement.

3.  What is your processor, i5, i7, etc.  LR is very processor dependent, just look at at task manager while it's running any job and or download any of the tools that will monitor your processors during the time LR is running.  LR more than likely will have all your processor threads.  So a i7 four core with 8 threads is the best I believe.

4.  GPU, anyone's guess.  It's my opinion that LR on the windows side can't begin to work with the 4 GB of ram on my GTX970 (I realize per new class action lawsuit nvidia cut some corners on total ram on these card), and the processor is barely touched.  So having a large nvidia card, may or may not help, other cards may be better, so far I have tried, 960 and 870 and 970 all with 4GB of ram, and none of them really added a bit to the speeds for LR.  It's my understanding that only the develop module is able to work with open CL, but that is the most important module.  It's easy enough to benchmark the GTX970 while LR is running and the card is barely touched.


Trying to merge 5 HDR's into a pano, will be tasking mainly the system processing in LR, so I would start there, but still increase system ram to 16GB min.

Paul C

[Pictus]

Process Explorer https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx
Can display/monitor all activity to help identify the bottleneck.

Understanding Process Explorer
http://www.howtogeek.com/school/sysinternals-pro/lesson2/



Process Hacker is similar
http://processhacker.sourceforge.net/
But with some extras like real-time information on disk access...

[disneytoy]

This is a fascinating topic, both lightroom and Photoshop. I'm about to build a new PC, because I'm maxed out at 16gb ram, and lightroom chokes just previewing my 42mpx raws.

I will build an i7 6700k modest over clock

32gb ram to start leaving 2 slots for upgrades. Ram prices have skyrocketed in the last couple months.

A Samsung new 960 evo 250gb M.2 SSD. Almost 3.2 gb transfer rate. Then later add a second for an array. The PCI bus maxes out at 3.5 gb so already close to the max speed of the drive for start up. And the cache/library files.

Not sure about the GPU. 1060/70/80. I don't play games, so I don't really know if the extra grunt of a 1080 really helps PS or LR.

There is so much on benchmarks for gamers, but it isn't clear if a super fast GPU really makes a difference. Yes, running a GPU enabled filter in PS, but that is maybe 1% of our workflow.

I do her and panos in LR. And a set of 42mpx raws can really choke LR.

[Pictus]

Right now for Photoshop/LR the GTX 1060 is enough.