The sad state of multi-threaded RAW processing

[knweiss]

I think some of you will find this diglloyd RAW processing comparison interesting: http://macperformanceguide.com/Shootout-Ma...processing.html

Quote: "None of the RAW-file converters make full use of CPU resources. Put simply, this is the result of poor software engineering, notwithstanding the lame excuses you’re bound to hear from software vendors. ... In the simplest approach, programs could process one RAW file per CPU core, with a worker thread delivering I/O services— but none of them do. They all are brain-dead on the algorithm: process one file at a time, in sequence. It’s an idiotic algorithm for a multi-core world."

Personally, I'm using LR 2.7 on a 8-core Mac Pro with 10 GB RAM and I can confirm that LR makes bad use of the CPU resources. It really should be possible to saturate all available CPU cores during the export of several images...

[Mark D Segal]

I think some of you will find this diglloyd RAW processing comparison interesting: http://macperformanceguide.com/Shootout-Ma...processing.html

Personally, I'm using LR 2.7 on a 8-core Mac Pro with 10 GB RAM and I can confirm that LR makes bad use of the CPU resources. It really should be possible to saturate all available CPU cores during the export of several images...

Thanks for the link. This is not encouraging news for those of us planning to up-grade from two to four or four+ cores. I'm wondering how you know that Lightroom 2.7 makes bad use of CPU resources, and I'd be even more curious to know whether Phase Capture 1 and Photoshop CS5 would have the same issue. It makes one wonder whether up-grading to such high-powered hardware really makes sense until the software catches-up with the hardware.

Also, I wonder whether there are other more decisive efficiency constraints in the hardware - for example front bus capacity.

[Ben Rubinstein]

On the same subject, is Bridge/ACR 64 bit yet in CS5?

[hsmeets]

Personally, I'm using LR 2.7 on a 8-core Mac Pro with 10 GB RAM and I can confirm that LR makes bad use of the CPU resources. It really should be possible to saturate all available CPU cores during the export of several images...

If that export writes files to you harddisk it's no wonder, the disk is the weakest link, it's slow compared to cpu/memory. while doing the export and file writing the CPU has to wait for the harddisk. 8 cores working on 8 files to convert and write the file to disc just would all be sitting and waiting and competing for a piece of the disc resource. Or files would be buffered in memory and 1 core would (slowly) flush them to disc.

Get 8 seperate disc and store the results expoerted file on different discs......that might improve performance.

The reference in Digloyds rant is as far as I read it more targeted at operations like sharpening, uprezzing which are compute intensive and, I agree with the article, could make much better use of available cores. Saving a RAW file as TIFF, JPEG or whatever is not compute intensive, it's what they call disc bound for speed.

[ErikKaffehr]

Hi,

Disk utilization in LR seems to be very low, but CPU utilization can be high in some situations, like building 1:1 previews.

It is not my opinion that discs are the bottleneck in Lightroom.

Best regards
Erik

If that export writes files to you harddisk it's no wonder, the disk is the weakest link, it's slow compared to cpu/memory. while doing the export and file writing the CPU has to wait for the harddisk. 8 cores working on 8 files to convert and write the file to disc just would all be sitting and waiting and competing for a piece of the disc resource. Or files would be buffered in memory and 1 core would (slowly) flush them to disc.

Get 8 seperate disc and store the results expoerted file on different discs......that might improve performance.

The reference in Digloyds rant is as far as I read it more targeted at operations like sharpening, uprezzing which are compute intensive and, I agree with the article, could make much better use of available cores. Saving a RAW file as TIFF, JPEG or whatever is not compute intensive, it's what they call disc bound for speed.

[bjanes]

On the same subject, is Bridge/ACR 64 bit yet in CS5?

On my Windows 7 machine, Bridge.exe is in the Program Files (x86) folder, which indicates that it is still a 32 bit application. I don't know that this is a disadvantage, since the main advantage of 64 bit programs is to access a large memory space. This is important those working in Photoshop with large layered files, but I don't think Bridge needs that much memory.

Bill

[Peter McLennan]

On a similar note: Does CS5 retain the memory leak problems of old?  

When stitching or other memory-intensive operations fail in CS4, reloading the program and re-trying the failed op usually rectifies the problem.  Is this still present in CS5?

[JeffKohn]

On a similar note: Does CS5 retain the memory leak problems of old?  

When stitching or other memory-intensive operations fail in CS4, reloading the program and re-trying the failed op usually rectifies the problem.  Is this still present in CS5?

Yes, in fact 32-bit CS5 seems to be worse than 32-bit CS4 in this regard. I don't think it's a matter of leaking per-se, but memory fragmentation.

[Mark D Segal]

Yes, in fact 32-bit CS5 seems to be worse than 32-bit CS4 in this regard. I don't think it's a matter of leaking per-se, but memory fragmentation.

What makes you say this?

[alain]

I think some of you will find this diglloyd RAW processing comparison interesting: http://macperformanceguide.com/Shootout-Ma...processing.html

Quote: "None of the RAW-file converters make full use of CPU resources. Put simply, this is the result of poor software engineering, notwithstanding the lame excuses you’re bound to hear from software vendors. ... In the simplest approach, programs could process one RAW file per CPU core, with a worker thread delivering I/O services— but none of them do. They all are brain-dead on the algorithm: process one file at a time, in sequence. It’s an idiotic algorithm for a multi-core world."

Personally, I'm using LR 2.7 on a 8-core Mac Pro with 10 GB RAM and I can confirm that LR makes bad use of the CPU resources. It really should be possible to saturate all available CPU cores during the export of several images...

Unfortunally they didn't test Bibble 5, that one seems to scale very good to a lot of CPU's.

But to be honest I don't care much about batch processing speed.  The UI speed is far more important and I have to say that Bibble is fast, very fast for that.

[knweiss]

Unfortunally they didn't test Bibble 5, that one seems to scale very good to a lot of CPU's.

I agree. From all I've read about Bibble (I don't have first hand experience with it) they really seem to care about multi-threaded performance and scaling up to a decent number of cores.

But to be honest I don't care much about batch processing speed.  The UI speed is far more important and I have to say that Bibble is fast, very fast for that.

Again, I agree. UI speed really is even more important. But IMHO the export (batch processing) should be the easiest part of the RAW converter to scale up to a large number of cores.

Soon there (probably) will be a new Mac Pro model with two Intel Westmere Hexa-core CPUs with Hyperthreading support. I.e. the software will see 24 cores. Given diglloyd's test results: Do you think it would make sense to buy such a machine for photography with today's software? Scaling to a larger number of cores becomes more and more important each year but the software seems to be way behind.

[JeffKohn]

What makes you say this?

That's just my personal observation. I'm seeing more of those program errors, clipboard errors, and plug-in failures with 32-bit CS5 than I did with 32-bit CS4. Looking at the working set for a freshly launched copy of CS5 versus CS4, it's apparent than CS5 uses more memory; so it should come as no shock that filters and plugins that run into memory problems on CS4 are going to have more problems on CS5.

[Mark D Segal]

That's just my personal observation. I'm seeing more of those program errors, clipboard errors, and plug-in failures with 32-bit CS5 than I did with 32-bit CS4. Looking at the working set for a freshly launched copy of CS5 versus CS4, it's apparent than CS5 uses more memory; so it should come as no shock that filters and plugins that run into memory problems on CS4 are going to have more problems on CS5.

Yuch. I already have Noiseware crashing CS4 routinely unless nothing else is running and the image has no more than one layer. But large image files need lots of RAM, so I'm wondering whether the key constraint is the use of 32-bit of OS with its inherent RAM limitation. I have no idea about how to write software, but we have what they give us, and if what they give us hogs memory and doesn't use cores to advantage, seems perhaos we just need lots more memory and not too many cores at this stage.

[Alan Goldhammer]

We have at least one Adobe engineer on this forum.  Perhaps he will weigh in on this topic.  It's difficult for any of us to comment in depth other than noting plug in problems (which may not necessarily be Adobe's fault).

Alan

[madmanchan]

On the Mac, only Photoshop is the 64-bit host under CS5. That is, if you want to take advantage of more memory on the Mac with CS5 in Camera Raw, you will want to open your files directly in Photoshop (i.e., have Photoshop be the host for Camera Raw), not Bridge.

[JeffKohn]

Yuch. I already have Noiseware crashing CS4 routinely unless nothing else is running and the image has no more than one layer. But large image files need lots of RAM, so I'm wondering whether the key constraint is the use of 32-bit of OS with its inherent RAM limitation. I have no idea about how to write software, but we have what they give us, and if what they give us hogs memory and doesn't use cores to advantage, seems perhaos we just need lots more memory and not too many cores at this stage.

I've seen some issues with built-in PS functionality, but plug-ins seem to be the larger problem. Nik Plugins have been particularly bad for me. I _always_ save my file before running Silver EFEX Pro, because sometimes the plug-in doesn't just fail, it brings down Photoshop with it. Now I'm not saying that's Adobe's fault. Nik's software engineers obviously don't know how to handle low-memory situations gracefully.  I was just relating the fact that these low-memory/fragmentation problems seem more likely to occur in CS5 than in CS4, due to CS5 having a larger memory footprint of its own. BTW, the problem is far worse for Mac users, because the transition from the old API's to Cocoa means that 32-bit CS5 can only use 2GB of RAM on the Mac, instead of 3GB.

It will be great when we can all run 64-bit with all of our plug-ins. But the reality is that most people have at least some plug-ins that are still 32-bit, and that situation may continue for a while yet.

[JeffKohn]

Getting back to the original point about multi-threaded raw conversion and batch processing, maybe you can comment Eric. I _thought_ I had read in the past that if you open multiple RAW's in ACR from bridge, and then save the files from there (as opposed to using Image Processor or Photoshop Batching, which works serially), that ACR actually multiple threads for saving files? Whether it's down to using multiple cores or something else, I do know that saving a bunch of RAW's to Tiff or PSD is faster when using the 'Save' functionality in ACR as opposed to using Image Processor.

[Mark D Segal]

We have at least one Adobe engineer on this forum.  Perhaps he will weigh in on this topic.  It's difficult for any of us to comment in depth other than noting plug in problems (which may not necessarily be Adobe's fault).

Alan

Hi Alan - yes I agree - not blaming anyone for anything. It just seems to be applications in aggregate demanding more processing capacity than some of our systems can handle, for whatever reason. I tend to think way more RAM is the principle ingredient to a solution and that means upgrading to a 64 bit OS which means re-installing one's whole computer and hoping for the best.

[Mark D Segal]

On the Mac, only Photoshop is the 64-bit host under CS5. That is, if you want to take advantage of more memory on the Mac with CS5 in Camera Raw, you will want to open your files directly in Photoshop (i.e., have Photoshop be the host for Camera Raw), not Bridge.

Hi Eric,

What about LR3 in a 64 bit environment? Should that run super smooth and fast?

[madmanchan]

Getting back to the original point about multi-threaded raw conversion and batch processing, maybe you can comment Eric. I _thought_ I had read in the past that if you open multiple RAW's in ACR from bridge, and then save the files from there (as opposed to using Image Processor or Photoshop Batching, which works serially), that ACR actually multiple threads for saving files? Whether it's down to using multiple cores or something else, I do know that saving a bunch of RAW's to Tiff or PSD is faster when using the 'Save' functionality in ACR as opposed to using Image Processor.

Yes, you're right, Jeff. ACR has its own batch save mechanism which can do a better job of taking advantage of multiple cores. This has been recently improved in CR 6 (relative to CR 5). This is a very reasonable way to go, if you want to take a large group of raw files and save rendered results as TIFFs.