Is There a Way in CS6 to Optimize Plug-in Performance with SSD?

[albedo13]

My wife and I shoot our son's high school football team, and we will typically shoot over 2000 shots per nighttime game.  Using CS6 on a Dell XPS 17 laptop running Windows 7 in 64 bit  with i7 2820QM 2.3 GHz processors and 16 GB of RAM, and my Seagate 5400 rpm external drive, in Bridge it would typically take 10-12 hours just to generate the previews for me to review them.  Once I have reviewed them and found the usable ones, I batch process them through ACR into PS, use Topaz DeNoise to try to get rid of some of the noise, and then output them to JPEGs for posting on the school website.  In this scenario, each application of Topaz would take about 2 minutes per image to run. At 2 minutes per image, batch processing 2000 images takes a while!

In an attempt to optimize this flow, I moved all the images onto one of my 7200 rpm internal hard drives, as well as setting the scratch disk to this drive.  I have played with the preferences for RAM usage, and have been using between 10 and 13 GB.  Doing this reduced the preview generation time to 5-6 hours, and sped up the ACR processing as well.  However, each application of Topaz still takes about 2 minutes per image.  In my latest attempt, I have purchased a Samsung 850 SSD and am using it in an enclosure attached via USB 3.0 cable.  I moved all of the images onto the SSD as well as set the scratch disk to be on the SSD.  Holy cow!  The preview generation for the entire directory of ~ 2000 images now takes less than a minute, and the ACR processing is much faster as well.  However, each Topaz application STILL takes about 2 minutes per image.

So my question is, how can I optimize any settings for a plug-in like Topaz DeNoise to run more quickly?  I have found no specific memory settings for plug-ins in any preference section.  And with everything else so improved by using the SSD, how in the world can Topaz still be running at 2 minutes per image?  Even if there are no specific memory settings for a plug-in, I figured moving all the data and scratch disk to a SSD would help Topaz run faster, but it is still processing at the same speed as when everything was set to a 5400 rpm Seagate.  BTW, these images are full sized raw images from a 1D4 and a 70D, so each image is between 20 and 24 MB.  Thanks for any insights you can offer, I am perplexed.

Jim

[bill t.]

Please forgive me for going off on a slight tangent, but this sort of job would probably be better suited to Lightroom.  It's relatively faster at generating previews than Bridge, but what might help you most is that its noise reduction is superb and applies in seconds per image.  Once you get everything balanced out in LR, you can batch process the whole lot to disc while you are in dreamland.  It's a one application solution for the whole problem.  There are other nice features such as copying settings from one image to multiple others, etc.

[Redcrown]

Little or nothing you can do about Topaz Denoise. It's notoriously slow, and 99% CPU bound. Speeding up disk I/O will have no effect. So you need to decide if the noise reduction quality that you prerceive Denoise has over ACR is worth the price. Personally, I doubt it.

Something in your post does not add up. An SSD can be 200% to 500% faster than a traditional 7200rpm HDD. And that's just pure mb/sec transfer speed on only the I/O portion of a task. The Bridge preview generation and ACR processing is part I/O and part CPU. I don't know the ratio for Bridge, but I've done tests on ACR that indicate it's somewhere around 30% I/O and 70% CPU. The CPU portion depends on what features you apply in ACR. ACR noise reduction and lens corrections, for example, add a lot of CPU time.

Going from a 5400rpm external drive (USB2?, USB3?) to a 7200rpm internal drive might account for a 50% reduction in Bridge preview generation time (10 hrs to 5 hrs). But it's inconceivable that going from the 7200rpm external drive to a USB3 connected SSD would reduce that time to less than a minute. In fact, it should be slower. The SSD+USB combo should be in the 115 mb/sec range.

I'm suspicious you processed the same files and thus the Bridge cache was already loaded when you got the less than a minute time.

(For SSD+USB examples, go here: http://www.macworld.com/article/2039427/how-fast-is-usb-3-0-really-.html)

Similar to Bill's suggestion, I'd recommend you use a different browser/preview program to cull and sort large batches of raw files. Check out this new program: http://www.fastrawviewer.com/ Free beta, no price yet, but I'd expect under $30. Even the free FastStone bowser would be an improvement as it uses the raw's embedded jpegs, so it's pretty fast.

[Mark D Segal]

For ingesting, browsing, sorting, culling, tagging image files I believe the best/fastest application on the market today is Photo Mechanic 5, by Camera Bits.

The slowdown for noise reduction is software-based and I agree with Bill T's suggestion to use Lightroom; it will most likely handle just about all your requirements from ingestion to final output. However, if you wish to remain with a Photoshop workflow I recommend you consider using Neat Image for noise reduction. It's a fast-acting and effective application for noise reduction and works as a PS plugin or a stand-alone application.

[Eyeball]

If he's shooting football with one of the Canon's that has significant banding noise, I can understand why he's using DeNoise.
I agree that DeNoise is likely CPU bound.

[Mark D Segal]

If he's shooting football with one of the Canon's that has significant banding noise, I can understand why he's using DeNoise.
I agree that DeNoise is likely CPU bound.

DeNoise is a very good application - I reviewed it on this website in its previous version. However, if speed is the O/Ps main concern, there are other options, two of which are now mentioned in this thread, that are also very good and may well give him faster performance - worth a try.

[Oldfox]

I find Bridge generating the thumbnails very slow. It helps a lot if you purge or compact the cache. If you dont do that periodically, the cache can get very big and very slow. You might try purging the cache.

I have also started to use my old Breezebrowser for just reviewing the images. Bb does this much faster than Bridge.

[albedo13]

Thanks for all of the inputs everybody!  I was wondering if I was just using already generated previews when I tested the SSD.  We have another game to cover tonight, so this will be a good opportunity to try out the process from scratch.

I really have 2 issues here.  The first was the preview generation taking forever; it appears that the SSD may be helping here.  I don't remember this being an issue before CS6, and I have seen postings by other folks who have found that it takes much longer in CS6.  I believe I can live with this using the SSD now.

The second was then taking the images I wanted and running them through ACR into PS, and then applying Topaz for noise reduction.  This process does not seem to benefit from the SSD.  I usually prefer to do 99% of my processing in ACR.  I am a big fan of Schewe and have his video tutorials and most of his books.  I have tried doing the noise correction in ACR, but the noise is so significant that in order to reduce it to what I consider an acceptable level, the images loose all the sharpness and just seem "glossy" to me.  I should point out that our stadium lighting is horrendous.  I am shooting with a 1D4 and a 500mm f4 lens.  I shoot in manual mode at 1/500 second at f4.0 at ISO 10,000.  Even at these settings the histtogram is telling me that I am shooting at least one stop underexposed.  Too make things even better, our uniforms are black.  I have done extensive comparisons between doing noise reduction in ACR (which I normally use and love for my "regular" photography), and Topaz just does a better job at getting rid of the noise and still maintaining spatial details in these very challenging conditons. I believe the noise reduction processes are identical between Lightroom and ACR, so I am not sure if using Lightroom will buy me anything here.

So my main concern/question, am I missing some parameter somewhere that affects the performance of plug-ins like Topaz?  Or has been pointed out, is Topaz really that CPU intensive and there is nothing I can do short of ordering a desktop Cray supercomputer to speed Topaz up?  Maybe I will ping them directly as well, that has to be some pretty intensive computation.

Thanks for the inputs!

Jim

[deejjjaaaa]

My wife and I shoot our son's high school football team, and we will typically shoot over 2000 shots per nighttime game.  Using CS6 on a Dell XPS 17 laptop running Windows 7 in 64 bit  with i7 2820QM 2.3 GHz processors and 16 GB of RAM, and my Seagate 5400 rpm external drive

may I suggest to buy a PC capable of 32-64Gb RAM (you have D4, so you can shell some $1200-1500 for a Clevo/Sager custom configured notebook that can do that - no need to overpay for Dells or HPs of this world - ram  for it you can buy separately) and just use a ramdisk... ramdisk throughput is 10-100 times more than SSD ? ramdisk throughout is tens of gigabyte/second, SSD is nowhere near that

http://www.ltr-data.se/opencode.html/#ImDisk = free software ( FAQ = http://reboot.pro/topic/15593-faqs-and-how-tos/ )

PS: for example Clevo 15" (or 17" - the difference is neglible) notebook with i4810MQ 2.8-3.8GHz, 2x8gb RAM (2 slots used and 2 free), 128...256gb mSATA, 1TB 7200 HDD, Nvidia GTX870m cost ~$1450 all inclusive, get extra 2x8 so-dimms for $100-150 and you can have 16Gb ramdisk (there are extra slots for one more mSATA and DVD can be replaced by 2nd internal HDD - if you go for 17" model there are even more slots for drives)... Windows 8.1 also does better job with memory usage vs Windows 7 IMHO (my prev. notebook was almost as yours, XPS15" though, retired after 3 years)

[deejjjaaaa]

in Bridge it would typically take 10-12 hours just to generate the previews for me to review them. 

if you can review files sequentially may I suggest FRV ( www.fastrawviewer.com ) and avoid generating previews in Bridge or LR for all files, instead culling them using FRV (which can also prepare some parameters in xmp for Adobe products used in the next step in your workflow)

[Bart_van_der_Wolf]

So my main concern/question, am I missing some parameter somewhere that affects the performance of plug-ins like Topaz?  Or has been pointed out, is Topaz really that CPU intensive and there is nothing I can do short of ordering a desktop Cray supercomputer to speed Topaz up?

Hi Jim,

Just check your Windows Task manager's Performance tab. If the CPUs are maxed out, then the noise processing is CPU bound. You can use the Windows Resource monitor to check for Disk activity, but I suspect the CPU being the most limiting factor.

Topaz Denoise does a great job, but it does some serious number crunching to get there. As suggested, you may want to check out NeatImage as well, because it allows to create a specific profile for your camera per ISO setting (which works quite good), and it uses the GPU capability of your graphics card to speed up the processing.

Cheers,
Bart

[deejjjaaaa]

PS: does OP really have 2000 keepers from a single game ?

[kirkt]

Consider extracting the JPEG thumbnails from the raws for initial culling if attempting to preview the raws is choking your machine.

Then generate the raw previews for the keepers and start your raw workflow  

Dcraw can extract the JPEGs from the raw files in a directory (often these are full res) with the command:

dcraw -e *.CR2

For canon cr2 raw files, .nef for Nikon, etc.

This essentially instructs dcraw to extract the JPEG from every file in the working directory with that extension.

Unless all 2000 shots are keepers. If so, buy a faster mAchine with more ram and a lot of fast drives.
Kirk

[BobShomler]

10 to 12 hours for Bridge cs6 to generate previews on the system you describe seems like way too much.  I have a faster system on which bridge generates high quality previews (but not 100% previews) from Canon 7D 17.9 megapixel cr2 raws at a 45 per minute, and an older slower laptop with I5, win 7-64, 4G ram and internal laptop drive on which bridge generates high quality previews at about 25 per minute.  Do you have you have bridge generate 100% previews checked on?  Or may be something with external drives.

What size are your raw files (megapixels? MBs raw file size?  What have you selected selected in your bridge preview generation?

If you want a faster initial preview in bridge you can instruct bridge to use the from-camera jpeg previews and not generate previews via camera raw: select Prefer Embedded (Faster) bridge preview generation, then move the ones you want to another folder for bridge to generate previews of your selected subset (as kirkt posted).

[deejjjaaaa]

10 to 12 hours for Bridge cs6 to generate previews on the system you describe seems like way too much. 

you can imagine all kind of scenarios including "and my Seagate 5400 rpm external drive" being plugged into USB 2.0 port for example with data traffic going back/forth there

[BobShomler]

There are at least three I/Os for each image preview generation: reading raw file, writing preview to bridge cache, and saving acr image settings -- writing acr metadata -- either to xmp file on disk with raw file or to the camera raw database depending on ACR preferences setting for saving image settings.

OP writes that moving image files to internal hdd reduced preview generation time by half, but this still seems long, would seem like even external hdd should have some disk cache holding some image files.  Wonder if the bridge cache is assigned to internal drive or 5400 rpm external/usb?  Also whether or not 100% previews are being generated.

[albedo13]

OK, I am back with some new images and some additional research...this has been a great learning experience!  And thanks again for all the inputs.  You all were right about re-reading/re-loading the already existing cache files.  On Thursday we shot about 3000 pictures, and starting the whole process from scratch using the SSD connected via USB 3.0 port, Bridge took about 1.5 hours to generate all the previews etc...not less than a minute, but still much better than what I had before.  I will check my settings and also look into the other software suggestions, but the SSD does seem to be helping here.

As I said before, I go through all the image previews once in Bridge and find all the images that are usable.  As far as the number of keepers, our goal is to get at least one picture of each player, and as many action shots of each player as possible, so with approximately 70 kids on just the JV team, we need a lot of images!  So the definition of keeper here is a little different than the Sports Illustrated definition!  I will typically process about 800 per game, by reading the image into ACR to increase exposure, contrast, shadow etc, then opening it, applying the Topaz Denoise, and saving out a jpeg, all done via batch processing.

I have taken the suggestion of using the Resource Monitor within Windows Task Manager to watch what is happening.  Let me preface all this by saying I am a damn user, not a PC admin, so please be merciful to my ignorance!  Looking at the CPU tag in the Resource Monitor, I was surprised to see that Topaz uses about 44% of the CPU as it is running.  There does not appear to be anything else significant running, as the overall CPU level at this same time is less than 55%.  Looking at the Memory tab, it says there is 20 MB reserved for hardware, 5800 MB in use, 10410 MB in standby, and 78 MB free for any given image when Topaz is processing; so it seems like Topaz is not using all of the memory (16 GB) either.  At the bottom of the memory table is this:

Available 10547 MB
Cached    10411 MB
Total        16364 MB
Installed   16384 MB

I don't understand what the "Cached" term here means.

And then watching the Disks tab, I am surprised to see as much internal drive activity as I seem to be seeing.  CS6 is installed on my internal 7200 rpm hard drive, all of my data is on the external SSD, and I think/hope I have all of my scratch disk etc on the external drive.  Every time Topaz kicks off on a new image in batch, I see a huge write occur to my C drive for a file called  AppData\Local\Temp\...\_topazDenoise_rgb .  The Resource Monitor says this is a disk Write at sometimes up to 5,000,000 B/sec when it first kicks off, and then settles down to about 600,000 B/sec as it nears the end of the 2 minute Topaz process, and then it goes away when Topaz has finished that image.  Could this be some kind of Topaz internal scratch file, and if this is being written to my C drive, could this be why my Topaz processing times seem to be pretty much unaffected by my addition of the SSD?  To my untrained eye, it does not seem like CPU or memory are the limiting factors here.  Does a plug-in like Topaz use where you specify the scratch disk to be in the Preferences, or does it go off and find wherever it wants to, and in this case possibly the internal C drive?  Or should I just quit worrying about it, there is nothing I can do (short of buying a newer faster computer, and then I will need suggestions on how to run that by the Significant Other and survive!), and just live with the time it takes and enjoy my SSD for all other applications?  Thanks again for your help!

Jim

[Wayne Fox]

For ingesting, browsing, sorting, culling, tagging image files I believe the best/fastest application on the market today is Photo Mechanic 5, by Camera Bits.

I have several clients who shoot professional sports such as real salt lake, national speed skating , Utah Jazz etc. they all have to submit images to their service, the team or league  during halftimes, which means importing sorting, and output from 1000 to 2000 images in a very short amount of time, and usually within 30 min of the end of the game.  I think they all use photomechanic.