Capture One heavy workstation build

[JimGoshorn]

Puget Systems has done benchmarks with Photoshop and Lightroom and compared hardware:

Pugetsystems.com

[Ben Rubinstein]

Puget Systems has done benchmarks with Photoshop and Lightroom and compared hardware:

Pugetsystems.com

Thanks!

[tived]

Hi Ben

Interesting project what did you end up with?

Henrik

[Ben Rubinstein]

Hi Ben

Interesting project what did you end up with?

Henrik

Covid delays is the true answer to that question. 

[tived]

Sorry to headroom, it has cause trouble for many of us unfortunately

[Ben Rubinstein]

Sorry to headroom, it has cause trouble for many of us unfortunately

Yup, the government here has shut down all funding for Heritage programs while it deals with keeping the economy moving. I can't blame them but it does put people like myself in a waiting pattern.

[tived]

Just read my own message - note to self, don’t reply without glasses 🤓- what I meant to say, sorry to hear!

Yes, here in Australia it all went to a standstill late March early April - I unfortunately got made redundant.

For your system, working with databases, it’s all about moving data around as fast as possible - the m.2 drives are where you want to be, although their capacity is limited, I have  2 x 2Tb in RAID-0 to see who they perform - all three of my m.2’s are PCIe Gen4, Seagate 520 1Tb for boot, and 2x Corsair MP600 2TB in RAID-0, when funding permits we will go to 4x 😉👍

Your bottleneck will be your mechanical disks and your network

All the best

Henrik

[Ben Rubinstein]

Just read my own message - note to self, don’t reply without glasses 🤓- what I meant to say, sorry to hear!

Yes, here in Australia it all went to a standstill late March early April - I unfortunately got made redundant.

For your system, working with databases, it’s all about moving data around as fast as possible - the m.2 drives are where you want to be, although their capacity is limited, I have  2 x 2Tb in RAID-0 to see who they perform - all three of my m.2’s are PCIe Gen4, Seagate 520 1Tb for boot, and 2x Corsair MP600 2TB in RAID-0, when funding permits we will go to 4x 😉👍

Your bottleneck will be your mechanical disks and your network

All the best

Henrik

Sucks Henrik, it's a tough time. I hope you get sorted out as soon as possible! I agree with you about M2 drives but as I'm going to be working with 6-12TB in a go, I need systems that can affordably deal with that amount of data and provide redundancy. I'm currently looking at a QNAP desktop NAS as I have a good relationship with the dealer here already. Specifically the 672N with a 10GBE upgrade card. I'd take the 672XT but they don't have it in this country worse luck.

[tived]

Thanks Ben,

if you can hang out till the 4TB M/2 disks comes out or become a little more affordable -  then run 4x 4TB with this https://www.gigabyte.com/Solid-State-Drive/AORUS-Gen4-AIC-SSD-8TB#kf

Just an idea ;-) a RAID-0 nearly 15gb/s

Henrik

[Ben Rubinstein]

Thanks Ben,

if you can hang out till the 4TB M/2 disks comes out or become a little more affordable -  then run 4x 4TB with this https://www.gigabyte.com/Solid-State-Drive/AORUS-Gen4-AIC-SSD-8TB#kf

Just an idea ;-) a RAID-0 nearly 15gb/s

Henrik

I've spent the day thinking exactly about something like this. Specifically stuffing a case full of 2TB SSD's running in RAID 0 (gulp) and screw the server, replace it with a simple raid 5 backup box to offset the risks of using raid 0 on the working drives. To be frank I was thinking of running regular SSD's via sata. That way I wouldn't have to worry about running out of PCI lanes. It won't be the same speed but it will still considerably faster than the NAS. I like the look of that Gigabyte solution but I'd need two of them and that requires a motherboard with 3x PCIe x16 slots (two of these plus the video card) and a processor with enough pcie lanes (40 or so) to keep up with it. The sata solution would be simpler if not as ridiculously fast.

[Ben Rubinstein]

I just bought a Lenovo Legion 5 laptop (to be used for tethered shoots) so I finally had a higher end build to do some benchmarking with. AMD Ryzen 7 4800HS, 16GB ram, GTX 1650 Ti graphics, Samsung NVME M.2 drive.

The test was a catalog import of 700 D800 raw files with processing, style and processing changes, output to tiff, bridge cache build, photoshop batch to jpg2000.

The CPU didn't go over 40%. Ditto RAM and GPU. Disk speed peaked at 169 mb/s.

So basically, not much need for high end gear, the software can't keep up with it for batch tasks of the kind I've discussed here. Heck my laptops capabilities are not even hitting the half way mark. I'm thinking of using a virtual machine to double up the tasks simply to use the capabilities the computer has. The high end computer build I was imagining is simply overkill at this point. The hardware is not the limitation.

[kers]

I just bought a Lenovo Legion 5 laptop (to be used for tethered shoots) so I finally had a higher end build to do some benchmarking with. AMD Ryzen 7 4800HS, 16GB ram, GTX 1650 Ti graphics, Samsung NVME M.2 drive.

The test was a catalog import of 700 D800 raw files with processing, style and processing changes, output to tiff, bridge cache build, photoshop batch to jpg2000.

The CPU didn't go over 40%. Ditto RAM and GPU. Disk speed peaked at 169 mb/s.

So basically, not much need for high end gear, the software can't keep up with it for batch tasks of the kind I've discussed here. Heck my laptops capabilities are not even hitting the half way mark. I'm thinking of using a virtual machine to double up the tasks simply to use the capabilities the computer has. The high end computer build I was imagining is simply overkill at this point. The hardware is not the limitation.

Photoshop batch to jpeg2000.... try Affinity Photo and it will be 4x faster....using all cores and the GPU... but maybe there is no jpeg 2000 output...

I have a raid0 4x 1tb samsung 970 on a 16 x pci3.0 connection and it can do about 400-7000mb/sec making a copy. However with porgrams like photoshop it also has to do other things while saving so it is about 1000mb a second if the file is not compressed. If it is compressed it takes much longer.

it is not just the hardware that defines the speed

[Ben Rubinstein]

Photoshop batch to jpeg2000.... try Affinity Photo and it will be 4x faster....using all cores and the GPU... but maybe there is no jpeg 2000 output...

I have a raid0 4x 1tb samsung 970 on a 16 x pci3.0 connection and it can do about 400-7000mb/sec making a copy. However with porgrams like photoshop it also has to do other things while saving so it is about 1000mb a second if the file is not compressed. If it is compressed it takes much longer.

it is not just the hardware that defines the speed

The question is whether is can provide lossless compressed jpg2000, will preserve the color space and what size the resulting files will be. All of these have been problems with other programs I've tried. I'm not sure how Adobe do it but their files are really small while still being lossless. I will try Affinity Photo though, thanks for the heads up!

EDIT, not according to the specs list, no jpg2000

I tried XNConvert yesterday, it's slower than PS, strips the color space and replaces the header with sRGB and has no lossless compression, you either get no compression or lossy.

[richs]

Hi Ben,

Not sure what your tolerance for command line utilities is and you might need to spend some time experimenting, but if acceptable, you could look at GraphicsMagick, which is (apparently), a more performant version of ImageMagick.

http://www.graphicsmagick.org/

https://imagemagick.org/index.php

The somewhat dense man(ual) page for it is here: http://www.graphicsmagick.org/GraphicsMagick.html but as it is based on Imagemagick, you will find more user friendly tutorials, etc. there: https://legacy.imagemagick.org/Usage/

Regards.

[Ben Rubinstein]

Hi Ben,

Not sure what your tolerance for command line utilities is and you might need to spend some time experimenting, but if acceptable, you could look at GraphicsMagick, which is (apparently), a more performant version of ImageMagick.

http://www.graphicsmagick.org/

https://imagemagick.org/index.php

The somewhat dense man(ual) page for it is here: http://www.graphicsmagick.org/GraphicsMagick.html but as it is based on Imagemagick, you will find more user friendly tutorials, etc. there: https://legacy.imagemagick.org/Usage/

Regards.

Wouldn't be my first choice to be honest. I'll have a look though.

[Ben Rubinstein]

Hi Ben,

Not sure what your tolerance for command line utilities is and you might need to spend some time experimenting, but if acceptable, you could look at GraphicsMagick, which is (apparently), a more performant version of ImageMagick.

http://www.graphicsmagick.org/

https://imagemagick.org/index.php

The somewhat dense man(ual) page for it is here: http://www.graphicsmagick.org/GraphicsMagick.html but as it is based on Imagemagick, you will find more user friendly tutorials, etc. there: https://legacy.imagemagick.org/Usage/

Regards.

I've spent the day with both programs, graphicsmagick does compression even when it says it isn't but imagemagick is working perfectly as long as I keep within sRGB for JPEG2000 which I've now learned is due to the JPEG2000 specification. Thank you for that!

[richs]

I've spent the day with both programs, graphicsmagick does compression even when it says it isn't ...

Good to hear. If you find graphicsmagick, despite the above, has advantages, I'd be surprised if the author didn't fix it promptly, were you to get in touch. Open source is usually pretty good like that

[richs]

If the TIFF -> JPG2000 step proves a bottleneck still and depending on your developer abilities, this may be of interest:

https://jpeg.org/jpeg2000/htj2k.html

[Joe Towner]

Just to throw it out there - have you looked at Photo Mechanic?  From a batch processing perspective, it may save you some effort.

[Ben Rubinstein]

If the TIFF -> JPG2000 step proves a bottleneck still and depending on your developer abilities, this may be of interest:

https://jpeg.org/jpeg2000/htj2k.html

I'm afraid jpeg2000 is part of the job specification, it's not my choice.