Luminous Landscape Forum
Equipment & Techniques => Computers & Peripherals => Topic started by: Ben Rubinstein on May 04, 2020, 12:44:08 pm
-
Hi folks.
I need a very powerful and fast Capture One workstation. To put it into perspective I'm coming from a i7-8700K with 32GB ram and a Quadro P1000 vid card. It's a good machine but this project requires some serious horse power.
Specifically I need to database 1.6 million raw files (not yet cached), nothing under 36 megapixels, apply styles, check and output to TIFF. I'll be doing it in batches of some 12TB at a time. I'll also need to be running extensive metadata entry and searching and file renaming. I will be building Adobe Bridge Cache for 6TB at a time and batching the TIFF's through Photoshop actions. The above mentioned machine is simply not powerful enough.
Through extensive testing at our studios I've realised that Capture One speed is hugely reliant on the video card. Benchmarked a bunch of our computers and assuming similar read/write speeds, only the video card seemed to make a significant speed difference in building cache and processing. I've been a big fan of Nvidia Quadro cards for a while but I'm not a computer hardware person these days and I don't know which kind of GPU for the usage that Capture One wants, I'm just not up to date.
My current plan is:
Top end i9 or Ryzen processor (does it make a difference for C1 or PS? Do either of them require or can they use all those cores?)
Plenty of ram
NVMe drive (for the database) feeding from a thunderbolt 2 raid drive.
Video card (but what?)
Any other thoughts?
-
Spend the time talking with CH folks. The volume you're talking about may be better done split out further. Finding a way to have 2-?? folks working in parallel will be more important than your one workstation. I think you're going to start with the DAM, figure out how to get the raw images in, adding metadata then pulling it out, applying styles then adding it back in as a TIFF version in the DAM. That way you can revert to the original file at some point in the future. Also go into how the TIFFs will be used, as an end user wanting a PNG or JPEG at some point in the future would need to be fairly automatic.
Doing it directly to an offsite cloud most likely isn't an option, so having a DAM that can be local, with plenty of high speed storage, then a way to manage backups or replicate to the cloud for remote users to interact with.
There are folks with this experience, be ready to pay for their consultations. This isn't a simple NAS plus fast computer problem.
-
Spend the time talking with CH folks. The volume you're talking about may be better done split out further. Finding a way to have 2-?? folks working in parallel will be more important than your one workstation. I think you're going to start with the DAM, figure out how to get the raw images in, adding metadata then pulling it out, applying styles then adding it back in as a TIFF version in the DAM. That way you can revert to the original file at some point in the future. Also go into how the TIFFs will be used, as an end user wanting a PNG or JPEG at some point in the future would need to be fairly automatic.
Doing it directly to an offsite cloud most likely isn't an option, so having a DAM that can be local, with plenty of high speed storage, then a way to manage backups or replicate to the cloud for remote users to interact with.
There are folks with this experience, be ready to pay for their consultations. This isn't a simple NAS plus fast computer problem.
It is actually because the DAM has all been worked out, ditto the workflow. It's a large studio having already paid some big bucks for consultation. I used to manage the studio and am now bidding for the contract. I know the work, workflow and DAM, all I need now is the best hardware compromise. It will of course be a large job with multiple people working on it and multiple computers. I'm working on the whole server question at the same time as you can imagine.
-
Hey Ben,
Cool, then you're covering all the heavy lifting. I would actually consider doing matching workstations & monitors. Using Ryzen 3900X, with a motherboard that has onboard 10gb ethernet and a few NVMe slots. The graphics card may be a more open question, I'd love to see how the PCIe4 W5500 or W5700 does, but there's some benchmarks out there like https://techgage.com/article/a-look-at-phase-one-capture-one-performance/
The one thing i would be debating is if I going faster than 10gbps networking makes sense. Doing 25gb or 40gb on the server side should be straight forward, but do you go faster on the client side. Take away the local copying, just reference everything over the network link.
Have fun and come back with some performance numbers. The storage setup sounds like a lot of fun. How close to a petabyte will it be?
-
Hey Ben,
Cool, then you're covering all the heavy lifting. I would actually consider doing matching workstations & monitors. Using Ryzen 3900X, with a motherboard that has onboard 10gb ethernet and a few NVMe slots. The graphics card may be a more open question, I'd love to see how the PCIe4 W5500 or W5700 does, but there's some benchmarks out there like https://techgage.com/article/a-look-at-phase-one-capture-one-performance/
The one thing i would be debating is if I going faster than 10gbps networking makes sense. Doing 25gb or 40gb on the server side should be straight forward, but do you go faster on the client side. Take away the local copying, just reference everything over the network link.
Have fun and come back with some performance numbers. The storage setup sounds like a lot of fun. How close to a petabyte will it be?
It's interesting that during our benchmarking we didn't see any difference between a regular 500mb/s SSD drive to a 2000mb/s NVMe drive in building cache, applying styles and processing out to TIFF. I'm not sure where the bottleneck is, no doubt the program is simply not working fast enough to use the extra read/write speeds. As such going over 10GBE might be overkill. I am however specifically dedicating NVMe for the catalog itself.
I had seen that benchmarking of GPU after I posted this thread. It's been very useful. I'm not sure the very high end stuff is actually being utilised.
As for storage, yeah. There is 100TB of RAW files, in conversion to TIFF they gain substantially though still not Petabyte territory. It doesn't help that different raw formats output a different percentage of the RAW file when conversion to TIFF. For example with our files from the Leaf Aptus II-8 the increase is 266% whereas the Sony A7rIII files, because the RAW is so big already, give a TIFF which is only 148% bigger. Makes calculating required space somewhat of a pain. On the other hand the Capture One Sessions this stuff is in, tens of thousands of them, take up a lot of space of their own, 3.34% of the raw files themselves, That's already over 3TB that doesn't have to be replicated.
I would actually argue about the matching workstations somewhat. For example I need the very heavy lifting for the Capture One work (those workstations will be identical) but the Photoshop Batch Actions, batch metadata entry, batch renaming, etc can be done via a virtual machine on the NAS because it can simply run 24/7 so ultimate speed is much less of a consideration.
-
Only 100TB of data in, that doesn't hit the fun size... ;D I recall some issues with color consistency issues between frames rendered on AMD & Intel way back, not sure if it's still an issue. You're right that the metadata and such is best done as close to the files & it doesn't impact the output.
-
Only 100TB of data in, that doesn't hit the fun size... ;D I recall some issues with color consistency issues between frames rendered on AMD & Intel way back, not sure if it's still an issue. You're right that the metadata and such is best done as close to the files & it doesn't impact the output.
Thanks for the help Joe!
-
Ben:
It may be work looking at the 3960x
With the additional cores, comes additional threads which could allow dual graphics and the NVME.
Just a thought - I'm sure Joe would have commentary on this as well.
Cheers
Andrew
-
1.6 million raw files
ouch !
-
Top end i9 or Ryzen processor (does it make a difference for C1 or PS? Do either of them require or can they use all those cores?)
Plenty of ram
NVMe drive (for the database) feeding from a thunderbolt 2 raid drive.
Video card (but what?)
Any other thoughts?
SuperMicro has MBs for 2-4 CPUs and 2+ GPUs plus TBs of RAM ...
-
ouch !
That's not the worst of it. These 1.6 million files are in some 25000+ sessions of repro work made with capture one versions going back to C1-6, thousands of said sessions are using specific LCC calibration, there are some 30+ different ICC profiles, 20 of so unique styles (as the studio added camera, lens and lighting systems plus a whole bunch of unique copy stand setups) and they all have to be sorted out, upgraded, have new updated ICC's applied, converted to preservation standards, processed as preservation grade TIFF's, renamed to fit the requirements of the new cataloguing system and metadata applied all while keeping them in the same basic folder structure as apparently the database isn't quite ready yet. Oh and upper management can't quite work out why it can't all be unattended batch processed and why it's going to take a) so long b) so much money c) require skilled labour.
:D :D :D
-
In general pretty easy.
128GB —> 3900
256Gb —> 3960
I can recommend MSI boards for both, but Gigabyte or asus are great, too.
Graphic card depends on the budget. I would recommend a RTX 2070 super until the next generation comes along. If money isn’t an issue you can go with two 2080 Ti. Capture one actually uses two cards quite well.
Storage is more difficult. You could look into intels optan storage. It is super expensive and hasn’t much benefits for normal use cases, however, it might be useful to you. Sadly there aren’t many reviews regarding optan and C1 or adobe products.
Samsung SSDs m2 are great as well. Or one of the faster pcie 4 ones.
-
128GB —> 3900
256Gb —> 3960
You'll have to forgive me, I don't understand the quoted part of your post.
-
If you need 256gb of Memory you will need a different Board CPU. So inwould Go for the Ryzen 3960
If your think 128GB is enough you can for with the ryzen 3900.
The 3900 is certainly fast enough and the additional cores 12 vs 24 isn’t doing much.
-
If you need 256gb of Memory you will need a different Board CPU. So inwould Go for the Ryzen 3960
If your think 128GB is enough you can for with the ryzen 3900.
The 3900 is certainly fast enough and the additional cores 12 vs 24 isn’t doing much.
Thank you, that makes sense.
-
Oh and upper management can't quite work out why it can't all be unattended batch processed and why it's going to take a) so long b) so much money c) require skilled labour.
OUCH!
These 1.6 million files are in some 25000+ sessions of repro work made with capture one versions going back to C1-6, thousands of said sessions are using specific LCC calibration, there are some 30+ different ICC profiles, 20 of so unique styles (as the studio added camera, lens and lighting systems plus a whole bunch of unique copy stand setups) and they all have to be sorted out, upgraded, have new updated ICC's applied, converted to preservation standards, processed as preservation grade TIFF's, renamed to fit the requirements of the new cataloguing system and metadata applied all while keeping them in the same basic folder structure as apparently the database isn't quite ready yet.
Yea, hope you are budgeting for plenty of small victory bits, a round or two of drinks or such. With big projects like this, selling the value / function of all the historical is really hard. Are they currently adding to this collection (pre-covid)? Are they still using the old processes & procedures? It must hurt that this work from home time would be ideal for cleaning & organizing data, and the platform you need to do this within is still under development.
-
Storage is more difficult. You could look into intels optan storage. It is super expensive and hasn’t much benefits for normal use cases, however, it might be useful to you. Sadly there aren’t many reviews regarding optan and C1 or adobe products.
Samsung SSDs m2 are great as well. Or one of the faster pcie 4 ones.
Optane is a type of memory chip, NVMe is a connection method. Optane chips is latency tuned, while other chips are capacity tuned. A quality Samsung 970 Pro or Crucial P2 NVMe drive is plenty fast, and a lot cheaper than a Optane card of the same capacity.
-
Well we don’t really know how it would behave with c1 or LR/ bridge. Optane won’t help at all as image storage, but could be beneficial as Catalouge / meta data / preview drive.
I decided not to test is and invest in 4 x m2s Samsung 960 2TB drives.
I would want to test a optane Drive in real world usage at some point.
Optane is a type of memory chip, NVMe is a connection method. Optane chips is latency tuned, while other chips are capacity tuned. A quality Samsung 970 Pro or Crucial P2 NVMe drive is plenty fast, and a lot cheaper than a Optane card of the same capacity.
-
Yea, hope you are budgeting for plenty of small victory bits, a round or two of drinks or such. With big projects like this, selling the value / function of all the historical is really hard. Are they currently adding to this collection (pre-covid)? Are they still using the old processes & procedures? It must hurt that this work from home time would be ideal for cleaning & organizing data, and the platform you need to do this within is still under development.
The museums studio is currently working under the local laws, i.e. slow. They are converting over to a new system but very very slowly and this project is only part of a much larger project to fully catalogue and database 8 years of digitization carried out under a budget philosophy of 'do the digitization because that's urgent as most of our collections are borrowed, the rest we can do later'. My bid for this project is specifically based on a bunch of my own projects being on hold till 2020 due to the current situation giving me the down time to concentrate on what is a monotonous task.
-
Well we don’t really know how it would behave with c1 or LR/ bridge. Optane won’t help at all as image storage, but could be beneficial as Catalouge / meta data / preview drive.
I decided not to test is and invest in 4 x m2s Samsung 960 2TB drives.
I would want to test a optane Drive in real world usage at some point.
It would be great if there were was a full benchmarking of hardware solutions for photographers. All the benchmarking seems to be based on games, no doubt because that is where the real money is. What do the video guys use as a benchmarking resource? Even that isn't going to be the same as LR/C1 benchmarking which would be what we really need and as you mention, would be weighed towards database usage rather than simply processing and preview generation.
-
Puget Systems has done benchmarks with Photoshop and Lightroom and compared hardware:
Pugetsystems.com
-
Puget Systems has done benchmarks with Photoshop and Lightroom and compared hardware:
Pugetsystems.com
Thanks!
-
Hi Ben
Interesting project what did you end up with?
Henrik
-
Hi Ben
Interesting project what did you end up with?
Henrik
Covid delays is the true answer to that question. :(
-
Sorry to headroom, it has cause trouble for many of us unfortunately
-
Sorry to headroom, it has cause trouble for many of us unfortunately
Yup, the government here has shut down all funding for Heritage programs while it deals with keeping the economy moving. I can't blame them but it does put people like myself in a waiting pattern.
-
Just read my own message - note to self, don’t reply without glasses 🤓- what I meant to say, sorry to hear!
Yes, here in Australia it all went to a standstill late March early April - I unfortunately got made redundant.
For your system, working with databases, it’s all about moving data around as fast as possible - the m.2 drives are where you want to be, although their capacity is limited, I have 2 x 2Tb in RAID-0 to see who they perform - all three of my m.2’s are PCIe Gen4, Seagate 520 1Tb for boot, and 2x Corsair MP600 2TB in RAID-0, when funding permits we will go to 4x 😉👍
Your bottleneck will be your mechanical disks and your network
All the best
Henrik
-
Just read my own message - note to self, don’t reply without glasses 🤓- what I meant to say, sorry to hear!
Yes, here in Australia it all went to a standstill late March early April - I unfortunately got made redundant.
For your system, working with databases, it’s all about moving data around as fast as possible - the m.2 drives are where you want to be, although their capacity is limited, I have 2 x 2Tb in RAID-0 to see who they perform - all three of my m.2’s are PCIe Gen4, Seagate 520 1Tb for boot, and 2x Corsair MP600 2TB in RAID-0, when funding permits we will go to 4x 😉👍
Your bottleneck will be your mechanical disks and your network
All the best
Henrik
Sucks Henrik, it's a tough time. I hope you get sorted out as soon as possible! I agree with you about M2 drives but as I'm going to be working with 6-12TB in a go, I need systems that can affordably deal with that amount of data and provide redundancy. I'm currently looking at a QNAP desktop NAS as I have a good relationship with the dealer here already. Specifically the 672N with a 10GBE upgrade card. I'd take the 672XT but they don't have it in this country worse luck.
-
Thanks Ben,
if you can hang out till the 4TB M/2 disks comes out or become a little more affordable - then run 4x 4TB with this https://www.gigabyte.com/Solid-State-Drive/AORUS-Gen4-AIC-SSD-8TB#kf
Just an idea ;-) a RAID-0 nearly 15gb/s
Henrik
-
Thanks Ben,
if you can hang out till the 4TB M/2 disks comes out or become a little more affordable - then run 4x 4TB with this https://www.gigabyte.com/Solid-State-Drive/AORUS-Gen4-AIC-SSD-8TB#kf
Just an idea ;-) a RAID-0 nearly 15gb/s
Henrik
I've spent the day thinking exactly about something like this. Specifically stuffing a case full of 2TB SSD's running in RAID 0 (gulp) and screw the server, replace it with a simple raid 5 backup box to offset the risks of using raid 0 on the working drives. To be frank I was thinking of running regular SSD's via sata. That way I wouldn't have to worry about running out of PCI lanes. It won't be the same speed but it will still considerably faster than the NAS. I like the look of that Gigabyte solution but I'd need two of them and that requires a motherboard with 3x PCIe x16 slots (two of these plus the video card) and a processor with enough pcie lanes (40 or so) to keep up with it. The sata solution would be simpler if not as ridiculously fast.
-
I just bought a Lenovo Legion 5 laptop (to be used for tethered shoots) so I finally had a higher end build to do some benchmarking with. AMD Ryzen 7 4800HS, 16GB ram, GTX 1650 Ti graphics, Samsung NVME M.2 drive.
The test was a catalog import of 700 D800 raw files with processing, style and processing changes, output to tiff, bridge cache build, photoshop batch to jpg2000.
The CPU didn't go over 40%. Ditto RAM and GPU. Disk speed peaked at 169 mb/s.
So basically, not much need for high end gear, the software can't keep up with it for batch tasks of the kind I've discussed here. Heck my laptops capabilities are not even hitting the half way mark. I'm thinking of using a virtual machine to double up the tasks simply to use the capabilities the computer has. The high end computer build I was imagining is simply overkill at this point. The hardware is not the limitation.
-
I just bought a Lenovo Legion 5 laptop (to be used for tethered shoots) so I finally had a higher end build to do some benchmarking with. AMD Ryzen 7 4800HS, 16GB ram, GTX 1650 Ti graphics, Samsung NVME M.2 drive.
The test was a catalog import of 700 D800 raw files with processing, style and processing changes, output to tiff, bridge cache build, photoshop batch to jpg2000.
The CPU didn't go over 40%. Ditto RAM and GPU. Disk speed peaked at 169 mb/s.
So basically, not much need for high end gear, the software can't keep up with it for batch tasks of the kind I've discussed here. Heck my laptops capabilities are not even hitting the half way mark. I'm thinking of using a virtual machine to double up the tasks simply to use the capabilities the computer has. The high end computer build I was imagining is simply overkill at this point. The hardware is not the limitation.
Photoshop batch to jpeg2000.... try Affinity Photo and it will be 4x faster....using all cores and the GPU... but maybe there is no jpeg 2000 output...
I have a raid0 4x 1tb samsung 970 on a 16 x pci3.0 connection and it can do about 400-7000mb/sec making a copy. However with porgrams like photoshop it also has to do other things while saving so it is about 1000mb a second if the file is not compressed. If it is compressed it takes much longer.
it is not just the hardware that defines the speed
-
Photoshop batch to jpeg2000.... try Affinity Photo and it will be 4x faster....using all cores and the GPU... but maybe there is no jpeg 2000 output...
I have a raid0 4x 1tb samsung 970 on a 16 x pci3.0 connection and it can do about 400-7000mb/sec making a copy. However with porgrams like photoshop it also has to do other things while saving so it is about 1000mb a second if the file is not compressed. If it is compressed it takes much longer.
it is not just the hardware that defines the speed
The question is whether is can provide lossless compressed jpg2000, will preserve the color space and what size the resulting files will be. All of these have been problems with other programs I've tried. I'm not sure how Adobe do it but their files are really small while still being lossless. I will try Affinity Photo though, thanks for the heads up!
EDIT, not according to the specs list, no jpg2000
I tried XNConvert yesterday, it's slower than PS, strips the color space and replaces the header with sRGB and has no lossless compression, you either get no compression or lossy.
-
Hi Ben,
Not sure what your tolerance for command line utilities is and you might need to spend some time experimenting, but if acceptable, you could look at GraphicsMagick, which is (apparently), a more performant version of ImageMagick.
http://www.graphicsmagick.org/
https://imagemagick.org/index.php
The somewhat dense man(ual) page for it is here: http://www.graphicsmagick.org/GraphicsMagick.html but as it is based on Imagemagick, you will find more user friendly tutorials, etc. there: https://legacy.imagemagick.org/Usage/
Regards.
-
Hi Ben,
Not sure what your tolerance for command line utilities is and you might need to spend some time experimenting, but if acceptable, you could look at GraphicsMagick, which is (apparently), a more performant version of ImageMagick.
http://www.graphicsmagick.org/
https://imagemagick.org/index.php
The somewhat dense man(ual) page for it is here: http://www.graphicsmagick.org/GraphicsMagick.html but as it is based on Imagemagick, you will find more user friendly tutorials, etc. there: https://legacy.imagemagick.org/Usage/
Regards.
Wouldn't be my first choice to be honest. I'll have a look though.
-
Hi Ben,
Not sure what your tolerance for command line utilities is and you might need to spend some time experimenting, but if acceptable, you could look at GraphicsMagick, which is (apparently), a more performant version of ImageMagick.
http://www.graphicsmagick.org/
https://imagemagick.org/index.php
The somewhat dense man(ual) page for it is here: http://www.graphicsmagick.org/GraphicsMagick.html but as it is based on Imagemagick, you will find more user friendly tutorials, etc. there: https://legacy.imagemagick.org/Usage/
Regards.
I've spent the day with both programs, graphicsmagick does compression even when it says it isn't but imagemagick is working perfectly as long as I keep within sRGB for JPEG2000 which I've now learned is due to the JPEG2000 specification. Thank you for that!
-
I've spent the day with both programs, graphicsmagick does compression even when it says it isn't ...
Good to hear. If you find graphicsmagick, despite the above, has advantages, I'd be surprised if the author didn't fix it promptly, were you to get in touch. Open source is usually pretty good like that :)
-
If the TIFF -> JPG2000 step proves a bottleneck still and depending on your developer abilities, this may be of interest:
https://jpeg.org/jpeg2000/htj2k.html
-
Just to throw it out there - have you looked at Photo Mechanic? From a batch processing perspective, it may save you some effort.
-
If the TIFF -> JPG2000 step proves a bottleneck still and depending on your developer abilities, this may be of interest:
https://jpeg.org/jpeg2000/htj2k.html
I'm afraid jpeg2000 is part of the job specification, it's not my choice.
-
Just to throw it out there - have you looked at Photo Mechanic? From a batch processing perspective, it may save you some effort.
Wouldn't really fit into the workflow I'm afraid, I'm being handed tens of thousands of Capture One sessions with the C1 specific processing inside.