Image processing beyond 4GB -- any options?

[bradleygibson]

Over the past couple of days, I've had the "pleasure" of using Photoshop CS3 to work on a 154Mpxl panorama, and I've come to the realization that the "4GB" limit imposed by 32-bit applications (more like 2 or 3GB in practice) is causing me to tear my hair out (some operations took 45 minutes to complete because there was so much virtual memory (hard disk) swapping going on).

Is anyone aware of any 64-bit image processing applications?  I am not referring to 64-bit pixels, but to programs which can make use of more than 4GB of physical memory.

Tips, leads or specially-compiled versions of Photoshop would be most welcome!

Thanks in advance,
Brad

[kal]

Over the past couple of days, I've had the "pleasure" of using Photoshop CS3 to work on a 154Mpxl panorama, and I've come to the realization that the "4GB" limit imposed by 32-bit applications (more like 2 or 3GB in practice) is causing me to tear my hair out (some operations took 45 minutes to complete because there was so much virtual memory (hard disk) swapping going on).

Is anyone aware of any 64-bit image processing applications?  I am not referring to 64-bit pixels, but to programs which can make use of more than 4GB of physical memory.

Tips, leads or specially-compiled versions of Photoshop would be most welcome!

Thanks in advance,
Brad
[{POST_SNAPBACK}][/a]

I know this is not the answer you (and most people around) wanted to hear, but... you can compile (or get precompiled versions) of the gimp for 64 bit OSs. That would give you access to a 40-bit physical memory space on opteron, i.e. 1TB (maybe "only" 36-bit == 64GB with intel processors? fixme please)

Of course you loose 16-bit editing, photoshop plugins and a boatload of features you probably liked so much...

Thai said, I assumed that a decent 64-bit OS should give some advantage to 32-bit applications too. Application cannot access more than 2/3/4GB of memory, but data that need to be written out can be instead cached by the OS in memory, assuming your physical memory exceeds the 4GB limit.

You may also want to ask the guys at [a href=\"http://haltadefinizione.deagostini.it/]haltadefinizione.deagostini.it[/url] how did they manage the huge images they have on display.

HTH

[DiaAzul]

Just to throw something else into the mix...Another option is to relocate you swap file to faster hard disks or even solid state disks. You might want to read through the linked article to see if it gives you any more ideas. As the price of CF cards is dropping quickly you may want to look at using a couple to support the photoshop swap disk.

USB/CF Card Swap Disks

[kaelaria]

Why would you gimp yourself by moving to a swap over USB??  Yes flash media is flaster than a hard drive, but you forget the other part - the interface.

[BernardLanguillier]

Not sure how it works on Windows, but on OSX PS CS3 is able to use RAM as swap beyond its 32 bits adressing space. This is what drove me to get 16GB of RAM on my new Mac Pro.

On top of that, set up 3 Raptors or SSD disks in Raid 0 and things should be much faster.

Cheers,
Bernard

[John.Murray]

Win64 versions work the same way.  It will be nice seeing 64bit desktops became a commodity item, allowing Adobe to release a true 64bit PS.  Here is Adobe's explanation of PS memory usage in a 64Bit environment (CS2 specific):

http://kb.adobe.com/selfservice/viewConten...20005&sliceId=2

FWIW, paint.net running under a 64bit .net runtime is very impressive; on a test server running Win2k3R2 with 64GB RAM, I was able to load dozens of 30D raw files with no swapping activity at all - too bad paint.net's internal processing is only 8-bit . . . .

-john

[tived]

Hi,

if you are staying in windows, windows xp x64 is of writing very stable, provided you have some decent hardware. x64 is able access and address more RAM then its x32 counterpart.
 Even though Adobe PS isn't a x64bit app it is still able access more ram then in windows xp x32 (AFAIK). I am sure Vista x64 will be similar but should be able to access even more memory again (Ultimate version)

However, your system, regardless of OS, is only as good as your weakest link and usually it is limited by your harddisks sub-system. As suggested by others here, a RAID-0 (Stripping) would be faster with 2 or more drives (well, you have to have atleast two otherwise there is nowhere to stripe to/with :-) ) and surely a SolidStateDrive would be faster again, however, if you have money for the later, then you might as well get as much ram as you can.

In order to get more the 4Gb, you will mostlikely have to get a workstation mainboard which supports more then this, there are a couple of desktop single processor boards, that came out last or early this year, that could take 8gb of ram, but with the workstation boards, you can use 16, 32, 64 or even 128Gb or ram (tho its going to cost!) see www.tyan.com or www.supermicro.com and ofcourse www.intel.com however, the first two are usually faster and better featured.

I think the MacPro is maxing out at 16Gb using Intel boards 5xxx series.

if you have any questions, ask away!

Good luck

Henrik

PS: Being a PC person, I have to admit that the Macpro offers very good performance and value for the money (...don't ever quote me of saying this!) :-)

[tived]

just for the record, I still think working on images in 2Gb and higher is slow, even with 8gb or ram and Dual Dual core processor (in my case AMD Opteron 285s)
so fret not.

However, it is noticable faster then it was on my P4 3.2Ghz HT computer with 2GB, just a few years ago...

Henrik

PS: ask yourself this, it is worth it, or are you better of having an extra glass of wino while working...NB! Health Warning!! if more then two bottles per session! upgradeat any cost!!

[BernardLanguillier]

FWIW, paint.net running under a 64bit .net runtime is very impressive; on a test server running Win2k3R2 with 64GB RAM, I was able to load dozens of 30D raw files with no swapping activity at all - too bad paint.net's internal processing is only 8-bit . . . .

[a href=\"index.php?act=findpost&pid=134640\"][{POST_SNAPBACK}][/a]

Wow.. 64 GB RAM... how much did you pay for that machine?

OWC now sells 32GB packages for the Mac Pro at about 5.000 US$, I guess that I'll wait a bit.

Cheers,
Bernard

[John.Murray]

Wow.. 64 GB RAM... how much did you pay for that machine?

OWC now sells 32GB packages for the Mac Pro at about 5.000 US$, I guess that I'll wait a bit.

Cheers,
Bernard
[a href=\"index.php?act=findpost&pid=134753\"][{POST_SNAPBACK}][/a]

LOL - it's a new server, we are looking at replacing 3 servers with it, using VM (virtual machine) partitioning.  For laughs, I initially installed Vista on it with suprisingly few driver issues.  The internal 6-drive 15k RPM SSCSI configured as RAID-10 scored a 9.8 on Vista's system performance applet

[bradleygibson]

Thanks for the replies, guys.

I am running OS X with 8GB ram and am already getting the benefit of the extra 4GB, but working with 150MP images non-destructively (adjustment layers, smart objects, smart filters, etc.) is terribly slow.  I've explored hardware RAID 0'ing Raptors, 15K RPM drives, and others, but even the fastest drives are only 2-3x faster than what I have now--multiple orders of magnitude slower than directly addressible physical RAM.  Some operations are taking 45 *minutes* to complete (hard drive swapping the whole time...).

Unfortunately I do want to keep editing in  at least 16-bits per channel, so that rules out GIMP (AFAIK) and Paint.NET (!).  That latter option was one that I hadn't considered...

Even with a huge RAM caching scheme, copying multiple gigabytes of RAM from within the editor's memory space to the cache is hurting performance, although without any true 64-bit solutions, it may be the best way forward at the moment.

Anyway, a big thank you for all the suggestions!  I'll keep my eyes peeled for price drops on 32GB+ machines and/or true 64-bit editing solutions.

Best regards,
Brad

[wesley]

Thanks for the replies, guys.

I am running OS X with 8GB ram and am already getting the benefit of the extra 4GB, but working with 150MP images non-destructively (adjustment layers, smart objects, smart filters, etc.) is terribly slow.  I've explored hardware RAID 0'ing Raptors, 15K RPM drives, and others, but even the fastest drives are only 2-3x faster than what I have now--multiple orders of magnitude slower than directly addressible physical RAM.  Some operations are taking 45 *minutes* to complete (hard drive swapping the whole time...).
[a href=\"index.php?act=findpost&pid=135356\"][{POST_SNAPBACK}][/a]

Hello Brad,

All things considered, it looks like your bottle neck could be CPU related. Your 150Mb files are adequately handled by system RAM (CS3 needs approx x3 to x5 file size for optimum speed). Perhaps your CS3 speed could be boosted by having a high speed quad core setup (eg 3.0ghz) since some CS3 functions are quad core aware. Just some thoughts.

Best regards
Wesley

[tived]

Given how we spend 10s of thousands of dollars on some of these machines here, there should be a better way to test and confirm where the bottlenecks are in a system (I should know better at least)

It would be really great if we had a "best Photoshop setup" guide, covering both hardware (both PC and Mac and ??) and software.

I know a bit but not enough, but I am sure it would be something we could all benefit from.

Brad,

it does sound a bit odd, that your system slows down this much. care to give a complete description of your system (I know that there are differences between Windoze and Mac's) but it could give a bitter picture knowing more details.

Also, what file format are you using and is the 150mb file, the size in photoshop or when you view the file size under its properties?

even my machine don't take that long with 2Gb files!! ??

thanks

Henrik

[BernardLanguillier]

Thanks for the replies, guys.

I am running OS X with 8GB ram and am already getting the benefit of the extra 4GB, but working with 150MP images non-destructively (adjustment layers, smart objects, smart filters, etc.) is terribly slow.  [a href=\"index.php?act=findpost&pid=135356\"][{POST_SNAPBACK}][/a]

Brad,

I have dealt wth 100+ MP images on my Mac Pro without noticing such huge performance panalities.

The difference might be that I am not using Smart objects.

Regards,
Bernard

[bradleygibson]

Hello Brad,

All things considered, it looks like your bottle neck could be CPU related. Your 150Mb files are adequately handled by system RAM (CS3 needs approx x3 to x5 file size for optimum speed). Perhaps your CS3 speed could be boosted by having a high speed quad core setup (eg 3.0ghz) since some CS3 functions are quad core aware. Just some thoughts.

Best regards
Wesley
[a href=\"index.php?act=findpost&pid=135357\"][{POST_SNAPBACK}][/a]

Hi, Wesley,

My files get quite a bit larger than normal because of all the adjustment layers/smart filters and smart objects that I use for a non-destructive workflow...  I'm defintely not CPU-bound during procssing.  (That would be nice, though!  At least I'd know how to fix that issue!    )

--

Hi, Bernard,

Yes, I agree--when I edit a big file destructively (and even just using adjustment layers) things are much more well-behaved.

--

Tived,

My system is a dual G5 2.7 GHz Mac with 8GB of RAM installed, a single 7200RPM HD.  It's nothing special in terms of hardware.  My CPU doesn't typically exceed 30 or 40% during these long operations as it's waiting on the disk.

Keep in mind the image is not 150MB--it's 150MPxl--at 48bits per pixel.  I can only save it as .psb (large photoshop document) as the file save is approximately 5GB (too large for TIFF or .PSD).

It really is just too huge for 32-bit addressing, and combined with the way history works, each time I perform a new operation, large amounts of data end up getting pushed to disk.  Working traditionally with destructive operations helps immensely with performance and gives me 'normal' filesizes.

I was just hoping there was a 64-bit application out there that I could use to continue with a non-destructive workflow.

Thanks again for all the suggestions, guys.  It looks like I just have to be patient for a solution.  

Kind regards,
Brad

[wesley]

Keep in mind the image is not 150MB--it's 150MPxl--at 48bits per pixel.  I can only save it as .psb (large photoshop document) as the file save is approximately 5GB (too large for TIFF or .PSD).

Hi Brad,

I think we've found your problem. Your saved file is is 5GB. But what is really important is the file size in CS2, it can be found under 'Document Size' at the bottom of the image. CS2 has to have at least x3 to x5 (possible even x6) worth of RAM for it not to hit the HDD for scratch disk. If you pump your RAM to x5 of the file size in CS2, OS X (or even XP64 or Vista) will use the extra RAM as scratch disk. That is what you are looking for, CS2 hitting RAM for scratch disk instead of HDD.

Just a quick and dirty estimate, I figure you'll need about 5GB x5 worth of RAM to avoid hitting the HDD scratch disk. I know it sounds ridiculous but you might need a 32GB RAM machine. The question is whether the increased workflow productivity will give you increased profits to cover the outlay.

Btw, looking at Adobe's progress with CS3 vs CS2, I don't know if they will upgrade the entire program to 64bit. From what I have read, it seems that the benefits are that much compared to the effort and time for conversion.

Best regards
Wesley

[thompsonkirk]

That was a confusing point:  I too - liker Bernard - mistakenly thought you were talking about a 150 MB file, not a 5 GB one!

Out of curiosity, what sort of images are you working on?  Producing panoramas from scanned large-format film, or digital-back images?  How large a print are you making?  Can none of the layers be merged without 'destruction'?

Kirk

[BernardLanguillier]

That was a confusing point:  I too - liker Bernard - mistakenly thought you were talking about a 150 MB file, not a 5 GB one!
[a href=\"index.php?act=findpost&pid=135794\"][{POST_SNAPBACK}][/a]

Allow me to differ, I had understood that he was speaking of MP, and I do indeed deal also with files larger than 100 MP without much problem. :-)

My files are typically much smaller than 5GB though, at most 2.7 GB until now.

My view is that the Adobe Smart Object technology is just not ready to deal with such large image files.

Regards,
Bernard

[bradleygibson]

That was a confusing point:  I too - liker Bernard - mistakenly thought you were talking about a 150 MB file, not a 5 GB one!

Out of curiosity, what sort of images are you working on?  Producing panoramas from scanned large-format film, or digital-back images?  How large a print are you making?  Can none of the layers be merged without 'destruction'?

Kirk
[a href=\"index.php?act=findpost&pid=135794\"][{POST_SNAPBACK}][/a]

Hi, Kirk,

I am working on a 4-image stitch from a P45 digital back.  The resulting panorama is 154megapixels (I'm spelling it out this time...   )  My goal is to go much larger, perhaps to 16 or 25 images for extremely large, exquisitely detailed prints, but to date I am finding the tools aren't quite keeping up.

When complted, I will be printing these images north of 40 inches--Epson's new 64-inch printer announcement is definitely of interest.  Even when the images are not being blown up huge, the images have a crisp, clean look at smaller sizes because there is detail down beyond the eye's ability to discern.  But all that information still has an effect on the 'feel' of the image.  Large format photographs viewed at small sizes have this same feel--put next to small format photographs, one can usually still distinguish them.

I definitely could go with a destructive image processing workflow (and indeed may have to for the forseeable future), but having worked with Photoshop since version 3, I've found that every time I bake my changes into my pixels, I come to regret it later.

For example, I once had one of my images licensed by Heineken and blown up to two stories tall from an oil-mounted drum-scan of Velvia.  I was able to repurpose the image readily because it was non-destructively processed, and was able to deliver optimal quality to my client.

Using this workflow I'm also able to apply a given look and feel to multiple images, even when that look and feel is not the result of a single operation.  I often learn as I go, make discoveries or get a flash of inspiration (or could it be desperation?)-I'm able to try out new ideas and make them work as if I had planned it from the outset.

Like I mentioned earlier, of course there's nothing stopping me from doing this work destructively, but even that way I'm going to hit the limit pretty soon anyway; my work isn't going to get significantly larger without better tools with which to make it.

Hope that helps!

Kind regards,
Brad