While a speedy graphics card is certainly a bonus for running Aperture, I have found that having gobs of RAM available combined with the fastest possible disk access you can lay your hands on will go much further in terms of providing a pleasant experience.
For example, at work, I run Aperture on a Quad 3GHz Mac Pro with 4GB of RAM and a 256MB nVidia 7300GT. All images are stored on the internal, stock hard drive. Performance is just fine on this machine (and I should bloody hope so, as well! )
At home, however, I am running it on a Dual 2.5GHz G5 with 8GB of RAM, but only a 128MB ATI 9600(XT?) for graphics. That machine, however, has two 10,000 RPM SATA drives in it running in RAID 0, and performance in Aperture is noticeably snappier. This, in spite of the fact that the Mac Pro utterly demolishes my home machine in virtually any benchmark I have run on it.
I haven't done any formal testing of Aperture, mind you.. but based on daily usage in both locations, I certainly prefer working at home when I can. I'd imagine the iMac will be acceptable, and if you can get faster I/O going somehow, I'd imagine it would be very usable indeed.