I can't comment much on the back, but from the Mac side there could be a few issues. One thing I didn't get from your history of things is: was all OK for the first few months and it has gradually gotten worse, or did the back always behave this way? If its gotten worse, it could be the back or a softare matter. However, the fact that things seemed a little better after changing from Capture 8.4.1 to 8.4.3 makes me think it is a computer/software issue and not the back.
To track down problems with the Mac, you might try turning off various features, like Bluetooth (the latest updates seem very unstable on my Mac and I'm using the same computer as you are) and Airport if they're active. Some USB devices when combined with an active Bluetooth causes problems, especially if the USB peripherals deviate from strict USB v2 protocols. The front USB port is a bit wonky on the dual G5, so I try never to connect much to it on a permanent basis. (The iPod won't even reliably transfer to and from the front USB port.)
Until you've resolved things, make sure to only boot up and run Capture with no other applications running and no other firewire devices connected. You might want to try a new Firewire cable, and maybe even another computer, if you have one available.
A 'techie' thing: after installing software, it is a good idea with OS X, to run the Disk Utility and do a 'Repair Disk Permissions'. This is a good habit after every software install or update that you do or is done for you. You also might check out where the Library folder (both the main Library and your user Library) in the "Applications Support" and "Preferences" folders; toss any Capture related items so they get updated with a fresh install. There might even be a Capture folder directly in the Library folder itself. You might have missed a corrupted Pref file.
On the software side, have you upgraded the Mac OS X system? Maybe the older version of Capture won't work as well with the latest OS software. If this is the case, you'd have to roll-back your system software with a re-install to test this out.
From a back perspective, if it is defective, changing it out should help. After you've eliminated the above computer stuff (short of rolling back the OS, which is a royal pain), perhaps your dealer could let you use a rental back for a few hours. Or maybe run your back on one of their systems to see if it misbehaves in their store.