I've been using the 20D/17-85 kit for a couple of years now, and have also been tempted by the 24-105L. A couple months ago I rented a 24-105 for a weekend of comparison shooting between the two lenses.
As for the 17-85, I have also noticed the CA that Jeremy reported, but have no trouble correcting it in LR. The bigger problem, IMO, is the oddly asymmetrical barrel distortion at the wide end, which is not as easily corrected.
The 24-105 is a bit bigger and heavier than the 17-85, but after a while I didn't really notice the extra weight, and in fact I thought the 20D/24-105 combination was a little better balanced than the smaller lens. CA was somewhat better with the 24-105; barrel distortion was less of a problem, and smooth enough to be easily corrected in PS.
Image quality between the two lenses was much closer than I expected, and with most subjects there was not a huge difference in IQ in a 10x15 inch print. Within its limits, I've never thought that the 17-85 is as bad as many reviewers claim, and in this particular shooting situation, the IQ alone did not completely sell me the L glass. That said, this was by no means a scientifically rigorous test - I was shooting handheld at reasonably fast shutter speeds, ISO 100-400, and aperture ranges were generally f/8-11. Shooting detailed landscapes on a tripod, for instance, may reveal differences that I did not see in my days of street shooting.
As for general performance of the lens other than IQ, I thought the 24-105 was a great lens. The wider max aperture makes a noticeable difference in the brightness of the 20D viewfinder, autofocus seemed a bit faster than the 17-85, and the extra telephoto reach is nice to have in a walkaround lens. [I have a 10-22 to take care of the wide angle, so I didn't miss the 17-23mm range.]
I still get tempted by the 24-105; if I find a good deal on one and am feeling flush I may give in to temptation. If you have access to a shop that rents equipment, I would definitely recommend that you rent one for a day and run your own comparison.
Eric