I have had the 20D, 30D and now the 40D and I think the upgrade from a 20D is definitly worth it. Except from image quality, noise performance, AF, live view, viewfinder, sensor cleaning and hanling it is also much faster. Nog only when taking images in burst mode, but just the whole camera. The menu is faster and with the 20 en 30D, formatting a large CF card could take some time, in the 40D this is done almost instantly.
If wildlife photography is your thing I would consider the 40D instead of the 5D. It is much faster and the cropfactor comes in handy here, since you can use the extra mm's. If you are a landscape photographer I would consider the 5D which is still better in noise performance en IQ.
I am thinking of purchasing a 5D as extra body, then I have a 40D for wildlife and a 5D for landscape.