You didn't mention what kind of light you are working in.
If you're on the long end, you have effective f/8 aperture and you're focusing manually with a camera not known for a bright viewfinder. Depending on the light and your vision it's going to be a challenge to focus perfectly when you've got a 560mm lens.
I am not knocking you here. I have a mild prescription, but my eyes are corrected to 20/15 and I'm young so I don't have any issues with focusing or with night vision. I have a 5D which has a brighter viewfinder. And shooting with a long lens in certain types of light I will absolutely have to work hard to focus manually and get the correct focus. Even if I use one of my f/1.8 lenses, if I'm in really low light the camera will actually AF better then I can MF.
If you could switch your focusing screen I might suggest trying that, but I don't think you can in the 20D, right?
I think our stock focusing screens really stink for MF and switching might help a lot, the dual prism setup on older cameras is definitely better for this kind of thing.
Since you didn't post your pictures showing a problem I would suggest taking a picture that has a range of objects at evenly spaced distances visible in it, that way you can see if the focus is just off as opposed to a problem with the lens. With long telephoto shots you can easily just get "everything" out of focus enough to make it just look bad.