As Fred mentioned, heat waves could be causing blurriness at that distance. It happens to me alot in Texas.
Just as a rule, I never trust setting a lens to infinity. I know it should work, but all too often one lens needs to be slightly ahead, another slightly behind.
The next time you're taking test shots, focus bracket and snap a couple shots slightly ahead of infinity and several shots slightly behind.
If it its windy, that can add some vibration to the tripod, so you might want to hedge your bets and aim for a 1/640th shutter speed (for a 200mm focal length).
Lastly, when testing out the system, it really helps to eliminate the variables. I suggest using a tripod, mirror lock-up and remote release or the timer.