Yeah, it messes with my vestibular system. The delay when panning trips it off.
Of course, I like the minimum of interference with the picture-taking process. As Slobodan says, too much interaction disrupts one's connection with the subject, the primary focus. I don't even use AF most of the time for that reason, and for the reason that the optimal composition for a given subject /never in life/ coincides exactly with the placement of an autofocus point.