I don't have this "problem" with my 5D or any of my current lenses. (50/1.4, 28/1.8, 85/1.8, 17-40 f/4, 300 f/4 IS)
However I have had it on another Canon body and/or lens and it never caused any issues with wear & tear or image quality...
I wouldn't worry about it.