The worst thing a polarizer can do to you is act like a 1 stop neutral density filter, unless you are trying to photograph reflections. I have a polarizer stuck on my 35-350mm f3.5-5.6L (it's a long, sad story, please don't ask) and the only time it bothers me is when working in low light and I would like the extra stop of speed.