I always use polarizers. If I find the glass is not worth the editing or its crazy results beyond what is reasonable, I make a shot without and jut paint in the parts I do want "polarized." Especially in interiors with lots of wood, you can get better color and less glare with a polarizer.
I use both b+w and Singh ray.