I tend to find if I want a polarizer, I want it on max setting anyway, and any slight turning of the ND doesn't make that much difference to the shot. I know some adjust the polarization a lot, and occasionally I need to do same. For the odd times that this is the case though, such as messing with reflections, I'm generally not also using a Grad ND.
I thought the same when I got it that it would be a problem, and to that end I have a second holder and the joiner so I can rotate one without the other. (I also have the 105mm polarizer).
Thing is, in practice. It never got used, so I dropped that config.
Personally I prefer being able to quickly add/remove the polarizer, vs having to screw/unscrew it.
The main two filters that I use a lot are a single grade and the Bigstopper, plus sometimes a VariND as well.
Regards
Mark