Using Sony bodies, I never bother turning off the IBIS even though most of my work is tripod mounted and I do long exposures fairly often. It makes no difference to me and I get critically sharp images. Wind is an entirely different problem and I've seen some good suggestions above on combatting that.
My engineering take on the reason is that, if there is no input into the IBIS system, then there should be no reason for an output to move the sensor. if there is, then the system may damp the response but then that same gust of wind that caused the input would have displaced a non-stabilised sensor from zero as well, with the net result that it makes no difference.