I recently took a heli-tour in Kauai and solicited similar advice. I shot with a Canon 30D and the same 24-105 f/4 IS lens. I was on a "doors off" helicopter, so reflections were not a problem, but shake from the helicopter and wind certainly were.
I shot the entire tour wide open at f/4 and at ISO 200. Since I shoot mostly landscapes this went against my normal routine. But it worked. I had a high percentage of sharp images. And once I thought more about the DOF issues (or lack there-of), it made sense. There is little, or nothing, in the foreground. Most objects will be at a distance, reducing the need for smaller apertures. In reviewing the images, I noticed that those showing little to no
shake had shutter speeds above 1/750th sec. You can see some of my images
here (the heli-tour images are fairly obvious).
You're probably losing 2 stops with the polarizer and that may be hurting your efforts. You could try wearing dark clothing to minimize reflections instead of using the polarizer.
Hope this helps.