Also bizarre is the brushstokes in CS4 look like this (click new window):
http://img16.imageshack.us/my.php?image=picture1yof.jpg
What that's showing is that you're "painting" with the pattern stamp tool, not a brush.
The bubble pattern you're seeing is the default texture for the pattern stamp tool.
So either you are hitting the button below the brush in CS4's toolbar, or somehow CS4 is using the pattern stamp instead of the brush.
One is basic user error, the other maybe far more awkward to resolve. It's unlikely to be anything to do with the openGL issue unless that's involved with the pattern stamp display.
I'm no Mac guru, so someone else will have help with the detailed rectification, but a start might be to try uninstalling and reinstalling CS4 and adding the 11.0.1 update, that might resolve the issue.
I'm using CS4 11.0.1 on Windows XP here and have absolutely no problems with painting on masks, with or without using the pattern stamp, even on a small lowly specified netbook.