Just as a little exercise I wanted to do a test to see if another application and PS would agree about what was in and what was out of gamut so, Color Think Pro seemed the obvious choice since it can visually indicate how far out of gamut a color is. My testing methodology may be flawed (and I'm sure someone will be quick to point out if it is) but here's what I did...
I took an untagged target, selected the profile I built with that target, and turned on the gamut warning. Using the wand I selected out of gamut colors, created a new layer, and filled the selection with black. When this layer is on it will only show colors that are theoretically in gamut. I then inverted that selection and created a layer where only out of gamut colors would be displayed. I then saved two TIFFs, one image that only displayed in gamut colors, and the other with out of gamut colors. From there I brought the images into ColorThink Pro and using a custom grid, placed a sample spot over each patch on the chart. I brought each image into a three dimensional graph and then called up a 3D rendering of the profile's gamut. In theory if I view only the in gamut colors, I would not see any points for the in gamut image falling outside of the profile's gamut and when turning on the out of gamut plot, I shouldn't see anything inside.
Well... that's not exactly what happened but it was close. When viewing the "in gamut" colors, there were a few stragglers that appeared outside of the profile but they stayed pretty close. For the out of gamut colors, they were all outside of the gamut as expected. For kicks I pulled up the US Web Coated (SWOP) v2 profile and had a look at it. According to Color Think Pro at least, the out of gamut warning is behaving more closely to what Jeff is describing, and not even remotely close to what Scott is describing. I'm not sure what accounts for the difference between what ColorThink Pro indicates as being out of gamut and what Photoshop indicates as being out of gamut. Could be differences in the CMM (Pat???).
Indeed if you setup your custom profile and select US Web Coated (SWOP) v2, that's exactly what happens, but it's based upon the profile selected for soft proofing, and it's not hard wired to CMYK as seems to be suggested. You can clearly see this for yourself if you enable the out of gamut warning, call up the custom soft proofing options, and then switch to different profiles. You'll see the out of gamut warning change with each profile you load.
That said, here's a specific detail I'd like more info on:
"The new Gamut Warning checkbox (beneath the preview area)"
Your initial question mentions the "Out of gamut warning". The out of gamut warning is not a check box, it's a menu item (also activated by command or control shift Y). What is the interface you're speaking of that has it as a check box? Perhaps that is a different scenario. The out of gamut warning is not a "new" feature either, it's been around for quite a while.
Cheers, Joe