Hi Bart (have I got your first name right?),
Actually this is very interesting because I use exactly the same method (which I thought I had 'discovered' myself!). But at the moment my head isn't able to figure out if this is a good method or not, although I have to say it seems to give pretty good results. The other technique I've tried out is to use the HSL plug-in to make a saturation mask and then to use this to tweak the saturation selectively, then to really dial-it down by reducing the opacity until the image looks good (after all, as Rodney says, it's down to looks at the end of the day, and all we're attempting is to make the square peg fit into the round hole a bit better).
As a kind of weird twist on the technique you use, instead of adjusting the opacity, what I did was to create a 'Perceptual' layer with the Relative mask (as per your technique). To create the 'Perceptual' layer what I do is to copy the image, render it to the output space using Perceptual, convert it back to the working space, and use this. As the perceptual rendering will bring the colors back into gamut, the effect is to get rid of the OOG pixels. HOWEVER ... I think this may be doing even more messing with things I don't quite understand and which at the end of the day is probably no better than letting the CMM make the adjustments.
Even though this seems to be the wrong thing to do (and both Rodney and Jeff are pretty clear that it is), the best result I can get with a problematic image (and I haven't tried too many, to be fair) is to convert the image right after import into Photoshop to the output profile and to then edit it to bring it back as close as I can to the original (or to what I want, at any rate).
Interestingly, this certainly doesn't seem to be anything new (is anything?). For example, in 'Real World Color Management' there is this comment: "
• Early-binding workflow converts all the color into final output space as soon as possible—traditional prepress workflows that took output CMYK right from the scanner are a classic example.
• Late-binding workflow delays the conversion to final output as long as possible. In-RIP separations in the platesetter’s or imagesetter’s RIP represent an extreme late-binding workflow.and then they go on to explain the pros and cons of the two options. Converting to the output space early on is what they call 'Early Binding', I think - and it would seem that there are advantages to doing it this way; with the major disadvantage being that it burns the image to the printer/paper.
Still, what is really going on when you convert to the output profile? Clearly there's an effect on the gamut, but also the white point is shifted and so are all the colors. Then I'm viewing this on my monitor, so what's happening there? There's potentially another gamut clipping, white point shift (back, presumably?), color shifts etc. I just don't know enough about color management to be able to figure that out ... which is why I posted this question in the first place

.
I've just had a look at Rodney's video, and he makes a very good case for letting the system take care of things. And I do agree with him. However, sometimes the system just doesn't do it properly, especially if there are fairly large areas well out of gamut - because then we get a flattened space with none of the life that should be in that part of the image. Then we do need to do something - and it could be that your method or the HSL plug-in mask, or early-binding ... or something else! ... could help to correct the problem. I think that this is one place that means exit from Lightroom to Photoshop, because Lightroom just doesn't have the tools for this kind of very selective editing (IMO, at any rate).
Robert