I'm getting a very close congruence, and I still think my method holds up. If anything, the OOG overlay is somewhat conservative and there seems to be a small threshold before it kicks in.
I'm seeing the conservative nature you point out in a recent test but I'm not certain about close congruence but it may not matter, nothing I know of proves the OOG overlay is 'correct'. Maybe the opposite.
I uploaded my test file which is part of the Gamut Test File. Here's what I did:
1. Duplicate the ProPhoto RGB image, convert to sRGB.
2. View first the Red Channel, use
Color Range with the eyedropper to select a color/tone. I made sure with the info palette I was hovering over pixels at 0/0/0. Note, the one attribute I'm not sure of is Fuzziness. I see two different values in your examples. All blacks in Red Channel gets selected. I then used
Save Selection and targeted the original ProPhoto RGB image to create an Alpha Channel.
3. Repeat step #2 on Green and Blue channel.
4. Back in the original ProPhoto RGB image I used Load Selection three times (adding the selection from Green and Blue Alpha to the Red to produce one selection of all three). I then saved that composite into a single Alpha Channels.
I went back to the sRGB image and reset History so I now have two images in ProPhoto RGB to compare.
Now I can show the selections from each, one is OOG using the old Photoshop command, the other the custom selection. Here's the results of the two where 'Black' is the selection:
On the left is OOG from Photoshop where I've set the overlay to black, 100%. On the right is your technique. Not identical, but an interesting selection.
So assuming I'm doing this correctly and can trust the selection, the next thing I have to test is what to do with the OOG colors selected.
I've uploaded the TIFF with all the Alpha channels if anyone wishes to comment. It's set with the Quick Mask on so at least on this end, after saving and opening again, that is 'sticky' so you'll see an overlay I believe although based on your PS preferences, it may not be in black at full opacity. It's here:
http://digitaldog.net/files/OOG.tif