Yes, that is indeed the main problem. The distinction seems to be whether all channels clip, or just one. Or perhaps two, but at any rate not all three.
But how to separate them?
Good question. I'm not sure we can. I was hoping we could select 0/0/0 from
each color channel and add those selections to a new, single alpha channel, then subtract anything that's
neutral, avoiding color and thus OOG. But
Color Range doesn't have this capability. Two other issues with the technique as I understand it:
1. The
Color Range selection is based on the Eyedropper sampling. So obviously setting point vs. 3x3 vs. other settings affect what Color Range selects.
2.
Fuzziness also affects the selection. No feather would be more precise in what gets selected. But the final selection channel if loaded and some edit applied (you used Channel Mixer) can produce some ugly selections of edges.
So I'm not sure this technique can work, again based on my understanding of the steps. We want only OOG colors,
not clipped
colors or rather clipped tones. We want to select OOG only so we can edit just those pixels but that's looking difficult to do with the tools provided.
So this takes me back to one possible solutions. Just soft proof, let the profile handle OOG and if an area looks poor, like the Magenta fabric that has no detail, deal with that selectively prior to conversion
while soft proofing. Forget the OOG overlay completely. Use the sponge tool set to desaturate with a low level of strength (flow) and along with the Fade command if necessary to fix this problem pretty quickly. The technique can be done on a layer so it's "
non destructive". Instead of using Sponge tool:
1.Create a new layer above the image, set blend mode to Saturation.
2.Set color to Black (or any neutral color value).
3.Select a brush with low opacity and paint onto OOG to desaturate. The Fade command is useful to fine tune but operates after each brush stroke.
If you go too far, set color to fully saturated color (R0/G255/B0) Paint saturation back in.
The other alternative is if the soft proof looks OK, just pick the rendering intent, and move on.