If I understand your situation correctly, where you paint the mask with black (or gray) to diminish the effect of the customized BW adjustment, then what is happening is you’re also undoing the conversion from color to BW.
You can just copy the adjustment layer, invert the mask, and what had turned blue will revert to BW. You can then work with the sliders on either adjustment layer to get the effect you want.
When I do this kind of two-layer work, I add a third BW adjustment (usually keeping the default slider values) to make sure that any parts of the image that haven’t been completely converted to "fully neutral" BW do get so converted. This can happen, for example, where the brush has some softness to it.