Going forward for new images.
Create a Smart Album with a colour tag, say Pink.
During Import add a Pink Colour Tag
When editing / adding to required Smart Albums remove the Pink colour tag.
As time goes by Pink Colour tagged images would not be in a " Real " Smart Album but one of their own
For Existing Images
Probably easiest, as already suggested, to add an unique keyword to all images in current Smart Albums then create a Smart Album that does not include that keyword.
Then perhaps create one further Smart Album to combine the two scenarios.
Does not have Pink Colour Tag or Does not have the unique keyword
Probably a more elegant solution but this is what comes to mind, just now.