Yes, but the joke is on you. It is not said to “people” but to a limited, specific group of individuals for specific reasons, i.e., for what they specifically said. Otherwise, what you posit means that no one can criticize anyone of a specific minority without being accused of racism (or sexism, or... pick your latest lefties’ flavor of the day)
I've already acknowledged that there's an element of this out there, but the other side of this is NOT, "..and therefore every attack on a person of color or a women has NO basis in racism or sexism."
If you say Hillary Clinton is corrupt, it's not necessarily rooted in sexism. If you saw she's a corrupt old nag, and ugly as sin to boot, THEN there's a good case to be made that by concentrating on appearance - a commodity that men have traditionally judged as valuable in women - you're firmly in sexist territory.
Likewise, if you tell Obama to go back to Hawaii, you're probably not being racist. If you tell Obama to go back to Africa, well, you're probably being racist. And if you actually have no idea whether Obama is from Hawaii or Africa, and tell him to go back to the country he came from, assuming it can't be the USA, you might be racist AND stupid, or just stupid. This isn't hard stuff.