I understand what you mean.
About two years ago, I was attending a fair. I saw these two young sisters with their faces painted like tigers. It was an awesome paint job. I knelt down about 10 meters away and took their picture. One of the girls squealed happily "He is taking our picture"
Suddenly it was like those old E F Hutton commercials.
The mother snapped around and glared at me. I noticed a few other adults staring at me.
I smiled and pointed to my own face and "their faces are painted". The mother just scowled at me and the impression I got was "OK, I will let this go.. this once, but you better watch it" And she did watch me until I moved away. I guess I am lucky she did not pull a gun on me or pepper spray me.
But then I am an old white man so I am screwed in photography.
If I take a picture of a bridge, I am a terrorist
If I take a picture of an adult, I am a stalker
If I take a picture of a child, I am a pedo.
I am actually none of these. I am just a harmless old man who likes to take pictures that he never shows anyone.
I bet if a woman took a picture of these same two children, no one would have cared.
It is not right, it is not fair, but it is what it is.