I'm now 71 with a busload of grandkids between 3 and 13. I follow them around with my FF Canon and the huge 70-200 lens. In the last year alone I've been attacked 5 times by people demanding to know what I'm doing. At a public beach, a park wading pool, a gymnastic class, etc.
I've developed a good response. I say, "I'm a grandfather photographing his grandkids. That's a normal and common behavior. To assume I'm a pervert based on my age and the size of my camera is not normal. It's a pervision. So, who is the pervert here?"
Note: In every case I look around and spot several young mothers photographing their kids with cell phones. They are never attacked.