I agree with that part, but not the word. I actually have fanily members who suffer retardation and we don't call them that.
As for those guys mentioned, either I'll give them my nicest smile and a wink, or stare them right back as angrily as I can. Most don't do it very long with the later. But I'm also not looking for trouble, so I weight it carefully depending on general demeanor. We actually have a widespread problem of homelessness and mental health issues in the city, so doing that with an agressive homeless person is a good way to get stabbed.