I think I was around sixteen when I first had to deal with it. Let me tell you, there's no worse feeling that not even being able to attend sermons held by your own father because of the looks and whispers. I spent much of my time feeling like I'd shamed him somehow. I'd commute to another church every so often where they didn't know me.
At some point, you just stop caring for the most part.