Either you can't control it and you're hearing every thought anyone around you is having or you can control it and you're deliberately poking around in someone's thoughts. Take a poll, see how many people would willingly let you do that. It doesn't matter how righteous and upstanding a person is, everybody has thoughts they wouldn't want people to know. No matter how you slice it, whether it's intentional or not, telepathy is invasive.