I can't help but think that the people realize something within them isn't right, but they would never suspect Harry.
And I think Harry is, in his own twisted way, still quite innocent. He's found a way to keep people close, friends and enemies, and he uses it so he won't be completely alone.