It's not that they could hurt him physically, it's what they represent and what Kyle thinks could happen to him psychologically. Kyle's brought up a couple times that he's worried about what would happen if he went crazy (like Hal, sorta) and that's why he voted for Batman in Tower of Babel, etc.