"I'm not most people," Harry pointed out. He looked over at the woman making a choice between the pygmy puffs. "Maybe not, but she's also on the opposite side of the shop from you, after you snapped at her. There's probably a correlation there." Though it was also possible he was over-reacting, not that he'd admit that to Goyle.
Harry could feel the tips of his ears burn, and willed away the blush. He wasn't going to apologize for jumping to conclusions, and hopefully Goyle was too dim to notice Harry's momentary embarrassment for the accusation. "Well, I hope you keep in mind that you could easily be in need of a job again," Harry said, though he thought he'd lost most of his authority by this point.