Harry decided that he'd dug himself into a hole on this one, and decided to let the matter drop. The truth was, the woman didn't actually seem scathed at all by Goyle's terse response, and was happily nattering on at her son and the chosen pygmy puffs. But he still would stand by his assertion that Goyle could be nicer to the customers, and would eventually chase a good number off simply by being, well, him.
"No, I just came in to check on things. Maybe say hello to George." He made it clear that by things he'd meant Goyle. Even someone of Goyle's questionable intelligence could cypher that out. "I drop in here from time to time. Whenever I have a moment, actually." It would be good that Goyle keep that in mind.