Harry knew because he was better at skulking around than Draco - largely since he had a cloak for it - and had spent last year following him around every free chance he had, since he'd KNOWN Draco was up to something.
Harry hated Draco. Draco gave him ample opportunity and reason. But even he'd felt a little bad for the ferret, after he'd caught him crying. It'd been. . . well pathetic. But still.
It didn't mean he liked Malfoy, trusted him, or gave a damn for his feelings. What little he actually had. "Yeah, right," he answered flatly. "Just get lost then." Which he had no right to demand, since Harry wasn't supposed to be here any more than Malfoy was. But he didn't care.