Harry had come into the library - not to lurk or spy on anyone, but to actually look up some old issues of the Prophet because he was curious about something - but seeing Mandy and Malfoy sitting together looking so chatty, he lost all impetus for what he was going to do.
Hearing comments about how he had apparently 'caged' Mandy, he moved over to their table, intent on defending himself and her ... but when he was looking down at them, all he felt was revulsion for Malfoy and the realisation that he probably wasn't good enough for Mandy, so she'd sought company elsewhere.
"Have fun studying," was all he managed before walking away again, out the door and down the corridor to the nearest exit outside.