Deep down, he might admit that he didn't want it to be purely physical. What Oliver wanted, more than anything, was to be cared for - as much as he denied it. The last seven months had been terribly lonely for him and though he pushed everyone away, he really wanted to gather them all close to him. Hell, the only reason he really pushed them away was for their own protection. He couldn't protect them in a fight so he pushed them away from the danger.
He was too scared of losing those around him to endanger them by being too near them. Everyone he associated with could be found guilty of harboring a traitor and arrested and he couldn't stand that. That's why he wouldn't have a relationship with Katie, or Lisa (more than the one he was having.) That's why he'd leave his Gram's and put Emilie up at George's. He wouldn't hurt those around him anymore, not until they figured this thing out, because surely McGonagall knew what she was doing.
The thought of having to end his relationship with Lisa at only the physical level didn't sit right with Oliver and he pulled back from her embrace a bit, resting his forehead against hers. He should end it with her too, because he knew the feelings rising up in his gut had not emanated from a lower source, but from higher up. He liked the girl, fancied her really. He liked her sweet laugh and shy eyes and how she seemed to adore the pretty picture of James that he painted. Oliver sighed and kissed her cheek before setting her aright on the bench, helping her smooth out her skirts. He clasped his hands before him, elbows resting on his knees as he looked away.