Oliver stepped back a bit to give Percy some space if he wanted it, but he kept a hand on the other's shoulder so he wouldn't think Oli was freaking out. He was freaking out a little, since the situation was so weird and unexpected, but he wasn't about to run off or anything.
"You told me that, sure. But being attracted to men doesn't mean you're attracted to me." Oliver had never even entertained the possibility that Percy could be interested in him. He'd never considered that they could be more than friends. And now Percy was saying that...