It would be easy, perhaps preferrable, to tell James he was wrong. It would probably bring out an angry reaction in the man, and no doubt Fisher would no doubt walk away with bruises, if he was able to walk at all. What would hurt James most would be that Fisher would obviously be lying, and since the other man was admitting to an awful lot, to get the door slammed in his face like that... Though that would be a surefire way to end this mess.
"You're not wrong," he admitted weakly, closing his eyes and lifting his other arm around James' shoulder. He let the underwear clasped in his hand fall to the floor so his fingers could tangle themselves in those soft, short blonde locks. "I'm trying not to, but... I think about you all the time. Your lips and your hands and that goddamn smirk on your face. I spent all weekend in solitary thinking about you."