He felt the muscle beneath his hand twitch and stir and eventually settle. He held there because the Doctor needed to make sure Remy knew he was still there; even if his words and very demeanor perhaps indicated to the contrary. He was here, Remy LeBeau, your friend the Doctor. He was here and he was listening. Even as you shrugged out of his grasp.
The Doctor's hand returned to his pocket, back into that defensive posture. The Doctor blinked. Once. Twice. Three times. He was thinking about those two simple words. "I'm sorry." The Doctor didn't want to ask how, so he didn't. "I so very sorry." He wanted to reach out again, offer a hug. That's what he did in times like this, he hugged. Best gesture in the universe, a hug.