He was unsure what sort of comfort the Doctor's words were supposed to offer, as it seemed he had no life to return to, and he couldn't imagine what unfortunate thing he might have learned that would reassure John.
And so he made no attempt to respond to it. Surely he'd asked all variations of 'what' enough times in this conversation that its effect had all but worn through now. "I remember...this can go anywhere. Even to Farringham? Why--why can't you take me back?" Surely he wouldn't just...just keep John when he asked to be returned if he could return John, would he?