The Doctor would have minded a great deal more if his kidnapping hadn't indirectly saved his life. As matters stood, he was torn. He hated the idea of being trapped here, particularly with evidence of his future there to be tripped over, but he also flinched away from the prospect of returning home and potentially becoming the man with the bow tie he knew was somewhere in New York, the man Amy called her Doctor.
"I have friends here," the Doctor admitted, "friends who shouldn't be here. Friends I haven't met yet, or who haven't met me. I don't know yet if I'd rather they were home or not."