That was a very astute observation. He felt conflicted even after Stane tried to kill him. He had a sit down chat with Vanko after the Monaco Grand Prix. He even offered Loki a drink, and then did some major takes-backsies after flying a nuke into a space sphinter. Tony was an odd adversary to have, and after all the weapons with his family name on them, that he designed to kill people in wars? There was a significant amount of survivor's guilt that he waded through on a daily basis.
He eyed the device and didn't reach out for it. Instead, he kept clutching onto his cup of coffee like it was a life preserver.
"Oh yeah," Tony said with a vaguely apologetic wince. "I don't do that whole things being handed to me thing. Could you just...slide it over and let go?"
And apparently there was quite a significant amount of odd obsessive-compulsive quirks involved, too.