"I'd expected you to come claim what's rightfully yours, but that never happened, did it? You're boring and you disappoint me." Still, after saying all that, he pulled back a step once Thor pushed forward. He didn't plan on doing anything past giving him Mjolnir and walking away, but he couldn't be so sure about Thor.
"To be honest, here, I don't think you deserve it. Maybe you did once, but not now. I'd keep it, but I'm bored with it and I frankly have better things to spend my time on."