"I'm just not in the mood for your pettiness," he pointed out. He didn't really want the Master to follow him, but what was he going to do? That would just lead to more arguing.
And really, the Doctor didn't know what to say about it. "I'm not going to sit here and listen to you berate me. That is one serious mood killer."