The Doctor went absolutely rigid . He had to. Because he would not hit Jack. He would not. First because he really did like Jack, loved one version, and maybe could love this one. Secondly, because that would only prove Jack's point.
But out came in an absolutely ferocious cold voice, "You could do that. If you could live with yourself afterwards. Look any version of me in face afterwards. Go right ahead. You fail to understand. I have had the power in that relationship. Always, in all the ways that matter, to him. To reject him. To disapprove. To praise, or love, or need. Begging won't change him. Forcing it down his throat won't change him. Only giving. Giving him enough of the real power I had over him and forcing him to see that what he was doing had genuine consequences to me. That I wasn't unbreakable or invincible. That's what made him take responsibility for what he does."