"Not even with the Master?" Gray stared back. "I mean, yeah, he talks more, and he argued a lot with the Master, but he doesn't always back down or argue if action works better. And I've never seen him have a problem with it."
Or he could just be projecting his own lack of concern over physical violence on the Doctor's lack of reaction afterward.