"Tell him go away rather than listening. He was worried about me and, yeah, ripped into me when I was stubborn about it. I told him get out. And I think he inferred a 'don't come back'," the Doctor said slowly.
The Doctor may have implied the 'back come back'. He wasn't sure. He hadn't been thinking clearly.