[Perhaps he should have rather said you know I wouldn't have done it if I'd mind, but this? This is more honest, and he's having difficulties telling her nothing but the absolute truth, like this.
He smiles wryly, at least having the presence of mind to not reply with never too busy for you. Casual flirting with everyone else is alright, but with her, it just won't do.]