He scoffed, but there was a smile in there somewhere, mostly hidden by the coffee mug again, "Been there, done that, rather not do it again unless I have to." He shrugged then, little more than a comfortable roll of his shoulders, really, "And I'll be fine, survived this long without a live-in assistant, figure I can keep on going all the same."
Far from being a jibe, it was an amused statement, he did like having her around, mostly because if he started talking to himself she'd pick up on it and it would be dialog instead of just a half-crazy old man talking to himself. Sure, Thatcher was good for that too, sort of, but it wasn't the same.