Considering that him watching the kids would just be giving them six or seven bottles of cough syrup and putting them to bed? He wasn't quite sure how that was going to make him miserable, but ok. If anything he'd just be bored.
"But, you know you love being here when I'm miserable. So you wouldn't be able to stay away anyway."