He's annoyed with Jack for speaking up before he can. That's his job, and besides, he would have welcomed the opportunity to get away for a bit, just do normal and routine and comforting things. Not have to think about Tosh, look at her, and wonder what's going to go wrong this time.
He's nervous and fidgety as he watches Jack walk away, uncomfortable with being left like this.