"Not really my thing." He's too serious for this sort of thing, in his suits and professional manners. He's good at sarcasm, but he's never been particularly good at fun. And this is the sort of situation that certainly qualifies as fun. So, obviously, he's avoiding it.
"Anyway I don't have a swimsuit so you're out of luck I'm afraid." Why does he think that excuse will actually work?