Castiel was first and foremost infuriated, but he didn't know how to put it into words. To say it hadn't been enjoyable would have been a lie. He wasn't good at telling those, or so he tried to convince himself.
He still didn't want to accept the reality of what he'd become in that respect. If he accepted that, then he would have to take what Crowley said about him into account and that was wrong on so many levels.
So instead, he just flew off. He'd gotten what he'd come there for. He knew that if he said anything, it would be something he would only come to regret.