"I don't want to talk to you," Castiel says to his counterpart without moving his gaze from where House had been. He finds that he's still seething.
His counterpart is a chore. It's been hard enough keeping straight the problems of his brothers and sisters, of the beck and call of the Winchesters; he doesn't want to have to deal with this as well.
Not to mention he wasn't finished with House, maybe he could understand a little of how Dean felt sometimes in the past.