Getting them out is a pain, keeping them out is a pain, and then whoever it is just becomes someone else's problem. The next thing I know, it's "your exiles have been forced into Orlais and they're ruining our ridiculous hats and stinky cheeses by existing, Your Majesty" and I have to convince the Council of Heralds that I am not, in fact, trying to insult them by making people I don't like live in Orlais.
Though if we're being honest, that's exactly the kind of torture I'd wish on my enemies.