Yes, and then that Magister wakes up without his slaves and guesses that someone took what he considers his. He'll get angry, and vow vengeance and possibly take it out on other people. What you then create is someone with a grudge, someone who's learned nothing of their mistakes and will probably go through every length they can to take back what they consider theirs. That gets more people hurt. It's why I talked to Leliana so much when she wanted to do similar things. It's not always the solution.
And sometimes the price of action far outweighs the price of words. If I can save more lives by talking and using political means then by throwing lives into the maelstrom of war then I will do it. Do you remember the Temple of Mythal and the Well? I could have gone straight after Samson without thinking twice. I chose to take the longer route, knowing it might cost me precious time and the lives of others. In the end I gained respect from it's guardians, and a temporary alliance that I would not have had had I just gone to actions.
I think the people of the South see and hear 'Tevinter' and automatically assume. I think it would take time, and a lot of effort from the former Tevinter person, to gain any sort of trust from others. Especially after this recent war.