Of course, which is why I'm not saying you shouldn't be. But others are going to try and defend Loki because he is their friend, their comrade, their son.
Think of it this way. Say you did something horrible at home but you've been here two years, you're not at that point in your timeline, you've been working with everyone, helping everyone and your time here has put you on a different path. People from your world show up from a time point after you've done something horrible. They have every right to distrust you because of what you might one day do, but everyone else who's here would have ever right to defend you because they'd experienced what you're like here for the last two years.
It's not black and white. Especially not if you're from Loki's world. Because then your world is like mine where there are about 2000000 different versions of it. And one instance can alter the course of a life.