You're in a prison, kid, with people you don't know. Who knows what they consider worthy of punching someone for? In my world, we solve things with duels.
And if it had been you, I would say that maybe they got the wrong guy or just misunderstood something. But your brother likes to antagonize people. He used to do it a lot before you showed up.