Sometimes there's a siren call for a game and you can wander over to the arena willingly to participate. Other times, you can just... be summoned there by a judge. Randomly, against your will, in the middle of whatever you were doing.
The only exception is if you're inside your dorm, or so I'm told. That's the only place the judges' power can't penetrate. I was given pretty firm instructions to never open our dorm door unless I know who's outside it first.