I don't know if they can or not... but it's a rule, that those of us on the colored teams can't go to the Pillar without one of them. They're in charge of enforcing the Judges' rules outside the games, too, and punishing those who break them.
It's not all bad, though; there's a lot we can do without breaking the rules.