Anyone who doesn't belong to this world will be... "sorted out," he said. The ones taken by the earth will be sent back to their own worlds, and with the ones taken by Judges, we'll be getting a handle on that, too.
Since we're not a part of this world, we won't be subject to his influence the same way the Judges will be.