Others did get pulled in. Plenty of 'em, though I myself try to mostly keep my distance. I'm no goddess, nor even much of a wizard. It's just better for me not to poke at them, especially since my dad's pissed off his fair share of ones that are bad to piss off. I don't like playing allegiance games with gods.
Anyway. If you look in your pockets and anything else you have with you, you might find something that'll tell you who you replaced and where they lived. That place is yours now, so is everything in it, and any money that person had. Which is not a bad deal for getting dumped into a strange world, unless you replaced some hobos.