Sounds like a hurdle, not letting it detract. I mean, worrying detracting from building a life is basically my MO, though, so I might be biased on the side of crippling anxiety.
A sorcerer, huh? That's impressive. I'm picturing robes and a beard, but maybe I've seen too many movies. Count yourself lucky there's no clowns.