Definitely something that needs to be looked into... There's no way for you to target the spell only for specific people, is there? I know you said it's complicated as it is, I don't want to push you outside of your limits or anything, I'm just not familiar enough with your world's magic to make a judgment like that.