There is more than one person in the city that would disagree with you. And even more beyond that.
My family needs to not be pulled into "rescuing" me all the time. Being with Alfred is the safest I can be, and if everyone would calm down, they would realize that.