I'd suggest basing it on how much one's avoidance interferences with functioning and the ability to maintain a reasonably normal existence. Moving away from one's parents is common and often necessary--it's an uncomfortable part of growing up. But the pangs of conscience brought upon by sins of omission - avoiding friends, avoiding jobs, avoiding responsibility, avoiding dealing with things - are damaging, I think. Much better to make a mistake and learn from it than to continue to avoid anomalies. Oftentimes what we learn from a mistake are the things we really need to know.
I don't particularly want to drive, but I'd like my license just because not having one is so damn limiting--better to have the option. My psychiatrist mentioned that, in his experience, not having a license is usually a good indication of some sort of pathology, like an anxiety or depressive disorder. I think he's probably right about that.
I feel that way, too. But one thing I noticed when I started to take risks is that I can, in fact, survive them, and each time I do, I become much stronger as a result. And it gets easier.