That's my guess entirely. Most of the men I've known in my life use women's bodies to escape something - if they don't love the woman.
It's not so much switching as switching the way they appear. Azrael is the Angel of Death, so it helps if sie can look different ways to different people to coax them along. Sie always appeared to me as a man with dark hair and pretty eyes, so the first time I saw hir in a woman's body, I was confused.