This backstory for Dolohov makes sense to me. I always felt that Voldemort initially gained his followers by preying on people who were vulnerable and rejected by society. If everyone else seems to hate you for who you are, and then you get someone like Voldemort who is powerful and charismatic offering you acceptance and a feeling of pride in your identity, well, it's not surprising people were willing to join him.