I like the parallels drawn between these two characters, neither of them a hero or a villain, motivated more by self-preservation than anything else. I think sometimes the reason characters like this aren't popular in fandom is that we fear we might have more in common with them than we want to. We want to identify with the heroes, but we wonder if push came to shove, whether our survival instincts might actually be stronger than our desire to do the right thing, as it is with Peter and Ollivander.
I also think your analysis of Voldemort's control over his followers is very perceptive; I've always thought his power lay as much in psychology as in magic. If you put enough fear into people, they'll do what you want even when you're not there to give them orders. And again, we'd like to think *we* wouldn't fall under the sway of someone like that -- but people do, and that prospect is threatening.