I'd go so far as to say it's not a minority- I think there's about a 50/50 split - among people who've never heard the Japanese voice, anyway. (heh.)
I was just replaying a couple of levels of Command Mission and, wow, there's much more of a contrast between his voice there and his voice in MHX, later on. But in general, yeah. I consider him definitive, and like you, it's just a gut thing. It feels like Zero to me when I hear him.