Our wizarding community is small, I mean not as small as people make it out to be but enough that it was easy for him to take over the Ministry. Granted, most of the government employees are incompetent, and if they had just listened when the warnings were there, we might not have gotten into this mess.