Mutants are scary because they represent a threat to humanity - at one point, they had the numbers and powers to represent a credible threat to the survival of homo sapiens, should they so choose, and more of them were coming into power every year.
People are scared of mutants because when Magneto finally gets his shit together, or Xavier finally snaps, and knocks out half of humanity, the friendly kid with the green skin who lives next door is going to be fine, as long he turns on you.