Good point.
You're right. You're absolutely right.
That, right there, is one of my biggest problems with the X-Men; why do people constantly harp on the mutants, but every other super powered being is completely unmolested? What is it about mutants that they are singled out above all others? You'd think that groups like Friends of Humanity or The Purifiers would be advocating against all superhumans, but no; sure, the Hulk goes on destructive rampages all the time, but he's not a mutant, so it's a-okay!
Now granted, Grant Morrison actually did a good job of explaining this weird paradox; more and more mutants were born every year, gradually developing their own culture and spreading throughout the world. Humans didn't hate and fear Mutants because they have powers, they hated and feared mutants because they felt the mutant race was replacing the human race. Of course, then Decimation happened and that totally logical reason was blown out of the water.