I have reservations about you because you were frightfully rude to me, and insulted my family, and I don't even know your name. It doesn't exactly inspire my trust.
This Tom Riddle obviously isn't pureblood. Now, whatever your opinions on our way of doing things, no wealthy pureblood would want to give someone of a lesser bloodline absolute power over the wizarding world. They wouldn't follow him. It's absolutely illogical.
They're trying to blame us for Grindelwald at home, too, as if we want some foreign wizard trying to annex our country! Some people find it very convenient to blame purebloods for all their troubles. But even if every word you say is true - I am trying to prevent such a situation from ever arising. That is all.