Most of the people I've spoken to were invited here, at least.
I think it's further proof that Aunt Phoebe exaggerates things. She also says that eating cake is a bad idea and that it's the height of rudeness for children to speak unless they've been asked a question. She isn't exactly my favourite aunt.
Perhaps it's only the dirty muggles who don't wash that have diseases, not all of them like she said.