They killed my mother, and made a mockery of my esteemed master. Not to mention they're responsible for the beast plague, the perversion of the ancient holy blood that was going to be the cure of Yharnam. They murder good hunters who defend those in need, particularly, and deliver their blood to their immortal queen.
From what I've read, most vampires here have to feed solely - or near enough solely - on human blood to survive. Vilebloods do not. The blood grants them immortality, yes, but they could dine on whatever they want. They choose blood because it renders them immortal, while the rest of us die on the streets of either the plague or Vileblood machinations.
They live off of corruption and death. I mean that literally - because we die, they do not. That is why I fight against them - for those they've wronged, and for my lost master.