That's how it always starts, love. Just a few comments. Those few comments should have never happened at all, nor should he treat people who aren't in his close circle, or purebloods, or Death Eaters, or Slytherins, or nasty to others for the sake of a laugh, the way he does. It's a classic example of manipulation: do wonderful, sweet things that make you ignore the bad moments until it's too late, and the bad moments are all that's left. And the apologies, of course.
Just be careful. I've been there. I know what it's like. I've dealt with his kind before, and the deeper you get, the harder it is to get out.