When Brian went to see his father to tell him that he's gay, Jack raises his hand against Brian, and Brian flinches in a way that always made me think that it was a move he had to make quite often as a kid. Or maybe it was just an instinctive reaction, who knows.
Brian was drunk, hurt, in an emotional turmoil after seeing Justin AND Michael throwing the things he said in his face. I don't want to excuse him for hitting Michael. It was just too much to bear at that moment. And I agree, Brian seemed as surprised as everyone that he actually hit his best friend. Partly maybe because of the physical abuse (that we can only assume but is more than plausible) he lived through as a child.