I'm catching up on my flist in the most chaotic way possible, just kind of jumping all over the place!
I agree with those who have said we don't have DEFINITE "proof" one way or the other, but that we're given a number of cues that Brian was physically abused as well as emotionally/verbally.
The fact that Brian doesn't call Joan on her saying she took the blows for her kids makes perfect sense to me -- that whole episode is about the things Brian DOES NOT SAY to his mother to shove her face in reality.
The fact that flinches when Jack raises a hand to him, the fact that Brian pulls his punch and hits the box or wall (?) instead of his father, what someone else called his being "oddly non-violent" -- despite the bad!fic fandon of revenge-seeking Brian, after he disables Chris Hobbes, he never again lifts a finger against him -- and Hobbes literally nearly killed Justin in front of Brian. He goes into paralysis when Craig is beating him, too. And that statement about his father showing how he felt when he'd had a few.
None of those things by themselves proves anything, but taken as a whole, they're pretty compelling evidence to me that we're supposed to believe Brian was smacked around as a kid.
Some people distinguish between "physical abuse" and pushes, a single blow, a smack across the face, but I don't in any case, but particularly here. When they're part of a pattern, combined with verbal abuse, and coupled with the fact that we know Jack beat Joan, those things are absolutely physical abuse, full stop. Maybe Brian wasn't systematically beaten throughout his childhood -- IMO he'd be even more fucked up than he is if that were the case -- but he was certainly knocked around and verbally abused.
Someone else mentioned the scene in S2 when Joan brings over the cake, and how odd it was that Joan mentioned when she and Brian used to be close. I'm sure they were. I think Joan probably did love Brian, and show him love, in her own way. There's no way whe'd be capable of empathic bonds and love if he hadn't had love as a child, however dysfunctional. And Brian is an extremely, deeply loving man, capable of profound connections with others and great loyalty.
So I think that his mother, before her alcoholism progressed, probably did show love to Brian. Which is almost certainly why he had such trouble accepting that he really shouldn't keep giving her another chance -- and yet even after she accused him of molesting his nephew, he nearly gives her another chance in S4.