You think, Brian meant his father was verbally abusive?
Well, I think there's PLENTY of other clues that lead to conclusion that there was physical violence at the hands of his father. It was his mother that was emotionally abusive.
One example is his reaction to Craig hitting Justin. Then there's the episode when his dad dies (doesn't he talk about how horrible the bowling was) and when he tells his dad that he's gay, he flinches when he thinks his dad is going to hit him.
Sorry, but it all leads to physical abuse to me. I'm sure there was PLENTY of verbal abuse too but that wasn't all.