I think he does say it in season five to Ted, "my father was an abusive drunk" - when someone says that, I'm sure they're referring to physical abuse of some kind. It was HORRIFYING that Brian went there for BRIAN. It was definitely a sign that he'd been very hurt and traumatized by Justin's leaving.