almost a year late, but the QAF companion book makes it damn clear, multiple times, that Jack beat Brian. Cowlip used terms like Jack was physically cruel, abusive hell, etc.
So it was at least their intention that Brian was physically abused, along with the mental stuff.