Yes, of course Joan is not to be too trusted, but I rely here more on Brian not telling her it was bullshit - I just think he would if what she said was not true, he certainly had no qualms about calling his sister's bullshit after the funeral.
Also, Joan strikes me as the martyr type, "sacrificing" herself for the child, in her catholic mind protecting him from the father, and failing to see the myriad ways in which the family dynamics ruined him.

But of course, the question of whether Brian was physically abused remains a matter of interpretation, but I always felt he never was abused in this way. His incredible pride in his body and the free and unashamed manner in which he uses it for pleasure also makes me think he was not hit. Although I am hardly an expert on the subject.