And it wasn't because Justin left him; it was PTSD related to the bashing and his childhood.
It's both. If Brian wasn't broken, he could certainly withstand Justin leaving, but he'd still miss him and be sad. But he was broken and Justin leaving had a terrible effect on Brian's well being. For one reason, I am sure Brian felt damaged and that, that damage had something to do with Justin leaving, etc.