When they play "You Are My Sunshine" at the end of 507.
Until then, we don't really know as viewers exactly where Brian's head is at. Yes, we all know he loves Justin, but he let him walk out the door not, like in S2, because the two of them were so fucked up and lost, but apparently calmly and with some kind of understanding of the impasse the two of them had reached. They loved each other but wanted different things. We could see it caused them both pain, we could see Brian's face and that he was suffering, but he was still clearly choosing this.
It wasn't until that song started to play that we were truly able to understand that Brian was, in fact, still trapped in something he was NOT choosing and did NOT control -- as he later said to Ted at Kinnetik, life with a frigid bitch and an angry alcoholic had made him unable or unwilling to commit to someone (ie, SCARED and PARALYZED) and he'd thus lost the two people he cared most about in the world -- and that he not only still loved Justin, but he loved him in that primal way that we always imagined/hoped/believed he did... but weren't quite sure sometimes.
The first time I saw this episode I knew what was coming, and while I found it upsetting, I was mostly okay (I'd been spoiled for all of S5 before I saw it). But when that song started to play, I burst into tears. It gutted me.