When I first saw it, I assumed Justin was just making it up. Later, it did seem some fans thought Brian, in the throes of being wasted, having a new son, etc., might have said it without meaning it.
Like you, I don't think so, because obviously Justin doesn't act like Brian ever said it to him and it's clearly a big deal when Brian finally does.
I also believe we're given that as a fact in 108, when Justin says, "He never said he loved me" when Brian tries to bring him back to his family. Brian's right there when he says that.
So I'm pretty much 100 percent convinced that Brian did not say it, and that Justin just made it up. He had a pretty vivid imagination when he was in high school. ;)