Yeah, I think I'm more with you on that one. I don't think Brian said it, but I also don't think Justin was completely lying and making it up either. Just misinterpreting and hearing what he wanted to hear.
My guess is it's obvious that Brian is impressed with Justin right from the beginning, and since he seems surprised by it, it's very plausible in my mind that he could have tossed a line out there like, "I love your ass," and Justin would have gobbled it up.