No, how could he? Justin TOLD him they'd see each other and for all his false bravado about it being "only time," Brian expects that too. NYC isn't the moon...they have the ways and means to spend plenty of time together.
And Brian would see that *and* Justin's coming and going as part of the direction their relationship is taking. It's all good.