Probably, but you two need each other. I don't think you'd be right without each other. You two love each other, and christ, the look he gets when he smells sex on you is surreal. I love you, and I don't want to hurt you by standing in the way of you two.
It's not like you're doing it with someone I don't know, or someone you don't love. And it's not like I won't be there sometimes.