"No, you're right. In the end what happens between you and Elena is only your business. I'm not going to disagree," how could he? Unless something became dangerous (which it wouldn't) then it really wasn't anyone's job to step in. Damon wasn't going to harm Elena, despite what Jeremy might think about the elder Salvatore. She was his everything too and Damon would protect that.
"Jeremy needs help, I think that's obvious. He's been through some heavy things, but he doesn't process grief like Elena, obviously. I'd say the best you could do is keep trying to make it work for your girlfriend's sake. If she wants you over, she should be allowed to have that. It isn't fair otherwise."
Stefan was quiet for a moment afterwards. "You are my family."