Yeah, it really is. If anyone's found a way to willingly leave, they've not come back to tell anyone else.
I have noticed people come from different points in time in their own universes without others from the same universe seeming to notice they've gone, so it could be a case of timeline divergences, where one of you is back in your universe with the people you want to get back to, preparing to maybe go to a wedding, while this you is here with us. In a very big mall. With cinnamon rolls and a credit card with unlimited credit.