PEPPER/BUCKY.
None of us really know how any of this ultimately works, be it how we end up in these worlds or where we go or don't go when those worlds and memories don't line up. Who's to say what is and isn't ours, when we know so little?
I don't think there's a right or wrong to what you keep and what you store away from what you've been given.