James
Maybe for some of us it is guilt that causes us to stay.
Back home, I failed. I failed my people. My friends. The people of Midgard who I swore to protect from the cosmic circus that they did not understand. Hell, with the destruction of Asgard, I have likely doomed the entirety of the nine realms because there is no one watching the gate anymore.
But, Laura is right. None of us seem to remember their time here. If they did, surely Stark would have said something. Or your mom. Or dad. Or Barton.
It'll be as if you'd never left.
But there are people who would mourn your absence, should you go back to when you came.