Sirius never wanted to let go of James, if Lily had been there he'd have been clinging just as tightly to her as he'd been to James. Like if he held on to them tight enough he could keep them, like nothing could happen to them if he kept them safe the way he hadn't been able to the first time. Things were - they were complicated and he didn't know how to have this conversation with James and tell him what he knew.
"I - James it's... we're not in our world anymore." He said finally, "This happened to me a few days ago," Had he been there a week already? Sirius wasn't sure, he was missing time after what had happened to him, but that wasn't important now. "I showed up here like you did, there's some sort of magic at work that pulls people here from different times and worlds." How was he supposed to tell his best friend what had happened that night? That he was supposed to be dead, that Lily had also died? Sirius' heart ached for his friend.
"We're stuck here, James." He told him, "This place, whatever it is - it keeps people here until it's finished with them. You can't get back." And in truth, selfishly, Sirius didn't want him to go back - he was still alive and real and here and Sirius couldn't give him up now.