"Wait. I displaced someone?" Someone who had a life, friends, possibly a family? Steve understood what Tony was saying: no one had found a way back. And, if Mr. Stark had been there a year, surely he'd tried. But. It made him unhappy, and the set of his jaw said as much.
"Do they ever come back?"
That was what was bothering him. Not 'you can't go home' but 'they probably can't return to their own lives.' Steve was used to loss. He wasn't used to taking from others.