Take the fact that we come from different points in time. If we're from the same timeline, then those of us who are from later spent at least some time interacting with the "clones" the wardens deposited to replace us. That possibility falls apart, though, when you consider that those "clones" still committed the crimes we were arrested for in the first place.
Another possibility is that upon our eventual escape, we return to the exact point in time we were taken from. The problem there is, those who came from later in the timeline didn't experience anyone from earlier on mentioning this place. That doesn't apply so much in our case, considering how little time has passed for each of us and how much we've had going on in the meantime, but people from other universes have reported the same phenomenon. It might be in our best interests to avoid spoiling each other on what's to come, but I doubt everyone is going to be that conscientious. Still, there's the additional possibility that we'll all have our memories erased. Which I don't like.