Don't quote me on it, but I think that when people return, they return to the same place in time that they were taken from. Geralt wouldn't have gotten the time to take notice of your absence. We're talking about the time that passes between each half of second.