No, they won't. That's one of the precautions Acumen takes. When he takes us, he puts a replacement in our place. Apparently no one from your world will know the difference.
That is what we're told, and it seems true enough. People who come in that know you won't have realized that you're gone. However, Acumen can also bring in people from our futures who we haven't met yet. That wa a little startling for me!