This is pure conjecture, I imagine the more the infinity gauntlet was snapped, the more destabilized it made things. You can't change the structure of the universe over and over again, without lasting consequences.
Everyone else seems normal. When I went to Kamar-taj, none of the mystics seem to think anything was wrong. When asked if they remembered seeing me around, they said 'recently' but couldn't recall when or where.
It's safe to say that those of us who were displaced into this universe know about it happening.