You're wrong! Rory isn't a dream, it's who you are! That woman makes all the difference in the world— isn't it confirmation? If you remember about being Rory, then cling onto those memories and think hard about it-- why would the gods curse you with something like that? It's completely illogical.