Rory. I have just woken up in a place without my memories where people either recognize me on sight or claim that I remind them of someone. One them has claimed that I have an "impossible face" while the other says that the person I remind them of is also called "The Doctor". Now, as someone who has just woken up, with so little to work on with his own memories, and getting to know the situation he's in, I may be liable to jump to impossible conclusions and therefore cannot trust my own judgment in these regards.
So. What we can do for now is. Settle with what we know for now. I'm so so very curious about all the possible connections but jumping to conclusions has never done anyone any good--with that in mind.