[haha, so he was right, you don't like Rory. He stays silent for a few good moments before saying anything]
None of those are easy questions to answer. It's obvious that the last memory you got caused Rory to disappear— or repress him, in any case. But I don't really know why he didn't turn back into a complete blank— that would have made more sense.
If I had to guess... back then, all those days ago, something relevant to you and Rory, and only you two, happened. There has to be a connection— a reason as to why you only remember up until the moment you took your first memory.