Eli didn't like to think about his father. It easier to not contemplate the life of a man who went to prison for his work, who ultimately lost his mind, and died a fragment of his former self. The more superficial part of his mind told him he didn't like to think of his father because it bothered him - that fear of losing his mind. It was his most valuable possession and yet that deeper part of him, the one that took things apart to see how they worked knew that aversion was far more significant than that.
"So what? Are you saying that you took my father's memories?" No, that didn't seem right. While she might have done anything for the family, Eli knew that Samara had a close relationship with both of his parents. The problem was that was the safer conclusion to go to because the other one opened up far too many questions, but he said it anyway. "Or rather my father gave you his memories, which probably indicates that I'm not meant to know about them unless it's unavoidable."
To some extent, it had always been her choice, but he was still giving her the opportunity to change her mind. To not take them down this path because deep down he trusted his father and his wife to do the right thing. "So, have we really reached that point? Are you sure this is the time?"