Oliver tried to absorb what she said about Sam, but he didn't know Sam and it didn't make a lot of sense, so he just let it go. He'd deal with it later, if he ever met Sam. He never met David, after all, so he could reasonably assume he'd never meet Sam either.