Yeah, but then there's the whole time travel thing. Since we didn't end up with fifty Boosters at Barbara's apartment, one can assume that the previous attempt was erased from the timeline each time he tried again. If Barbara doen't remember, they can just say that the last timehe tried, he was taken down outside of the apartment, and Barbara was unaware. Therefore she does not remember him being there.
But really, I meant 'maybe' as in a potential story idea, not that Oracle saw him then forgot.