"You.. want me to perform reverse memory charms on myself? Are you crazy? Do you know how dangerous that is?" While, obviously, not all of his memories were good ones, he'd rather keep them were they were, rather then end up like Professor Lockhart. Which is what would happen if something went wrong.