Naivety and pride are a dangerous combination, and I had both in spades at one point in my life. Having such secrets is not a duty I wish upon another person, wizard or not. I made my bed long ago but only now I am finally forced to lay upon it, but I'm doing all that I can to keep my secrets from being the downfall of others. I can't handle more people being hurt because of my sins.
Memory retrieval and removal are both delicate tasks. They require elite magic and can only be used by those of us who have such prowess. That being said, I've known the practice to be abused before. I've no doubt it will be abused again.