That's true with everything, Night. I don't know about your situation, but... Life is made up of joy and despair. They don't outweigh each other, if you don't want him to remember you so he won't feel sad you might as well wrap him in cloth and never let him out.
You don't have to tell him. Just keep in mind he'll remember eventually, and how he might react.