It's sort of like how it works here too - we can rationalize and say that those people aren't us, that the memories aren't ours either, but they are. There's no getting rid of them.
But with this, with you, I don't know. It's like you need someone to go in and clean up the junk and the extra crap to leave your headspace clear.