See, each Keyblade is attuned to its owners hearts. How it looks, what kind of spells it can use... They're all different. They're based on the owner. So when someone else holds it, they realize the heart is different, and they return to their real owner.
But... if I ever doubt what I can do with it, if I don't feel worthy of it, that's when it just becomes a regular weapon, and someone else can get away with it.