If on some level, you don't believe yourself worthy of the Keyblade, the bond can be broken. It's... well, I willingly gave it up once.
[For about ten seconds before Master Yen sid he was worthy anyway, but that's another story.]
But I've only ever heard of the Keyblade actually leaving its master one time. I never thought it would happen to me... but, in the end, it was for the best, since it made me stop tryin' to hurt you.