I encourage people to challenge characters, personally - especially given how often people apply for characters and then let their holds fall through.
I've had two people challenge a character before and still not received an application for either.
However, when I do it's with the following rules in place: -I don't allow further challenges to an application after I've received an application for a character. -I do not grant challenges extensions (although the first person to put a hold on that character is allowed one).
I usually put a section in my app asking why the player wants to play that character/ things they're looking forward to doing with him. That's usually a huge factor in who gets the character.
I think the best approach to telling someone that is just being nice but curt.
"Thank you for submitting an application. While we very much liked your application, we've decided to go with the other player's application as we feel they would be a better fit for the game. Please don't let this discourage you from applying for a different character because we'd still love to have you."
Or you can bluntly steal borrow from my favorite college rejection letter ever (Duke's) and make a rejection like it:
"It is with real regret that I must inform you that we've decided to go with CHARACTER's other applicant. I realize this is disappointing news, but I want to assure you we considered your characters carefully and it was not an easy decision. AWESOME STUFF ABOUT SAID CHARACTER.
So, if you're asking what your application was lacking, the honest answer is nothing because our decision does not reflect any deficiencies or weakness in your application. The other applicant's character just appears to be a better fit for this game and the characters s/he will be most frequently interacting with here.
I know, eventually, you're going to play this character at another game and that the game you choose to do so at will benefit from your presence."