(Now, this is why I cannot buy him as a priest. He is not of the chaste. He's very faithful, in all senses of the term, but he likes sex too much to write it off forever and who can blame him?)
He also chastises himself for lusting after her. Kurt's always been the kind for keeping a clean house, morally. That's why I can totally see him wanting to become a priest - to strive for moral perfection. Of course, his biggest problem turned out not to be lusting for women, but that he couldn't stop beating people up.