The only way you can perma-remove people from your 'friend-of' list is to ban them. However, for a game, that's not a great idea, in case anyone ever wanted to recycle one of those journals. If you go to 'manage profile' and unclick the 'friend-of' option, it will hide it from people who are looking at your user info page.