Custom is a custom filter, like the filter that lets you lock certain entries just to certain groups of friends.
Don't feel bad about not knowing what it means; it's not a term used by all circles of RP. I've been in RP for about 10 years myself and have never been in a game or done a storyline with "customs" involved.