If I trust the person I was warned by, if they have knowledge I do not over a person, I would heed the warning. This is a mortal life, there is more danger than we can assume.
Be polite, be distant and simply say I do not wish to pursue a relationship for my own reasons.