Having also known at least a couple gay guys who were into yaoi, I'd say that opinion might not be as universal as you thought ...
And speaking as a fangirl who's read gay erotica/porn written by (presumably) homosexual authors, I can definitely say the feeling goes both ways for the 'doesn't do anything for me' camp ...