I'm pretty sure most people care more about the mental state of their cheating partner than the actual act. Like that scene in "Love Actually" when Emma Thompson confronts Alan Rickman about the necklace he gave his secretary, and she says, "Was it just a necklace, or was it a necklace and sex, or even worse, a necklace and love?"
Obviously, most of the time, if someone's cheating physically, even if they don't love the person they're with, they probably don't care all that much about their legitimate partner either.
And honestly, this isn't "just a thought," it's the telepathic equivalent of phone sex. How many people would be ok with finding your spouse having phone sex with one of your associates?