Say you end up on Rot Lop Fan's world for some reason. No one's got eyes, so if they're all just sitting around in a room, some of them are picking their noses or scratching their asses or something because who can tell? Then you show up, and you've got eyes, so you know when they're doing this.
So, what's fair here? Is it a violation of their privacy to look at them? Is it rude? Is it fine? Are you going to walk around the planet with your eyes closed, or wearing a blindfold, except when you're given permission to look at someone? What if you hear a noise and instinctively look up? What if you're running and need to see where you're going? How much do you have to compromise, and how much do they, in order to be fair to everyone?