All things considered you could potentially say that about anything. Humans, or otherwise, find all kinds of reasons to be violent it doesn't just have to be sex based. Most crimes of jealousy are speculation or jumping to conclusions anyway. Humans have emotions, there's no limiting that. You may as well take away all the freedom you've given everyone simply out of paranoia.
I know first hand that trying to limit something doesn't make it go away. Technically you're not preventing that slim chance something could happen, you're creating more opportunity for it. People find their ways, and there's more excitement in knowing you're doing something potentially dangerous.
In the same sense, giving us a little extra leeway could also help prevent anything further. We'll have our outlet. That one thing to get our fix and be done with it. No hassle, no worries. Everyone goes on with their lives in a glass bowl.
Redefining what you consider sex may not be a bad idea. If you want to win our favor and keep us happy, that's a guaranteed dead ringer and you still keep your control.