He leaves me, or I him. Regardless, rumors abound. It would be easier if one of us left town, even, but that is only a patch over the bigger issue: assuming he repairs your relationship, does he insist on keeping it a secret? I suspect he would.
We stay together, save face, fend off the vultures as need be.
The latter is obviously preferable to me, as it's simpler and I know the rules.