Jacen
Imagine if she was to come back. Like Anakin did. Maybe remembering this place, maybe not, that detail doesn't matter. Imagine seeing her again and having to explain why you stopped caring.
Eventually it will be found out. The wrong word, the careless gesture, returning home smelling of my perfume, anything. Secrets like those don't remain that way forever.
It could cost you your marriage.
If she wasn't here, or if she wasn't an issue, I wouldn't hesitate. Anakin has already made it clear he cannot stand me, why not add more disappointment to that.