The damage is done, though. I don't think that you leaving will bring things back to the way they were two weeks ago. The problems and distrust that has been created will stay and we don't know what the castle will do to us.
It's seems like there are only drawbacks to you leaving by yourself.