Unfortunately when the "fancy rich lady in the big house" calls, you go. So if you inform them you've been summoned by the Queen, they'll have no reason to go after your spot on the team.
the 'and they want to drag me' bit makes that sound less than convincing. Forgive us for not believing it.