Alice
What you do is your business. It shouldn't be anyone else's. just because you aren't famous doesn't mean you deserve to have your affairs splashed over some gossip rag.
I'm sorry, I'm sure I'm presuming too much. If you still want to go for tea that would be nice but please, don't feel obligated.