Daphne wanted to say that Astoria should have already known that. No matter how jealous Daphne may have been, she was incapable of being so heartless toward her baby sister that there couldn't even have been a tiny part of her that was happy for her.
"Do whatever you want. You don't need to stay if you don't want to."