No, I'm fighting with all of you. I understand that.
What you aren't getting is that they're the weapons. Sam Winchester is the weapon. You could always go back and try to destroy him, but I'll just bring him back. Which means, inevitably, he's going to say yes to me. Which also means I'm going to have my true vessel, therefore I will be completely unstoppable.
Hey, maybe you'll be able to convince Dean to say yes to Michael this time around. I don't want to fight my brother, but I'm sure that'll make for an interesting turn of events, don't you?