The thing was that Oliver almost said 'yes'. It wasn't like he wanted to break up with Charlie, hardly. Oliver wanted nothing more than to be with Charlie forever, but the reality was that Oliver was convinced he would fuck this up, sooner or later he would do something that would make Charlie leave him. So was it worth it? Was it worth having people hate them because they wanted to be together? Was it worth having Charlie's own brother be unhappy with him because Charlie liked Oliver slightly more than a friend? What really made the relationship between them worth such a... hassle? Oliver certainly didn't think it was him that made it worth it. He could see why it might be easier for Charlie to just back out now, before they were in too deep.
But Oliver was selfish. He wanted to be with Charlie, all he had ever really wanted was to be allowed to be with Charlie. Allowed by Charlie. Oliver hadn't actually ever considered that others might feel like they have a say in this, too. They didn't. No matter what Ron Weasley or Daphne Greengrass or anyone else might think or say, Oliver didn't actually feel like they had any right to decide whether him and Charlie should or should not be together.
"No," he said with a shake of his head. "I really don't," Oliver admitted with a small but rather bitter laugh. "I just--" he paused, trying to decide what the best way to express himself was. "I don't want you to fight with your siblings because of me," Oliver finally said. It was true, after all. Oliver really didn't think he was worth Charlie falling out with his siblings, or one sibling even. "I don't want people to say things to you, bad things, just because," he shrugged whilst saying it "because of me," Oliver said, before looking down at the floor. "I don't think I'm worth such a bother," he admitted quietly.