"I'll leave you alone so I won't scare you," he said. Harry backed away, hands up. "I'll go home. Just see your mum, or George, or Ron, okay? And have them owl me."
Harry was worried about James, but he didn't want to scare Ginny into an attack. He didn't ever want to duel her. She was scary.