The thing was that James didn't believe him. Because if he really felt that way, he would've done what any of the rest of them would have done: kept the secret, even if it meant dying in the process. James knew he would die for his friends, including Peter, maybe even now-- no, probably not now. Not anymore. He had seen Sirius throw that hex just at the same time as Remus did, and Remus had been the one to jump in front of it, not him.
He didn't want Peter dead. But that didn't mean he was still willing to sacrifice himself to prevent it from happening.
"But you did, Pete," he said grimly, shaking his head. "Maybe it hasn't happened yet, but you did. And if you can't give me a reason why you could see yourself doing it, then I can't trust you. I have to know what it would take to make you do something like that, even if it's just a guess, or else I'll just-- always be wondering."
Maybe Peter really didn't know. But when it came down to it, that didn't make anything any better. He couldn't be friends with someone when he was always wondering what might make them turn against him. Not when this world held so many dangers. What if they just... went outside together, and a pack of mountain lions showed up? Would Peter push him into their path so that he could get away first?