Barry knew that he couldn't argue with what Patty was saying. It wasn't his place to make that decision for her and he should have trusted her to make it for herself. But he couldn't change the past. Well, technically, he could but given how that worked out last time, he wasn't about to try.
"I wanted to, Patty. I l-... care about you. And I was selfish. Because I didn't want to take the risk of losing someone else that I care about. And I realize that wasn't fair to you."