"Because I loved you!" The moment it comes out of his mouth, he realises it was in the past tense, and that perhaps in his heart of hearts he knows it's over too. He closes his mouth abruptly, and deflects by focussing on the other strand of the argument.
"And it wasn't the second they went. It was over a year!" Give him some credit, Ianto. He tried. "We were exclusive for years, the three of us." Think what that means, for him. How hard he worked at it, for a long time. Right up until they'd exhausted all the options and realised they could never leave and the terrifying prospect of being stuck in one place, in one relationship, for eternity finally struck home.
"And I came back! I tried again - I tried for years - for you! It's not my fault you're so inflexible - that in your world it has to be all or nothing!"