Ironically (or perhaps not, given his general attitude to such things) Jack has never really blamed him for cheating with Spike in the first place. It was the prospect of Ianto leaving him for Spike he was far more upset over - and apparently with good reason, given how things have gone in the long run.
He's getting really frustrated now, bitterness overflowing at all the things he gave up for Ianto, all the time he wasted, all gone for nothing. Whatever he says, Ianto just shoots him down. What's the point in even trying?
"I have to admit, it's hard to see what I saw you back then," he says bitingly, face dark.