"He's the one who scurried off! I was quite prepared to keep trying till we eventually got there!" Possibly. He was nearing the end of his tether too when Ianto walked out, but still he does have the moral high ground there, just about.
He pushes angrily against Spike's infuriatingly smug frame. "Why don't you make sure he's actually willing to talk to me before you come here accusing me of giving up!"
(Perhaps he should build a crypt, in lieu of that cell they're always talking about needing and never get round to sorting out!)