He's just trying to do his best in what is to him still a very new area, but an area he's working at - hard.
"Oh." He's a little jealous of that statement, even if it does sound like her. It still feels like a long way away from what he and Ianto have managed to achieve.
And then he gives her a bit of a look at the second part. "You were the one who was worried about how he'd react! I'd say it's no big deal, but you made it sound like it was!"