That's just a little bit too close to a reminder of how he ended up dying, far too soon, and Jack pauses for a moment, swallowing it down, before the renewed tightness of Ianto's hug gets through to him and he relaxes somewhat.
Then lets out a surprised laugh at that last part. "Of course you're worth it! Your're - you're one of the best people I know!"