I think Osborn's history with JJ was referenced when he said "Not everyone is forgiving as you are."

While I want to learn more about what JJ is thinking, it makes sense he wouldn't want to make Osborn his enemy now that Osborn is the most powerful man in the country.