They should be worried. Oh he was playing along with the truce. Except admittidly for the Loki thing which was admittidly hilarious, and Simmons could hardly fault him for that. In spite of all he'd done, Phil Coulson was a good man. He knew that. And he knew he didn't deserve what Loki had done to him, even if it was in the Asgardian's future.
The comment about orders though. That stung a little. He could make decisions himself. He wasn't just some damn kid Garrett had control over even now. He was gone, everything was gone and there wasn't a lot of point to thinking about it anymore.
And Romanoff was here somewhere too. He was impressed she was hiding well, Oh he'd get eyes on her eventually. They were evenly enough matched in terms of espionage. So said the test scores anyway. "I don't blame you for being wary. You're right, you have no idea who I am. But you're still here, still willing to be the one to put yourself in this position, so spill. Where is she?"
It could only have been about Skye. It could only ever have been about her. Why else would they need to tell him directly.