Things weren't fully adding up and it was confusing James if he thought about it. The full mind control hadn't had time to fully set like it had with Bucky.
"I'm glad everyone is okay. SHIELD can be dicks." So far as James knew, it had been SHIELD who had taken him and his friends and HYDRA were the good guys.