He could've just said yes but last time he thought ti was completely under control he was very wrong.However Banner seemed calm enough to carry out this conversation even while Hulk wasn't thrilled to be in S.H.I.E.L.D's Head quarters. That had to be proof enough that his control methods were working.
"Over three years without an uncontrolled incident." He explained. "And if this conversation is leading to a job offer, I can't. To be honest he already kind of wants to rip your face off just because you're sitting in the bosses chair. And my loyalty falls with Stark and his Avengers."