"I doubt you'll ever need a babysitter. Tony probably would let me anyway." Steve was good enough to have sex with, but not good enough to have the child be left with him. Oh. That was a nasty thought and he immediately shook it off.
"I know you guys do but you...are starting a family. A real family." In other words, there was no place for him. He was just a fuck on the side for Tony. That is the only thing that made him special. Outside of that he was just another friend.