Imagine all the medical and insurance bills Hank Pym had to pay. From his constant injuries, to his worsening heart condition from all that random shrinking and growing. Not surprising Iron Man, Giant-Man and the Wasp were the first to willingly leave the Avengers. All due to physical and mental health issues that were still present in some form when they returned.