Aside from all the strong similarities to Kuttner and Moore's "Baldy" stories, I'd say the other big inspiration for the X-Men would be A.E. van Vogt's SLAN - also a classic of pulp science fiction that Lee and Kirby would certainly be familiar with and likely have read. There was an earlier generation of sci-fi geeks who had the slogan, "Fans are Slans."