Well I was singled out for a national gifted education programme after a series of completely inscrutable tests (and this was when I was 9, imagine the horror), so I would say it does depend on the *treatment* of those people.
I think that having inborn magic, especially if it has destructive potential, is something we really have no equivalent for in this world. Intelligence often provokes envy and subtle put-downs, but then intelligence can't actually incinerate you, or better still make you incinerate yourself.
I suppose, like many inborn things, magic would set the 'haves' far, far away from the 'have-nots', especially if there are no strugglers and ambitious 'learning mages' in between. They would probably be isolated, and people would go out of their way to avoid them - do you *really* want to risk saying something wrong to mr. fireball? - and that could lead many ways, to arrogance and contempt, or isolation and bitterness or anger...