It's worth keeping in mind he came up with that particular style choice when he was under the delusion he was his own murder. That was not the fashion pick of a sane man.
. . . and then of course it became his favourite outfit.
On, Avengers, why do you never pay attention to the warning signs?