I suspect it's A) The entire power-differential thing. Some people seem to find that sexy.
B) Since secretary work was one of the few jobs a middle-class woman could take (working class women worked in all sorts of places) it also tended to be a job that young women took in order to meet men in a socially acceptable setting.
By the 40's that should be on it's way out already, but was definitely a factor earlier on.