I don't think most people did in my world either. My parents loved it, but they were an unusual case. Usually it's scholars, military, Traders... Some richer nobles have homes in the cities as well as their estates, but they generally only move between the two.