Mostly because of the tremendous amount of repression in the culture of that era (sex even between a married couple was frowned upon except strictly for procreation), wealthy men usually found 'relief' with their servants on the slide.
See also, Thomas Jefferson and his you-know-whats.