It was actually not his intent to lump her in with the rest of the smelly, gross dullwits surrounding them, but if he knew it bothered her so, he'd be quite glad to pretend he meant it that way all along.
It upsets him that her mockery of his reputation, or lack thereof, is so accurate. He seethes quietly as he drinks his ale.
"My reputation has suffered of late, as a result of my allowing women onto my ship and not demanding...improper services from them as payment. It is most hypocritical of you to mock me for it, as you are the one who benefits."