It may seem like a minor thing and well worth the loss to placate the merfolk, but Jack's hat is very important to him. It's a symbol of his station and a source of pride and vanity. He's loathe to part with it to the point of further bartering and negotiation with the sea people.
They seem uninterested and insulted by his offer of rum. Clearly not very intelligent creatures. He wrinkles his nose when the bottle is tossed back in the boat to begin spilling out. Jack sets it upright again. The hat seems to be all they're interested in, but a good and well fitted hat is hard to find. He remains wary when one goes out of sight, and he listens to the others as if he has some chance of understanding them.
The mermaid's pleas are sweet and he realizes she's raising the offer, but Jack's hat is his own and he has only limited use for the shell. Perhaps she'll accept something else? "Anything but the hat, love..."
She won't understand, so he tries to make himself clear. Jack gestures at the shell, then to each item on his person and on his boat. Then he pulls his hat more firmly onto his head and shakes his head. "Not this. Anything else. Savvy?"