He did only listen with half an ear to what Stoker said. So, it was no surprize that her last comment took him blind-side.
You have a handsome voice. That's what she said. Why did she say this, he wondered.
"Spare me the boot-licking, doctor. I know where my position is."
Vader's words were cold like the dunes of Hoth and at the same time as hot with anger as only the rivers of Mustafar could be. He never liked it when females tried to win him with their... charme.