The name was not Tom recognised (other than Potter being a pureblood name), which increased his confusion. If he had terrorised someone to that extent he would have remembered, likely as it would have been such a pleasant memory.
"You are a fool. I have never met you," he sneered softly. "Nor do I have any ability to leave even if I wished to do so. Again, where is here?"