Well, yeah. It was clearly the ghost who was screaming "don't kill me." Takahashi responds with "Wha... what?"
But remember, that was actually his second and most recent encounter with Funny Eyes, which happened yesterday. The first encounter, which happened "some time ago," was the one where she almost strangled him.
So first she tried to kill him before letting him go, and then she showed up begging him not to kill her.
Also I don't believe for a second that sickle-handed ghost really did have the wrong house. He was just trying to cover his ass because he was so damn scared of Body Blade Guy.:p