I think that seems to be the case for a number of people. It's difficult to not think of a shapeshifter or some other situation where they are purposefully trying to impersonate you and it usually isn't a good thing.
You know, I've not looked much into what the actor thinks of things -- Maybe I ought to.