[that is a resounding "no", even if he were remotely suspecting what you were thinking, which he is not]
Or a minor modification. Like-- [he lifts his hand to his collar, but instead of tugging it, he brushes his fingers back along it, as if flicking back hair]