Well, out of those, Scott's not much of a killer, he knows he's safe from Peter and as for Doom and Loki, he probably believes (probably mistakenly) that gods and statesmen might be villainous but they're not about to kill him like petty thugs while he has something to offer them.
However, Mystique's the kind of person who'd kill you in your sleep, smile while doing it and then take your form and live your life in your place. She's someone to be nervous of. Especially when she has a large bomb inside her. She's not as... predictable as some.
Mind you, he hasn't demonstrated a similar nervousness around Bullseye or Sentry, who are even more unpredictable, but then they're not stroking his arm, presumably.