"Out of curiosity, wouldn't being around me if I don't want you around be worse?" He squints, not really sure about the whole concept. Jim tends to be very anti-social at times. The only people he can stand and enjoys being around on a regular basis were Sherrinford and Sebastian, but those were half out of necessity and obligation in a way.
And those that he wants to be around, he does. He makes that effort probably too forward. How many times had he simply be peeking over Sherrinford's shoulder or annoying Sebastian while he cooked?