I believe it was something about not wanting the Defense Force to find him, because he attacked his brother.
[The words are spoken lightly, but he's studying Kisame, looking him over closely. Did he really see nothing? If Kisame knew Sasuke well, and looked at him as most people did, the other reason for the kid staying with him would be obvious.]
If you look at my face and only see me, then it's hard to explain.