I was in the neighborhood! [Which... is actually perfectly true but doesn't really motivate why he actually went in.] I just... I didn't want him to think he had the upper hand. That I'm scared of him or that he's the only one who can show up unannounced.
[The term you're looking for is "mark your territory"]