[He's never been really good with goodbyes, especially with those he really cares about. Even if he knew that Hiyori, David, Snow and Neal were going to leave, he can't help but be sad about it. So to help with that, he's readying himself to go towards the mountains and find those baby foxes. Because it sounds like a really nice distraction.
So perhaps he runs into Penny either in the mountains when he's already there, or while he's getting ready to get going.]