With those portkeys around, I can't imagine it would be hard to travel all you want. And if you need a companion, I can do that now.
I would be more worried about it if he didn't adknowledge you at all. That he cares enough to give you shit is a start (as awful as that sounds). And Navi? You're welcome. I'm glad I have you. More every day, and more than you know.