In Forest World, with the rusty key you dig up at the hand tree?
What you have to do is get the key, continue to the next area where there will be a ghost - the very ghost I was cursing at, because it kept getting in my way - a golf club and a portal. Go through the portal, dump the key in the chest, go back through the hole, run back to Wish House, go through the Wish House portal, grab the key, go back through the hole so you're in the Wish House Courtyard with the key.
Essentially, you can't have the key in your inventory when you try to leave by the paths, or you will be 'doomed to wander'. Mmmm, vague puzzles! SH4 is full of them!