Another excellent question. I would avoid mentioning that you are from Midgard. Midgardians do usually not have access to Asgard. If you pretend you are a generic traveller or someone from Vanaheim you should be fine.
That said, it should only really raise suspicions if it gets to the chief of guards or Heimdall. Common folk with not question guests of the Royal family.