[Regardless of Goku's misgivings, there's no attempt from Sebastian's side to coax him into following. After he starts walking, Sebastian pauses once to look back. He doesn't repeat himself, simply gives Goku a look that seems to say "come, then", before he turns around and keeps walking.
He's heading back to the bakery, by the way. No way is he bringing back a stray child to the why-is-this-a-thing shared household with Time. She'd just get ideas.]