I wouldn't say that humanity in your world won't spread further given the opportunity. We're very adaptable.
Once time travel became available, people inevitably made mistakes or misused it. The Time Agency was responsible for fixing those mistakes and stopping anyone from doing anything too disastrous. A little gentle sight-seeing is all well and good, outright meddling would get our attention.