The tl;dr version of events is that piece of shit Chatwin brother broke your hands and it was too dangerous for you to do any magic and we were on a timeline to save Fillory. Unfortunately, when everything happens, it pulled us from Earth and to New Fillory with no way back and left you behind with Julia and Kady.