He's a terrible communicator and essentially kept pushing her away. It started around the time their sister passed away but Rory just let it slide but it got worse over time to the point where she couldn't do it anymore. She realized it was a losing battle and to save her sanity she had to walk away.