"Why the hell would you willingly spend time in a library?" Connor's not game for the whole bookworm scene, so I can't imagine there's be any reason for them to know each other outside of the very occasional school-related issue.
Walker has no opinion (Wow I say that a lot.). He'll be in the library more often than Connor will be, but he's not one for small talk...or social interaction.