Time-travel. Sirius remembered that part fully now; that was why Harry and Draco just showed up. Brains were weird, he knew that Harry was James' son and came to school with them when they were all the same age, but the piece that connected the knowledge with the possibility, time-travel, had not slotted into place until this moment. Despite seeing Harry almost every day, it was as if he forgot about the time-travel.
Then there was that Azkaban comment, why did that feel so familiar? "Why would I be in Azkaban, pray tell?"