He shook Matthew off rather quickly, saying (or whispering loudly, more like), "No, no, I'm fine, just...scared, you know. I've right to be scared, haven't I? And are you really sure about that? I mean, they must read the papers at least. I just don't want to risk it!"
At this point he was looking around, deciding whether or not he should go out through the front door or the back. The back door would afford him more secrecy, if Matthew had nosy neighbors of any sort, but it was also further away and maybe Matthew's aunt would come down the stairs and see him and...well, he'd just have to risk it. He only needed to get out of range of the wards, and then Matthew could just say it was a neighbor or something if his aunt asked. "I don't want any trouble. You understand, don't you?" he asked as he began to walk towards the back door.