Adam didn't know whether to be mad or sad. He was just a complete jumble of emotions right at that moment, and he hardly had time to really react to what he was seeing. He didn't have time to think about how he should have reacted.
But when she said she didn't know that he was coming by, it kind of pushed any other emotion he had aside, and went to anger. Not that Adam was the type to yell, really. "I didn't think I really needed a reason to come by. I guess I should have thought otherwise." You could hear the bitterness in his voice, but he wasn't yelling. He was trying hard not to completely lose it.