[Even though Subaru's sadness is evident, almost tangible, it doesn't make him feel compassion or empathy - instead, he feels happy, pleased with his words, with knowing that he's special, that no others have been this close to Subaru]
[... You're pretty even when you're sad ♥] It makes sense.