I'm actually really good at memorizing things, I will admit to that. Poems and digits of pi and countries on a map, those sorts of things.
I tried drums too and it was really hard! I didn't have the coordination of having to do two different things at once, I very quickly realised. Just playing the beat with one hand and playing something else with the other? Beyond me! Weirdly it doesn't bother me as much with the piano, even though it's kinda the same deal. You should try piano - it's actually a lot easier than it seems and even if you're just making up things on it, you can't really ever play out of tune like most other things. The biggest problem of playing the piano is needing access to a piano!