Well, there are some born with magic, and some who aren't. We tend to keep magic hidden from those who can't use it, they don't know it exists. Or at least they don't believe it does.
A bit like a staff, a lot smaller though. No one really uses wands in your world? You may not need to use one to properly control your magic.