Dexter actually finds Dark Arts a lot more useful than Muggle Studies so he's willing to put in the work for it. Also he's been pretty bitter this last week, which may have helped him perform the spell a little better.
Once he got the hang of the spell he had no trouble with the bear's hands and feet, but somehow he had a sort of mental block about hitting its face. He kept trying but never quite got it to connect. It's just as well he didn't get to try with the big bear, really.