He doesn't have that many enemies who can go one on one with him, though, so I don't mind there being a few who can. Of course, those that can, like Harley and Bane, seem to end up converting to anti-heroes. There's still Hush. That's why I don't really mind that Hush can give Bats a fight, even though they've never explained how he got those skills.
Back in the '70s, the Joker used to be able to give Batman a run for his money in the fighting department. I guess he's just gotten out of shape over the years. :)