If you don't have a dragonfire shield, you're toast before you can even get a hit in. And they're incredibly strong even then. So you can't just wander up without bringing your armor and plenty of food. And since a lot of them are in the Wilderness you need some sort of escape plan in case a Revenant shows up.