Man, I had to look up "prestidigator". Was magician really that hard to say? ;P (Even though "magic shows" obviously should have just clued me in, duh)
True, more diversity would be fun. And if you think about it, they should use all of it. He'd be dangerous if you get close to him, cus like you say he could have been unstoppable with his hands and could maybe stab the hell out of anyone without them seeing the weapon coming. Then again, Taskmaster seems like the guy who wants to brag and want to have a bit more distance, say maybe the distance of a well placed leg, to the other person.
Anyway I could just imagine him going "I picked this bit up from Bruce Lee. And I learned this from So You Think You Can Dance!" He should be one hell of an unpredictable foe, as long as he mixes it up.