Just because no one has been hurt physically doesn't mean no one's going to be angry with you two. The only difference is you're the nice one, so I'm not surprised he's the one who got hit first.
But you may be next if you keep it up. You forget the people you're dealing with here when you do things like that. All of these people aren't your friends, they aren't your servants, they aren't people who have to put up with how you act.