Rodolphus could not believe his brother actually thought he was going to undress him. Rodolphus was trying to pull Rabastan away so he did not have to undress. He actually felt bad his little brother did not understand his intention. Had he hexed Lucius he would have most assuredly not only been kicked off the team but more than likely kicked out of Slytherin. Not to mention hexing him was a bit heavy handed.
When Lucius offered Rabastan the bottle again Rodolphus spoke softly to his brother, as he was still standing next to him. "You don't have to drink or undress any more Rabastan. Just go find your robes" Sure he was crossing Lucius, but it was his little brother, and he saw how uncomfortable he was. Rodolphus despised making his brother go through this. Rabastan had made the team on his own merits and sure he should be treated like the rest of the boys, but he wasn't like them. He was Rabastan Lestrange, little brother to Rodolphus Lestrange, seventh year member of the team who was nearly team captain.