I think the only times he is physical are when Harry comes out of the Pensieve-he is gripping Harry's arm hard enough to bruise, shakes him, threw him from him. Then if you want to say their duel in HBP (which I wouldn't count myself) when Snape hits him with a whip-like spell and he feels the air has been knocked out of him. But I too don't care for Snape being physically violent. I think he uses words as weapons. So, yeah I agree with you.