Of course he may not be entirely wrong. Snape's punishments aren't the human rights violations you might suppose from his harshest critics, but it's easy to read him as unfair to Harry when we don't know if he's equally hard on everyone else. For instance, if any student who was reading in class would get humiliated like Harry in GoF, well and good. But if Snape shows that he enjoys catching out Potter in particular and then smacks him down harder than he would some random Gryffindor, that's not cool. Not even if Harry is one of his most disruptive students.
We are never shown anyone else getting caught doing the things Harry gets punished for. We don't know if Severus would have punished another Gryffindor or another Slytherin the same way. We do see in COS Slytherins getting away with tossing ingredients (and Harry thinks how unfair this is) - but in the same lesson Harry gets away with tossing a firecracker into a cauldron and actually causing an explosion and splashing students! Harry got away despite believing Severus knew it was him. Also, at least once we see Slytherins reserving misbehavior to when they are clearly out of his line of sight. So either the Slytherins expected to be punished if caught or they didn't want to embarrass Severus in a situation where letting them off would be too obviously unfair.
We are never shown anyone else getting caught doing the things Harry gets punished for. We don't know if Severus would have punished another Gryffindor or another Slytherin the same way. We do see in COS Slytherins getting away with tossing ingredients (and Harry thinks how unfair this is) - but in the same lesson Harry gets away with tossing a firecracker into a cauldron and actually causing an explosion and splashing students! Harry got away despite believing Severus knew it was him. Also, at least once we see Slytherins reserving misbehavior to when they are clearly out of his line of sight. So either the Slytherins expected to be punished if caught or they didn't want to embarrass Severus in a situation where letting them off would be too obviously unfair.