Yes, a lot of people have said that. But the infidelity in my mind was Harry cheating on Snape with Draco, which is not really cheating, because, as Snape notes, he is a PORTRAIT!, but there is definitely a betrayal on Harry's part. In my mind (although I don't spell this out, because dammit the fic was long enough), this betrayal is partly why he is able to make a pact with himself. *Harry* believes he betrayed Snape and he wishes to atone. The fact that Snape was a portrait was immaterial in Harry's eyes.