I'm friends with people who happen to be friends with people I avidly dislike. I've argued with my friends, we disagree on some things, agree on others.
I let my opinions form not because of what my friends think, but because of what I do. My opinion on you is not based on Lydia. But I feel yours of me is.
My opinion of you is based on what I've seen. I won't deny you're brave. I won't deny you'd defend this place and the people here. But personally, you are very quick to judge those who don't conform to your view of the world.
Of course Lydia does that too which explains why you get along so well.