Because Dalish have been disliked since winning our freedom after the fall of Halemshiral. We fought against our oppressors, and strive to reclaim our past.
Of course, it's probably the vallislin that most people hate, since those elves still serving in the city do not bear those marks.