Nothing stirs up tension on the internet like two non-racists arguing over how to not be racist. Imagine if an ACTUAL racist had shown up!
I think part of the problem is that many people, especially those of privilege, assume that racism consists only of stating outright "I hate [minority group X]," advocating or committing violence against minorities, belonging to a white supremacist organization, or explicitly saying "You're fired/not hired in the first place, because you're a [minority group X]." While those actions are the most obvious and dangerous forms of racism, they are far from the only, or even most common, form. A statement doesn't have to include a blatant ethnic-slur word to be offensive, and cause harm, to a person of said ethnicity. An action doesn't have to be a legally codified criminal or civil offence in order to be offensive, and cause harm, to a member of a minority group.
Also, it seems to me that both those of privilege and members of minority groups often confuse "racist" as adjective with "racist" as noun, i.e. as person. A person who runs a "whites only" country club, or works/volunteers for the Aryan Nations organization, or can't hold a single conversation without making ethnic slurs...yeah, it's accurate to call such a person "a racist."
In contrast, a person who, upon being introduced to a professor of colour (without being informed of their field), says "So, you're an African-American Studies professor then?", has said something racially offensive but may not be (and, nowadays, likely is not) necessarily "a racist" through-and-through. Nevertheless, the fact that said person doesn't go around wearing a swastika, never uses the "n-"word and would never intentionally and maliciously do something discriminatory, does not excuse that person having verbalized the assumption that a professor of colour must necessarily be in the African-American Studies field, and that person should still be called on it, in a civil manner.
I'll give a further example related to my own religio-ethnic group. Although, as a Jew, I'm offended and hurt when I read about Mahmoud Ahmadinejad claiming that the Holocaust is a Jewish-perpetrated hoax, or Mel Gibson telling a police officer that "Jews are responsible for all the wars in the world." Nevertheless, I know to distinguish between those two, er, gentlemen and someone on an online comics forum who assumes that Veronica Lodge is Jewish, presumably because of her wealth and materialistic values. I don't automatically assume that said forum member is an anti-Semite who wants me segregated from the non-Jewish majority or dead. For all I know, said person could have Jewish friends or relatives, have wept over a Fiddler on the Roof performance, and such. But that person has still voiced an assumption (if wealthy, spoiled and materialistic, then Jewish) that I find offensive, and I'm still going to (and have) politely point out that error.