While it may be classist in some situations, it is indeed racist (even if unintentionally) when the recipient is a black person. You gotta understand for a large portion of American history, black people were considered 3/5 of a person. So saying you're "well-spoken" to a black person just isn't a dig at their socio-economic background, it's a back-handed commentary on them as a human being. It's basically saying, it's impressive that this primitive form of a human being is able to string together coherent sentences! Kudos! And it falls in line with the old myth that black people have extra bones to further the trope of the athletically superior, intellectually inferior black sub-human.
So the whole "well-spoken" thing cuts a lot deeper than classism when it's directed at black people in America. And while certain white people in America with thick regional accents may get ostracized from time to time. The 99% of white people who just speak the regular Queen's English don't get singled out left and right for being "well-spoken." Whereas literally EVERY black person I know who speaks regular Queen's English, repeatedly encounters the same situation I do regardless of their class. So this is NOT a color blind issue.