Which is why it is able to survive as a symbol of ill-intent. If it was as obvious as the nazi-salute, it would not be as effective.
But to counter, you are placing a lot of trust on the people around you, which let me be clear, is a good thing. You seem to be assuming that most people are good, and kind, and generally well-meaning.
However, it is not the experience everyone is going to have had with society. As a result, it is not the assumption everyone is going to make, and that difference allows those with ill-intent to drive a tiny wedge between those impacted by that hatred and those who more easily see that positive interpretation.