Then use a different term for the wide swaths of corrosive folks. Having people you find corrosive mixed in with actual NAZIs waters down he term completely
Unfortunately, it is an acknowledgment of reality. Movements tend to work in terms of coalitions, and the alt-right is not an exception to that. Navigating around the alt-right requires more than questioning "okay, who here is an out-and-out white supremacist?"; it also requires looking at how people like Steve Bannon and Milo Yiannopoulos coopted it, how things like GamerGate overlapped with it, how fascists and "free speech" activists use each other, and so on.
This, from Vox, is just scratching the surface:
The label blends together straight-up white supremacists, nationalists who think conservatives have sold out to globalization, and nativists who fear immigration will spur civil disarray. But at its core are the ideas of a movement known as neoreaction, and neoreaction (NRx for short) is a rejection of democracy.
So I'm not saying to simply throw every harmful person into the alt-right mix. I'm saying these groups are directly intertwined and amplifying each other. You could probably convince me that, on balance, we should be more cautious with how we use the term alt-right; however, I suspect there's an element of "the best way to understand x is to look at it through a very narrow lens and otherwise you're watering it down" going on there, and that's rarely helpful to anyone, whether x refers to the term alt-right, the term Nazi, or the term racist.