You shouldn't advocate for vigilante assaults for your own safety. Punching people for their opinion is immoral because it creates a situation where society can't function properly, which causes overall harm.
So if you want me to change your opinion, it's very simple, whatever someone's opinion, assaulting them extralegally MUST be immoral, as doing so breaks down trust in the safety that society provides, and by extension, creates further harm.
Look at countries where people can get away with vigilante assault and murder, would you rather live in a country with that kind of societal structure?
And if you believe punching a nazi for reasons other than self-defense is ok you've established a principle that violence is acceptable against persons of certain creeds. White supremacists and Nazis are acceptable moral targets for violence.
So the next guy comes along. And he's not like you. He's mad. And he knows he can commit violence against these people. So he does, and he ends up killing somebody. And maybe that's condemned, but really, all those other Nazis he punched, that was justified. And then his buddies take it a step further. And this isn't speculation - this is how it's worked every time before.
And what about when the people who you don't like come around to hit you? "But my ideology is correct!" They don't believe that and they don't care - all they know is that now, violence is proper and acceptable. And they graduate from low-level to high-level and it's exactly the same thing.
The principle here is key. If you don't hold a principle true in every case, you don't hold that principle. And if society lets that principle fade, then it's not a protection for anybody, and that's a hell of a lot worse than letting Nazis mouth off, because now they can do the things we actually consider them evil for.
I want racism and nazism gone as much as the next person. The problem is that I want it to actually go away. Violence, in that context, must be nonexistent or total. Either you convince them peacefully or you murder them all. Anything in between creates martyrs and gives them reason to dig in and resist (see: early Christianity, modern terrorism, etc.).
And even if that were wrong, I wouldn't be willing to implement violence. It violates key principles, and is contrary to a functional civilization. It has never ended well, in any case throughout history.