The idea is "don't tolerate intolerance." Nazis are definitely intolerant, so it's ok to not tolerate them. Since Nazis promote death squads and gas chambers for people they don't like, punching them is very justified.

There are also some literal Nazis in the United States that are pro-Trump. They are also very punchable.

Does that mean that all trump supporters are punchable? No, it does not. But should it give you pause if you are on the same team as the Nazis? Yes, it should.