The issue is that wizards can instantly wage a guerilla war. All it takes is Muggle Studies being taken seriously and integration in muggle society. Then it's impossible to identify them. Wandless magic could be a focus to be certain you're never carrying an identifying marker.

Then the wizards don't need to attack. They can destabilize the economy by duplicating money, precious resources, etc. These will deteriorate eventually but cause chaos and devalue the economy for a time. Imagine taking a $100 bill and just duplicating it to prop up your car-wash business as a front. But that's baby-level chicanery.

The real sauce is the mind-control. Imperio (who cares if it's unforgivable when you're at war?) and more importantly the Memory Charm and False Memory spells. Let's just get in there, stun a very important person (start low, work up) and insert the memories you need to get them to act accordingly. Remove the memories of that transaction and boom! Access to the most powerful people in the world and the ability to influence them. Now a wizard has their finger (remotely) on the nuclear button!

Seriously. Voldemort was an idiot. He used basic terror tactics for years and the only reason he wasn't killed quickly was because Magical Britian was written to be as stupid as possible because Rowling was writing a children's story about incompetent leaders.