Some expirence on some self-hosted VOIP solutions from my EvE online days and I self-host a Teamspeak instance (my nerds like it, get off my lawn).

Mumble in terms of its UI and user expirence, the worst of the major VOIP projects (looks very 2008), however it is by far the best in terms of server stability, plugin compatibility and security. To quote my old EvE admin "Mumble will take the team two weeks to set up correctly, and drive them mad, but once thats done they will not need to touch the config again". Plus it not requiring a license allows large orgs to use it freely. Ever have a need 2.5k+ VOIP users all trying to talk over eachother? Mumble is the only free application that will handle that without issue.

Teamspeak3 is what I run, and for small communities its perfect. TS5 exists, and the devs keep trying to make "We have Discord at home" and its just a UI fork, they all run the same server backend. As for features, TS3 has the best of ease of set up and granular permissions with API tools to allow for remote or automated managment. For user counts, anything beyond that of a small guild in any game will require a license, they are cheap (I just renewed my 30$ a year license and didnt have to reboot). Its drawbacks are that it struggles after several hundred users (its heavier on server hardware than mumble is) and user accounts with permissions can break the server. Fortunatly settings are managed by a local database so backups of server state and files are easy.

I remember Ventrilo existing, thats about it.

Hardware wise, a new pi should be fine, older models might have issues based on expected user load. Network load is not significant for normal hobbiest user counts, security is not any different than normal homelab internet services.

Let me know if there is anything I can help with.