So like zoom but open source? Jitsi is the first thing that comes to mind.
You can try out Peer Calls. It's a peer-to-peer video and text chat that uses WebRTC, and is very lightweight. You can either use the flagship instance or self-host your own. One of the disadvantages is that text messages are not stored and are lost when the chat room is closed.
IRC could work as a backend for text chat, Twitch based their chat system on that and heavily modified things. Jitsi, or Jami support video out of the box. Could probably also roll your own with XMPP and WebRTC
As much as I love IRC, it's a terribly insecure platform and I'm trying to avoid rolling my own if I can help it, but yeah WebRTC support would be necessary. Do Jitsi support text now, it's always appeared video first to me?
I've used nextcloud for this but it's not great. I'm sure there are better alternatives.
Have you looked into Matrix?
Matrix
It seems extremely convulated and offers far more features than just video-chat?
All about open source! Feel free to ask questions, and share news, and interesting stuff!
Community icon from opensource.org, but we are not affiliated with them.