This will be a good lesson in how difficult it is to setup servers with high availability.

I'd suggest getting redundancy working on your own network first before distributing it. How do you plan to handle storage? Will that be redundant as well?

Tailscale might be the best bet at this point. It will manage the wireguard mesh for you, and use nat holepunching for handshaking instead of needing listening ports.

You might just run it in the cloud. A DR failover process is a bit more complex than most home server environments, and is rarely automatic or smooth.

Tailscale. Use Headscale if you prefer self hosting over using their cloud service.

What services are you running? You don't want to cluster with multiple locations as the latency will kill you and it will eat bandwidth like crazy.

Maybe use something like syncthing to sync data directories across. I would use Netbird to connect them and the. Ansible for administration.

The comments about storage still stand, that's a tricky thing to do over the internet.

But I would recommend a meshing VPN, tinc can mesh and will route round network problems.

Also don't forget about backups, as that could be a better thing to have off site

You could use HAProxy on the client side to load balance apps in multiple locations, but it really depends on the application.

I like to manage my software with Ansible but Docker stack files might make it simple enough for you.