The best "bang for the buck" in your use-case is to use Nextcloud - Nextcloud Talk is your Jitsi replacement, and the files feature can be extended with the Nextcloud Photos plugin (https://github.com/nextcloud/photos).

As for your domain question:

1. You should use any computer you'd like that meets the Nextcloud recommendations, the key is of course isolating this machine on your home network so any "funny business" stays on the server. You can do this with VLANs or an entirely separate LAN connected to a different WAN (ISP).

2. Many places, I like porkbun.com for real custom domains for cheap, but for your use case, you might be able to use a Dynamic DNS provider for free. It just likely won't be an easy to remember URL (or at least, as easy as a root domain only). If you have a newer ASUS or Netgear router/modem they both have Dynamic DNS built in and you can select from a few different providers that have both free and paid tiers. ALSO it might be better to use Google Domains (now squarespace domains) since, IIRC, many DynDNS configs for routers support Google Domains too. Cloudflare can also be a decent registrar, and I'd recommend using them if you use any other cloudflare services (see below).

3. Other things to consider: Your ISP may block port 80, meaning lots of issues. If this is the case, you might want to use a tunnel of some sort. Cloudflare has a great solution here. Even if they don't block port 80, they may aggressively throttle and shape your incoming traffic - causing issues. Again, the tunnel is a good solution here. And, of course, your upload bandwidth matters a lot, you'll need something around 100Mbps upload for a decent experience when accessing your stuff over the internet. The 30Mbps that's typical of DOCSIS modems won't cut it. Outside of these concerns it's all about making sure you isolate your server from your "home stuff" to keep things secure.