This is the approach I use, not sure if it'll work for your use case but I can assure you it works for at least a few users. It's all sort of manual set up but from your comments it sounds like you're just doing this for friends and family and not on an enterprise level. I admire your efforts!

First off, I have a purelymail account on which I set up domains and accounts for each user. I have mine set up so user1@anydomain.com all goes to the user1 mailbox (and user2@anydomain.com goes to the user2 inbox regardless of domain, etc.) but you can set up some pretty complex routing if you want - and if you know a bit of sieve there's even that. Purelymail handles the actual email sending/receiving so I'm putting a lot of trust in them, but it seems like they have a good track record and I don't think I could do better on my own. Plus they're dirt cheap. My big concern with email is always deliverability. Anyway, you'll see this is all set up in such a way that I'm using purelymail now, but I'm not tied down to them.

Second, I use this image (linking to the repo and not the docker hub version so you can inspect the Docker file for opsec reasons. In my set up I build it from source because I have a couple modifications) which is a dovecot IMAP server + getmail. This is python getmail not go-getmail and not fetchmail. The repo itself has some pretty straightforward instructions but the way it works is basically that users inside the docker container each map to a mail directory. So each user's credentials is actually a Linux username and password within the container. I have mine set up so it's like user1, user2, etc. (which confused my users initially because automatic set up forms are never set up this way) but you could set it up however you need. Then, there's a Cron set up to run getmail which you have to configure yourself within a cron.d folder that you mount on the container. For mine I have it configured to use POP3 so that when it gets stuff off purelymail it's automatically deleted.

Finally, you just set up your mail clients to use this IMAP server and purelymail's SMTP but if you know how to set up a forwarder you can always have it relay through purelymail. Purelymail even has the ability to relay emails to your SMTP server.