I've been using Matrix/Element for around three years. I have there my family, a couple of friends and a couple of services that use it for alert, information. I host a non-federated server on a VPS in Germany. My approach has been simple: do you want to chat with me? Use Matrix. Otherwise, call me or send me emails.
If privacy is important for you, then make it a priority.
I've been using it for many years with my four mailboxes and I am very happy with it!