This was likely a state-sponsored attack. If your SSH isn't exposed to the internet this probably wouldn't have effected you. Also most people run stable distros like Debian on their server, and this particular vulnerability never made it to the stable branch. I would guess that most of the computers you have ever used have backdoors. Even if you run Linux (which may itself have some) you might still have a proprietary UEFI on your motherboard. Something like xz wouldn't effect you because no government really cares what's on your server. Smaller attacks can be avoided through common sense. Some people will expose services that require zero authentication to the internet. Follow basic best practices and you will probably be fine.

Edit: Also remember, google photos once flagged a picture of a child that the child's father had taken for medical reasons as abuse, so self-hosted may not be completely private or secure, but it's better than the alternative.