Just wondering what tools and techniques people are using to keep on top of updates, particularly security-related updates, for their self-hosting fleet.
I'm not talking about docker containers - that's relatively easy. I have Watchtower pull (not update) latest images once per week. My Saturday mornings are usually spent combing through Portainer and hitting the recreate button for those containers with updated images. After checking the service is good, I manually delete the old images.
But, I don't have a centralised, automated solution for all my Linux hosts. I have a few RasPis and a bunch of LXCs on a pair of Proxmox nodes, all running their respective variation of Debian.
Not a lot of this stuff is exposed direct to the internet - less than a handful of services, with the rest only accessible over Wireguard. I'm also running OPNsense with IPS enabled, so this problem isn't exactly keeping me up at night right now. But, as we all know, security is about layers.
Some time ago, on one of my RasPis, I did setup Unattended Upgrades and it works OK, but there was a little bit of work involved in getting it setup just right. I don't relish the idea of doing that another 40 or so times for the rest of my fleet.
I also don't want all of those hosts grabbing updates at around the same time, smashing my internet link (yes, I could randomise the cron job within a time range, but I'd rather not have to).
I have a fledgling Ansible setup that I'm just starting to wrap my head around. Is that the answer? Is there something better?
Would love to hear how others are dealing with this.
Cheers!
A few simple rules make it quite simple for me:
This has been working great for me for the past several months.
For containers, I rely on Podman auto update and systemd. Actually my own script that imitates its behavior because I had issues with Podman pulling images which were not new, but which nevertheless triggered restarts of the containers. However, I lock the major version number manually and check and update major versions manually. Major version updates stung me too much in the past when I'd update them after a long break.
Yep, this is what I was thinking I'd have to do. So, from your perspective, Unattended Updates is still the best way to achieve this on Debian, with the right config? Cheers.
Correct. And getting the right configuration is pretty easy. Debian has good defaults. The only changes I make are configuring it to send emails to me when updates are installed. These emails will also then tell you if you need to reboot in subject line which is very convenient. As I said I also blacklist kernel updates on the server that uses ZFS as recompiling the modules causes inconsistencies between kernel and user space until a reboot. If you set up emails, you will also know when these updates are ready to be installed because you'll be notified that they're being held van.
So yea, I strongly recommend unattended-upgrades with email configured.
Edit: you can also make it reboot itself if you want to. Might be worth it on devices that don't run anything very important and that can handle downtime.
Yep, cool. The single host I have with UU running on it does send the listchanges via email already, which I've found useful.
Well, time to refresh my memory on how I have it setup and build up an Ansible playbook to repeat success everywhere else.
Cheers.