I built a systemd-sysext with nmap, screen, iperf etc based on https://github.com/fedora-sysexts/fedora

I should add rhat it is very easy to extend the immutable os usong a short dockerfile, if your tooling is so important that you want it in the immutable layers.

Distrobox or toolbx are the canonical dev environment approach (persistant containers) but bluefin also comes with linuxbrew for host level tooling. I actually use mise (mise en place) instead.

Use homebrew for Linux.

Isn't the purpose of an immutable OS supposed to be for things like specific services that generally aren't supposed to be logged into? For example a web-proxy, or log-forwarder or maybe some kind of LB front-end?

I didn't think "daily driving" an immutable OS as a user who needs to invoke a shell was its purpose.

Distrobox is pretty shrimple

Install them in your $HOME

For example $HOME/MY_CLI_TOOLS

Add it to your $PATH

brew install btop vim

https://docs.projectbluefin.io/command-line/#installing-applications

I think you're supposed to use brew on uBlue.

Full caveat - not personally into immutable, 90% of the time I'm in Debian or a derivative. 9% arch or derivative. 1% work requirements made me have to use something else.

So I'm less making a rec on method and more commenting on this:

I've read that containers are preferred for development, but they aren't persistent

They absolutely can be, thats the point of mounting volumes. I don't want to do the same thing more than once, so whether I'm playing with something stupid at home or I'm doing something critical at work, I'm going to make a spot for any and all changes I might want to make to use it again elsewhere, without much effort. That could mean mapping a directory to a volume, setting specific variables in my compose/kompose, having a container grab data from elsewhere every time it starts, or whatever, but the parts I want persistent are, the parts I want variable are.

Keeping whether or not containers are the "right" way on an immutable distro aside, what isn't persistent for you that should be?

I switch back and forth between bazzite and bluefin quite often

on these and other immutable distributions, /usr is read-only, and the recommended is to use installation methods that write to your HOME (or to /var which is where docker and flatpak --system save files)

i really should muck about with container-based development flows

my current preference is flatpak, then whatever per-language package tooling (e.g. cargo for tools written in Rust, npm with a custom HOME prefix for tools written in Node.js, uv for Python projects, etc) when there's no flatpak, then homebrew, then rpm-ostree as a last resort

for editing files in /etc my recommendation would be to set the EDITOR environment variable to point at whatever you like, installed however you like, and edit with sudoedit /etc/fstab, because then your editor is not running with root permissions

you could also point EDITOR at a custom script that mounts the target file into a container running your desired editor

Layer things that genuinely need (often in boot sequence) low level access like filesystems (e.g. I have mergerfs+snapraid on my desktop). If you're OK with a longer rpm-ostree update, you can layer some self contained things like btop with little risk, perhaps also your preferred shell. Also anything you want in TTYs if something breaks.

vim edit /etc/fstab works fine from within a distrobox, you just need to do sudo vim /run/host/etc/fstab or distrobox-export the binary to your main shell, which means that the container will start, but you don't have to enter it. If you fire a terminal profile into the container by default at login you won't need to start the container when you use an exported command.

Embrace the distrobox experience for development and generally mucking around, use Arch's AUR, archive entire environments, there's lots going for it.

Linux brew is coming along nicely, use it first if there's a formula, but I've been fine with flatpak, distrobox and layers (in that order) for a couple of years now.

There's a way to create aliases to the programs in the containers so that you can run them on the host as if they were installed. Look into the Boxy app (should be able to find it as a flatpak) for a GUI way to do it, but you might also look into nix to set up different dev environments. If it's not already a ujust recipe, look into how you can layer it and how to load up different nix configs for your different environments.

That's the neat part - You don't! (unless you want incredibly long update times as every new util is a new overlay!)