Id be pretty wary of using any system that "cooked" an nvme. That not the sign of an actual healthy system.

Was the failure just heat damage?

I wouldn't do this. If you are spending the time to do netboot you might as well get a proper boot drive.

Kind of, but probably not. I started writing this and was like "totally it could be stateless". Docker runs stateless, and I believe when it starts it is still stateless (or at least could be mounted on a ramdrive) - but then I started thinking, and what about the images? Have to be downloaded and ran somewhere, and that's going to eat ram quickly. So I amend to you don't need it to be stateful, you could have an image like you talked about that is loaded every time (that's essentially what kubernetes does), but you will still need space somewhere as scratch drive. A place docker will places images and temporary file systems while it's running.

For state, check out docker's volume backings here: https://docs.docker.com/engine/storage/volumes/. You could use nfs to another server as an example for your volumes. Your volumes would never need to be on your "app server", but instead could be loaded via nfs from your storage server.

This is all nearing into kubernetes territory though. If you're thinking about netboot and automatically starting containers, and handling stateless volumes and storing volumes in a way that are synced with a storage server.... it might be time for kubernetes.

You can host docker volumes over NFS, but the actual container images need to exist on a filesystem that supports overlay (which NFS does not) unless you want things to be slow as shit. And I really do mean miserably slow. A container image shared over NFS will take forever to spin up because it has to duplicate the entire container filesystem instead of using overlays, and then it'll blow up your disk usage by copying all these files around instead of overlaying them. It's truly unusable.

I've done something extremely similar with a custom NixOS iso for my docker VMs to make versioning and backups easier (golden image live disk with SSH+Docker+Dockge shared between all VMs + local persistent storage specific to each VM).

You can configure frigate via OCI container with custom config, as well as NFS mounts, SSH server, etc and then have a read-only live disk that boots up, mounts NFS share, and then starts up frigate.

You mention frigate specifically. Were you running this on the system when the drive failed, or is this a future endeavour?

I bring this up because I also use frigate, and for some time I was running with a misconfigured docker compose that drove my SSD wearout to 40% in a matter of months.

Make sure that the tmpfs is configured per the frigate documentation and example config. If misconfigured like mine was, all of that IO is on disk. I believe the ramdisk is used for temp storage of camera streams, until an event occurs and the corresponding clip is committed to disk.

Good luck!

It's possible, but I recommend more RAM because you won't have any swap.

But adding a network boot is a hassle, I recommend buying another drive, it'll be cheaper than RAM esp since you don't need much storage.

I do it with k3s right now on fedora. I like it personally.

Nice thing if you use k8s settings up persistent net storage with something like longhorn is an option too.

Run a livecd of whatever Linux distro you like, and get a USB drive. Store persistent files on the USB stick.