I recently managed to recover from a corrupted libstdc .so. Turns out I shouldn't have bothered because the it was a Pi and, of course, the SD card had shit the bed, but I was pretty happy with myself for like 30 minutes.

Accidentally put grub on the wrong partition on the device, which it was not happy with. Was able to copy some files over, manually boot the OS, and reconfigure grub to be in the right partition, took me about 2 hours? Then I did it again on a different machine, and speedran it lol

I managed a CentOS system where someone accidentally deleted everything from /usr, so no lib64, and no bin. I didn't have a way to get proper files at the time, so I hooked the drive up to my Arch system, made sure glibc matched, and copied yum and other tools from Arch.

Booted the system, reinstalled a whole lot of yum packages, and... the thing still worked.

That's almost equivalent to a reinstall, though. As a broke college student, I had a laptop with a loose drive, that would fall out very easily. I set it up to load a few crucial things into a ramdisk at boot, so that I could browse the web and take notes even if the drive was disconnected, and it would still load images and things. I could pull the cover off and push the drive back in place to save files, but doing that every time I had class got really tiring, so I wanted it to run a little like a live system.

Some of the crap I had to do back in the late 00s to get wifi, sleep and power management even barely working on some machines felt like the hardest thing at the time. I wonder how I’d fare with those issues today, 17 years later, knowing quite a bit more about the underlying OS and working with the OS daily… I don’t know that I’d qualify that as difficult more than it was extremely tedious and a bunch of trial and error of configuration options I didn’t know anything about.

If we’re talking about modern day… not so much honestly. btrfs snapshots saved my ass a couple of times, the rare issue I encounter I just rollback and wait for an upstream fix, and the rest I typically ignore or use something else. Everything tends to run quite smooth for me as a general rule, though.

Installed fedora on btrfs and upgraded from 38 to 39 week after installation, everything broke so bad, even ssd which was used for it locked, not just filesystem, ssd was new btw

Used to be messing with kernel arguments and installing/tweaking boot parameters. That was until Grub broke, I learned systemd-boot and chrooting into the system via live USB

Now if I break anything it's just a matter of "sigh, let me get the USB and type a few commands"

My first home server would get lost on the network every week, at different times and without any apparent reason. I performed hard resets by unplugging and plugging it back in.

After several months, I decided to connect a screen to it, and I initially thought it had hung up, but it hadn't. After some investigation, I discovered that every time my router obtained a new dynamic IP address, the server lost its network connection, requiring a reset. I wrote a script to check the network connection every minute, and if it's lost again, it will be reset.

Hmm I have come up with a bunch of neat solutions over the years. Where to start?

One time I broke the sudoers file on a distro without a root account, thoroughly locking myself out. I used docker -v /:/chroot to get myself root access to my root filesystem where I fixed the sudoers file. Protip always use visudo

I feel seen here, I was building a Ubuntu server and messed up the firewall settings not being able to get an internet connection, hours of trying to get back to where I was I gave up and plan to just start from scratch next time.

Is there a way of taking system snapshots with Linux?