303

submitted 2 months ago* (last edited 2 months ago) by Kris@feddit.org to c/fediverse@lemmy.world

105 comments fedilink hide all child comments

Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

(page 2) 50 comments

sorted by: hot top controversial new old

[-] match@pawb.social 7 points 2 months ago

Ty for the update! slrpnk is my favorite instance that isn't my home instance

[-] khaliso@lemmy.world 6 points 2 months ago

Thanks for the heads up! My Main account is in slrpnk, so I'll set a reminder in an month to switch back. :)

[-] tenchiken@lemmy.dbzer0.com 6 points 2 months ago

Sending positive vibes, and passing offer of technical assistance once you get access again.

Hopefully it's an easy fix once you get into things again.

[-] PM_ME_VINTAGE_30S@lemmy.sdf.org 4 points 2 months ago

That sucks. Thank you for your time and running the instance. Gonna miss you all like hell while you're gone. 😞

[-] originalucifer@moist.catsweat.com 4 points 2 months ago

people hate the big cloud solutions, but this is the kind of thing their HA infrastructure prevents against... hardware failures

i dont enjoy using (or paying for) aws, but i will never have a firewall or disk failure.

[-] drspod@lemmy.ml 7 points 2 months ago

Their HA infrastructure is all built on open source projects. The thing they have that we don't is teams of SREs on-call 24/7.

[-] viral.vegabond@piefed.social 3 points 2 months ago

You guys have always been excellent. As others have stated, no need to worry.

I'll be giving piefed some love in the meantime 👍

[-] grue@lemmy.world 3 points 2 months ago

I also can’t access it physically during the next few weeks and my usual back up that could restart it is not available either.

Sounds like your 'bus factor' is too high and you need a few more backup admins.

load more comments (13 replies)

[-] Brkdncr@lemmy.world 1 points 2 months ago

Opengear cellular backup appliance with serial/console connectivity to hardware.

[-] silence7@lemmy.world 1 points 1 month ago

Thanks for letting us know what is going on.

[-] Kris@feddit.org 0 points 1 month ago

Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.

[-] silence7@lemmy.world 1 points 1 month ago

Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.

[-] wit@lemmy.world 0 points 1 month ago* (last edited 1 month ago)

I am a slrpnk user. Oh well, shit happens. What makes me a bit sad was that a few months (?) ago something similar happened. Slrpnk.net was down for aa day or two? Don't remember the exact amount of time. What I do remember is asking if there was intention of adding a second admin and making the point that it would be good to create that sort of redundancy. I was ignored.

Mid of july is basically 1 and a half months from now. Hopefully a new solarpunk instance emerges, one that is safer from such scenarios.

Enjoy your vacation and don't stress too much! Shit happens.

[-] Kris@feddit.org 0 points 1 month ago

I am genuinely confused. What makes you think you were ignored? We do have additional admins, the problem is physical access to the servers to replace hardware that seems to have broken.

We plan to add additional safeguards to such scenarios, but in the end it is a trade-off between reliability and costs. Running a Lemmy instance of our size on a fully managed rented server would easily triple the monthly costs. This would be beyond what I am willing to cover out of my own private pocket, thus making the instance dependent on donations. And that IMHO makes it more likely to be shut down in the future as donations are very variable.

I am not on vacations, but on a work deployment abroad, which further limits the time I can spend on this. Not that it would have made much of a difference, but at least the domain redirection and placeholder text was only possible to do on the weekend.

[-] wit@lemmy.world 1 points 1 month ago* (last edited 1 month ago)

Well, I got no reply and I didn't see anyone else in the "mods:" bit in the bottom of the sidebar on slrpnk.net. I understand that having more admins wouldn't help in this specific case, but it would help in others.

Regardless, having a public lemmy instance only ran by 1 person is, in my opinion, a bad practice. And please know that I am not personally attacking you. I am just stating my thoughts. This time you were out on a work deployment. Next time you could be sick. The next time maybe a family member had an emergency and you had to help. And the next time maybe your house caught fire or whatever.

Resilience and redundancy is extremely important in public lemmy instances. And SolarPunk is, in my opinion, all about community. There should be more (trustworthy) people with the same level of access as you do.

Regarding the price and all that, maybe that is an interesting discussion to have with the slrpnk.net community? And it would also be good to let slrpnk.net users know how the instance is being ran, what hardware, where, etc, so users can be an educated decision on whether they want to see an instance or not? Maybe providing that info on the sidebar would be nice?

But again, shit happens. Hopefully this serves as a learning event. If not, so much shit is going through the world, losing a lemmy instance is not something about which one should lose sleep.

EDIT: I may be missremembering about the whole "something similar happened a few months ago and my question about new admins was ignored"... I think that might have been on mander.xyz and not slrpnk.net. I really can't fully remember, but I think it was mander.xyz after-all. Sorry! The rest of the post stands though!

[-] PirateFrog@lemmy.dbzer0.com 3 points 1 month ago

I see in your edit that you confused us for a different instance, but I want to mention regardless that Slrpnk.net has 3 active admins, it's not just poVoq (Kris).

[-] Kris@feddit.org 2 points 1 month ago

Regarding the price and all that, maybe that is an interesting discussion to have with the slrpnk.net community? And it would also be good to let slrpnk.net users know how the instance is being ran, what hardware, where, etc, so users can be an educated decision on whether they want to see an instance or not? Maybe providing that info on the sidebar would be nice?

We have been very transparent about that and documented that in the wiki which is linked in the sidebar. But it seems few people ever read the Wiki and having it integrated like it is with Piefed would probably help.

I agree that more admins with remote access would be good to have, and we have been working towards adding more (there are already more than one, who are not directly involved with slrpnk, but rather the mother project f-hub.org), but physical access will always be limited by the location (Azores) unless we move to an entirely different data-center (which in turn would make it hard for me to access the physical servers).

[-] wit@lemmy.world 1 points 1 month ago

We have been very transparent about that and documented that in the wiki which is linked in the sidebar. But it seems few people ever read the Wiki and having it integrated like it is with Piefed would probably help.

Oh ya. I knew about that, and I am fine with that. Though I think having it behind a link is bad UI. People will not read it haha.. It should be presented on account creation, in my opinion. Just 1 sentence "This instance is ran on private hardware and may suffer some down time here and there" would suffice.