199
2024-05-06 lemm.ee downtime (lemm.ee)
submitted 6 months ago* (last edited 6 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
11 comments fedilink hide all child comments

Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!

Technical details

Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

all 12 comments
sorted by: hot top controversial new old
[-] Kroxx@lemm.ee 31 points 6 months ago

Thanks for the update, even with that little downage .ee is insanely stable!

[-] notabot@lemm.ee 21 points 6 months ago

Thanks for the update. If you have the time and energy after the migration I'd be interested to hear details of the new setup, we have servers using vswitch too, and while I haven't run into trouble myself I'd like to avoid it if possible. You know what they say about the wise learning from others' experience!

[-] Evil_Shrubbery@lemm.ee 19 points 6 months ago

In the future please schedule your unplanned downtime around my (also unplanned) pooping schedule, thank you. Today's experience was an almost barely noticeable inconvenience!

(Ofc I kid, lemm.ee is awesome!)

[-] Vent@lemm.ee 14 points 6 months ago

Thanks for your hard work!

[-] curled@lemm.ee 13 points 6 months ago

Thanks for the update and your hard work!

[-] aStonedSanta@lemm.ee 7 points 6 months ago

You rock!

[-] Legend@lemmy.sdf.org 3 points 6 months ago* (last edited 6 months ago)

Goddammit he's minerals how many times do i have to say this ?

[-] Zatore@lemm.ee 5 points 6 months ago

Thanks for telling us what happened! I know there's nothing we can do to help, but I'm always curious to know what happened. It can also be a great learning experience.

[-] casmael@lemm.ee 3 points 6 months ago

Thanks for the update - seems to be working for me now 🫡

[-] Legend@lemmy.sdf.org 3 points 6 months ago

Even with all these your instance seems to be one of the best ones out there lol nice work.

[-] gregorum@lemm.ee 2 points 6 months ago

FYI, image upload are broken again :(

this post was submitted on 06 May 2024
199 points (98.1% liked)

Meta (lemm.ee)

3473 readers
1 users here now

lemm.ee Meta

This is a community for discussion about this particular Lemmy instance.

News and updates about lemm.ee will be posted here, so if that's something that interests you, make sure to subscribe!

Rules:

If you're a Discord user, you can also join our Discord server: https://discord.gg/XM9nZwUn9K

Discord is only a back-up channel, !meta@lemm.ee will always be the main place for lemm.ee communications.

If you need help with anything, please post in !support instead.

founded 1 year ago
MODERATORS
sunaurus@lemm.ee