165

Lemmy.world updated to 0.18.1-rc (lemmy.world)

submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

29 comments fedilink hide all child comments

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

all 32 comments

sorted by: hot top controversial new old

[-] Tontoz@lemmy.world 10 points 1 year ago

Really like seeing posts like this that explain what's going on. Thanks for all the hard work!

[-] ayyndrew@lemmy.world 0 points 1 year ago

The transparency is so refreshing

[-] gfdoto@lemmy.world 0 points 1 year ago

Seeing this post immediately made my ditch my neglected Lemmy instance to sign up for an account on this server. Thanks for the hard work.

[-] Sneckster@lemmy.world 3 points 1 year ago

This post encapsulates what I love about this place.

[-] deweydecibel@lemmy.world 1 points 1 year ago

How so?

Admin communicating information about the site to users like genuine human beings, instead of the corporate-sanitized pablum reddit admins speak in? Yeah that's refreshing.

[-] Dogs_cant_look_up@lemmy.world 2 points 1 year ago

So no more random sports posts at the top of every thread? That was a weird bug.

[-] marsokod@lemmy.world 2 points 1 year ago

Thanks a lot for your hard work, take your time with your children

[-] free@lemmy.world 2 points 1 year ago

U peeps are awesome. 🙏🙇

[-] tmpod@lemmy.pt 1 points 1 year ago

Awesome work!

I'd like to know more about the exact container topology you have, since I may try something similar on my instance as well.
Is it something like this?

┌───┐       ┌───┐
│WEB│       │WEB│
└─┬─┘       └─┬─┘
┌─┴─┐ ┌───┐ ┌─┴─┐
│BE ├─┤IMG├─┤BE │
└─┬─┘ ├───┤ └─┬─┘
  └───┤DB ├───┘  
      └───┘

Thank you! :3

[-] ruud@lemmy.world 1 points 1 year ago

Yeah with nginx doing load balancing

[-] flamekhan@lemmy.world 1 points 1 year ago

Huge thanks to the lemmy.world team over the last couple of days to scale and maintain the instance! There's a link for donating on the sidebar for lemmy.world - just a couple bucks a month can help us support this instance!

[-] xikufrancesc@lemmy.world 1 points 1 year ago

How beautiful and cute the new lemmy world looks. thanks boss.

Que bonito y lindo se ve el nuevo lemmy world. gracias jefe.

[-] BitOneZero@lemmy.world 1 points 1 year ago

Edit still see some performance issues. Needs more troubleshooting

Federation overheard is putting a lot of load on servers. Creating one task for every single post, comment, and vote in RAM-only queue.... pending changes: https://github.com/LemmyNet/lemmy/pull/3466

[-] rustytree@lemmy.world 1 points 1 year ago

Nice. I can use jerboa now.

[-] F3F3N@lemmy.world 1 points 1 year ago

A bit off topic, but does anyone else hate how when you click on a post and then go back, the page auto-resets to the top? Wish it would remember how far down you scrolled and return to that point.

[-] minimar@lemmy.world 0 points 1 year ago

I've always opened in a new tab (middle/ctrl click)

[-] guyman@lemmy.world 0 points 1 year ago

How are you doing that in the app (Jerboa)?

[-] minimar@lemmy.world -1 points 1 year ago

Not

[-] minimar@lemmy.world -1 points 1 year ago

Not

[-] Secret300@lemmy.world 1 points 1 year ago

we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Had no idea nginx did that. There's so much I need to learn

[-] Magister@lemmy.world 1 points 1 year ago* (last edited 1 year ago)

I cannot connect with Jerboa, it always says user or password incorrect 🙁

Edit: worked after a few minutes

[-] MyOpinion@lemmy.world 0 points 1 year ago

It has been near unusable since the upgrade.

[-] infotainment@lemmy.world 1 points 1 year ago

It seemed to work fine at first, but starting today I'm noticing a huge number of async failures; I've had to refresh the page in order to get almost anything (comments, posts, communities, list of communities, profiles, etc) to load.