827
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

(page 3) 50 comments
sorted by: hot top controversial new old
[-] Pillarist@lemmy.world 2 points 1 year ago

I had an issue of being logged out of my account and could not log back in, after closing and reopening the site, closing browser, etc until I cleared my cookies, then it let me back in. If that helps anyone.

[-] MarekKnapek@lemmy.world 2 points 1 year ago

So what happened:

  • Someone posted a post.
  • The post contained some instruction to display custom emoji.
  • So far so good.
  • There is a bug in JavaScript (TypeScript) that runs on client's machine (arbitrary code execution?).
  • The attacker leveraged the bug to grab victim's JWT (cookie) when the victim visited the page with that post.
  • The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.

Am I right?

I'm old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

  • User provided content (post using custom emojis) caused havoc when processing (doesn't matter if on server or on client). This is lack of sanitization of user-provided-data.
  • JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
  • How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
  • The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Am I right? Correct me if I'm wrong.

Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.

Marek.

load more comments (1 replies)
[-] Rooki@lemmy.world 2 points 1 year ago* (last edited 1 year ago)

Is there a rough time range when it happened? and any news about other big instances like lemmy.ml? Are those safe? Currently they are not on the same version as lemmy.world.

[-] Ahmed@lemmy.world 2 points 1 year ago

Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.

[-] Molecular0079@lemmy.world 2 points 1 year ago

Thanks for the post-mortem and the quick fix! Glad you guys around to help battle test Lemmy's code.

[-] padjakkels@lemmy.world 2 points 1 year ago

Thanks for the quick reaction and TRANSPARENCY!!

[-] JohnDolt@lemmy.world 2 points 1 year ago

Good shit! Thanks for keeping things up and the pretty quick response as well.

[-] TWeaK@lemm.ee 2 points 1 year ago

Is a password change advised? How does the JWT cookie and exploit effect apps eg Jerboa?

load more comments (1 replies)
[-] xortingen@lemmy.world 1 points 1 year ago

Thanks for the quick response. Do we know if there was any data leak?

[-] downpunxx@lemmy.world 1 points 1 year ago

******* This happened to me, one of my posts had it's photo deleted (I didn't delete it), then when I replaced it, the next time I checked the entire post had been deleted.

[-] medvedev@lemmy.world 1 points 1 year ago

thanks to admin team for resolving that quickly

[-] Noedel@lemmy.world 1 points 1 year ago

Is this why I had to sign in and out of my account on liftoff?

I couldn't comment untill I did that. There may be others!

[-] gabriele97@lemmy.g97.top 1 points 1 year ago

How are you preventing it to happen again until a patch is released from devs?

load more comments (1 replies)
[-] chokidar@lemmy.world 1 points 1 year ago

Well done on acting on it so quickly. I think I did see some of the fake announcements you were referring too but were taken down very quickly. Keep up the good work team and thanks for everything you are doing!

[-] mayo@lemmy.world 1 points 1 year ago

Been busy for lemmy team lately hopefully it's not too much

[-] tobier@lemmy.world 1 points 1 year ago

Thanks for being open about this and quick to fix it!

[-] ext23@aussie.zone 1 points 1 year ago

So is it safe to log back in?

load more comments (1 replies)
[-] mourkeer@lemmy.world 1 points 1 year ago

I lost some of my post history. Is there a data issue that's come from this? Why are my comments gone?

[-] pgetsos@kbin.social 1 points 1 year ago

If it is only recent post history, maybe it was purged along with many malicious comments/posts

[-] mourkeer@lemmy.world 1 points 1 year ago

I know but this was over a week of comments lost.

[-] coconutxyz@lemmy.world 1 points 1 year ago

lemmy explorer still acting kinda funny, not sure who is the person in charge to inform

load more comments (2 replies)
[-] mayo@lemmy.world 1 points 1 year ago

Had an issue at work not long ago involving stolen tokens and back then it looked as if the token was scraped along with a lot of other web traffic and then about 12 days later they gained access.

load more comments
view more: ‹ prev next ›
this post was submitted on 10 Jul 2023
827 points (99.4% liked)

Lemmy.World Announcements

28381 readers
2 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS