828
submitted 2 years ago* (last edited 2 years ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

(page 3) 50 comments
sorted by: hot top controversial new old
[-] JohnDolt@lemmy.world 2 points 2 years ago

Good shit! Thanks for keeping things up and the pretty quick response as well.

[-] TWeaK@lemm.ee 2 points 2 years ago

Is a password change advised? How does the JWT cookie and exploit effect apps eg Jerboa?

[-] henfredemars@infosec.pub 1 points 2 years ago

You will have to login again for those apps. As far as we know, the exploit doesn't allow someone to actually steal your password directly, just the session you were logged into.

However, it is my personal opinion that you should change your password anyway out of an abundance of caution.

[-] MarekKnapek@lemmy.world 2 points 2 years ago

So what happened:

  • Someone posted a post.
  • The post contained some instruction to display custom emoji.
  • So far so good.
  • There is a bug in JavaScript (TypeScript) that runs on client's machine (arbitrary code execution?).
  • The attacker leveraged the bug to grab victim's JWT (cookie) when the victim visited the page with that post.
  • The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.

Am I right?

I'm old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

  • User provided content (post using custom emojis) caused havoc when processing (doesn't matter if on server or on client). This is lack of sanitization of user-provided-data.
  • JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
  • How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
  • The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Am I right? Correct me if I'm wrong.

Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.

Marek.

[-] abhibeckert@lemmy.world 1 points 2 years ago* (last edited 2 years ago)

I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

I'm a modern web developer who used to be an old-school one.

User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.

Yeah - pretty much, though there are some mitigating factors.

Strictly speaking, it was the alt text for the emoji. Alt text is HTML, and rather than allow arbitrary HTML they allowed another language called Markdown. Markdown is "a plain text" language with human readable syntax specifically designed to be converted into HTML.

Markdown is the right format to use for emoji alt texts, but you do need to be careful of one thing - the original purpose of Markdown was to allow HTML content to be easier to write/read and it is a superset of the HTML language. So arbitrary HTML is valid markdown.

Virtually all modern Markdown parsers disable arbitrary HTML by default, but it's a behaviour which can be changed and that leaves potential for mistakes like this one here. Specifically the way Lemmy injected emojis with alt text into the Markdown content allowed arbitrary HTML.

This wasn't an obvious mistake - the issue over on Lemmy's issue tracker is titled "Possible XSS Attack" because they knew there was an XSS Attack somewhere and they weren't immediately sure if they had found it in the emoji system. Even now reading the diff to fix the vulnerability, it still isn't obvious to me what they did wrong.

It's fairly complex code and complexity is the enemy of security... but sometimes you have to do complex things. Back in the "old-school" days, nobody would have even attempted to write something as complicated as a federated social network...

JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS.

Yeah - the Lemmy developers made a mistake there. There are a few things they aren't doing right around cookies and JWT tokens.

Hopefully they fix it. I expect they will... It was already actively being discussed before this incident, and those discussions have been seen by a lot more people now.

How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.

There are several levels of isolation that could have blocked this:

  1. Users should not be able to inject arbitrary HTML.
  2. A flag on the page should be set telling the browser to ignore JavaScript in the body of the page - this is a relatively new feature in the web and disabled by default for obvious backwards compatibility reasons, but it should be set especially on a high value target like Lemmy, and I expect once it's been around a little longer browsers will enable it by default.
  3. A flag should have been set to block JavaScript from contacting an unknown third party domain. Again, this isolation is a relatively new web feature and currently disabled by default.
  4. As you say, JavaScript shouldn't be able to access the JWT token or the cookie. That's not a new feature in the web, it's just one Lemmy developers didn't take advantage of (I don't know why)
  5. Even if all of those previous levels of isolation failed... there are things Lemmy should be doing to mitigate the attack. In particular instance admins have had to manually reset JWT tokens. Those tokens should have expired somehow on their own - possibly the moment the attacker tried to use them.

The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Yep - the modern best practice is for admins to manage the site via a completely different system. That adds considerable complexity and cost though, so it's rarely done unfortunately. But you know, Lemmy is open source... so if someone wants to take on that work they can do it.

I'll add one more - it should have taken less time to close the exploit... but given this is the first serious exploit I'll forgive that.

Ultimately several of failures contributed to this attack. I expect many of those failures will be corrected in the coming weeks, and that will make Lemmy far more secure than it is right now - so that next time there's a bug like the one in the Markdown parser it isn't able to cause so much disruption.

The good news is no harm was done, and a lot of people are going to learn some valuable lessons as a result of this incident. Ultimately the outcome is a positive one in my opinion.

[-] padjakkels@lemmy.world 2 points 2 years ago

Thanks for the quick reaction and TRANSPARENCY!!

[-] V4uban@lemmy.world 2 points 2 years ago

Thank you for your fast answer!

[-] Ahmed@lemmy.world 2 points 2 years ago

Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.

[-] Simodeus@lemmy.world 2 points 2 years ago* (last edited 2 years ago)

It seems that I lost all my subs. There were not many but still annoying.

E: Still subbed but can't see those in Voyager.

[-] vampyre@lemmy.world 1 points 2 years ago* (last edited 2 years ago)

I thought I’d lost mine too, and when I checked the community I wasn’t subbed. I could still view my profile, comments and posts though.

I cleared the cache, then tried to post here and it said I was logged out (even though I could see all my activity except subs). I couldn’t see any way to logout, so I edited my profile and re-entered my password then hit save. That seems to have fixed it, now I can post and my subs are back.

[-] Molecular0079@lemmy.world 2 points 2 years ago

Thanks for the post-mortem and the quick fix! Glad you guys around to help battle test Lemmy's code.

[-] Atiran@lemm.ee 1 points 2 years ago

One of the reasons I used a throwaway email here.

[-] hacktheegg@programming.dev 1 points 2 years ago

Huh, i think i got lucky by forgetting that there is something i can consume other than youtube

[-] tobier@lemmy.world 1 points 2 years ago

Thanks for being open about this and quick to fix it!

[-] mayo@lemmy.world 1 points 2 years ago

Been busy for lemmy team lately hopefully it's not too much

[-] PolarBone@lemmy.world 1 points 2 years ago* (last edited 2 years ago)

I’ve been unable to login on desktop since this happened. Only been able to login via Memmy on IOS.

I put in my info and it kicks me back to the front page and doesn’t log me in.

I’ve tried clearing cache too

EDIT: Switching browser to Edge seemed to let me. Weird. Even reinstalled Firefox and still won't let me.

[-] xortingen@lemmy.world 1 points 2 years ago

Thanks for the quick response. Do we know if there was any data leak?

[-] chokidar@lemmy.world 1 points 2 years ago

Well done on acting on it so quickly. I think I did see some of the fake announcements you were referring too but were taken down very quickly. Keep up the good work team and thanks for everything you are doing!

[-] gabriele97@lemmy.g97.top 1 points 2 years ago

How are you preventing it to happen again until a patch is released from devs?

load more comments (1 replies)
[-] weeahnn@lemmy.blahaj.zone 1 points 2 years ago

is that why I can't log into my lemmy.world account?

[-] weeahnn@lemmy.world 1 points 2 years ago

ok not a problem anymore. seems like I just had to clear my cache and it let me log in

load more comments
view more: ‹ prev next ›
this post was submitted on 10 Jul 2023
828 points (99.4% liked)

Lemmy.World Announcements

28381 readers
14 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS