827
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

top 50 comments
sorted by: hot top controversial new old
[-] LuckyLu@lemmy.world 176 points 1 year ago

Very impressed by how quickly action has been taken by this and other instances to patch the issue.

[-] trouser_mouse@lemmy.world 24 points 1 year ago

Very, seems like great work.

load more comments (10 replies)
[-] ThisIsMyLemmyLogin@lemmy.world 65 points 1 year ago

I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.

load more comments (32 replies)
[-] TheVampireSaga@lemmy.world 62 points 1 year ago

what steps are being taken to ensure it doesn't happen again? was any personal data compromised for users?

[-] ruud@lemmy.world 56 points 1 year ago

Good point, I'll update the post.

[-] TheVampireSaga@lemmy.world 31 points 1 year ago

Also I am curious, what's the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?

[-] ruud@lemmy.world 35 points 1 year ago
[-] sirnak@lemmy.world 8 points 1 year ago

Why wasn't there an info on /lemmy-world.statuspage.io ?

load more comments (1 replies)
load more comments (2 replies)
load more comments (1 replies)
[-] BustedPancake@lemmy.world 14 points 1 year ago

So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I'm sure they were only interested in admin cookies, so most others were "useless" to them? I see nothing wrong with my posts so I should be safe, right?

[-] cantevencode@lemmy.world 15 points 1 year ago

Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless

load more comments (5 replies)
[-] InverseParallax@lemmy.world 8 points 1 year ago

Nice work on the recovery, especially from a 0-day.

[-] giant_smeeg@lemmy.world 8 points 1 year ago

Thanks! Is there any understanding as to why? Or are we thinking some script kiddies because they can?

[-] hawkwind@lemmy.management 13 points 1 year ago

They defaced it with dicks and changed the federation list to be only threads.net. I don't think it was a state sponsored chinese hacking group. :)

[-] linearchaos@lemmy.world 10 points 1 year ago

I'm ok with the dicks but the threads are TOO FAR!!! shuffles off to the angry done**

Thank you all for staying on top of it.

load more comments (1 replies)
[-] alaxitoo@lemmy.world 51 points 1 year ago

Thank you for your work πŸ™

[-] trouser_mouse@lemmy.world 46 points 1 year ago* (last edited 1 year ago)

First - really good summary and sounds like everyone is working hard.

Cross posting the below comment.

Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

[-] ruud@lemmy.world 24 points 1 year ago

Thanks for the info. We're looking into this.

[-] hawkwind@lemmy.management 10 points 1 year ago

Out of curiosity, where would the regulators go for a case like this? There's no "company" running it per. se.

[-] linearchaos@lemmy.world 12 points 1 year ago

It seems the general consensus is GDPR applies even to OSS non company entities, but it would appear that there's very little being done to honor it.

https://www.zwilnik.com/better-social-media/activitypub-conference-2019/oss-compliance-with-privacy-by-default-and-design/#:~:text=Although%20GDPR%20directly%20applies%20only,sysadmins%2C%20including%20in%20the%20Fediverse.

This article outlines Fediverse and responsibilities, I think it mostly requires someone to file a lawsuit before there's any action.

In another case a man had cameras in his back yard that could also see a public area and was fined and forced to move them.

https://www.termsfeed.com/blog/gdpr-exemptions/

Mainly it just seems to be fodder to be used in lawsuits to make people comply with others security wishes. Not certain how all that works since cities are covered in public cameras.

load more comments (1 replies)
load more comments (2 replies)
[-] nosut@lemmy.world 33 points 1 year ago* (last edited 1 year ago)

Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

EDIT: it has been added back to the block list.

[-] AlmightySnoo@lemmy.world 31 points 1 year ago* (last edited 1 year ago)

the details of the vulnerability are already known now anyway since there's a fix that was proposed on the Lemmy GitHub so I don't think it will hurt others to talk about it

[-] namelivia@lemmy.world 11 points 1 year ago

Could you please link the issue? Thanks!

[-] namelivia@lemmy.world 24 points 1 year ago
[-] AlmightySnoo@lemmy.world 25 points 1 year ago

yup that's the one

what I find weird is that the "fix" still focuses only on the front-end, the issue is still that unescaped HTML is being stored in the database and still trusting the front-end is nuts

[-] TheVampireSaga@lemmy.world 6 points 1 year ago

I think the main developers are aware of either of them but I'm not sure, haven't seen anyone site admin wise talk about this mess.

load more comments (2 replies)
load more comments (5 replies)
load more comments (1 replies)
[-] bluemellophone@lemmy.world 29 points 1 year ago* (last edited 1 year ago)

Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

[-] ruud@lemmy.world 39 points 1 year ago

Good point. I did post about this on Mastodon @mwadmin@mastodon.world

[-] bluemellophone@lemmy.world 9 points 1 year ago
[-] hawkwind@lemmy.management 11 points 1 year ago

Don't fall for it. They're also an admin on mastodon.world! :)

load more comments (3 replies)
[-] brittleback@lemmy.world 22 points 1 year ago

Well done all involved. Sounds like it was caught and mitigated quickly

[-] wazoobonkerbrain@lemmy.world 17 points 1 year ago

IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.

load more comments (2 replies)
[-] dr_scientist@lemmy.world 15 points 1 year ago

Good job. I don't understand very much of that, so that makes me all the more grateful. Thank you.

[-] MeshPotato@lemmy.world 13 points 1 year ago

Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I'm genuinely hopeful that this will be a good alternative.

[-] jarfil@lemmy.ml 12 points 1 year ago

How does this impact those using mobile apps like Jerboa or Liftoff, instead of the website directly?

load more comments (8 replies)
[-] PagingDoctorLove@lemmy.world 11 points 1 year ago

Can I ask some possibly dumb questions?

  1. What is JWT?
  2. Was any private user data compromised, and if so will users be informed?
  3. Is there anything regular users can do to avoid their data being compromised? For example, not accessing lemmy on certain web browsers?

Thank you!

load more comments (2 replies)
[-] AlmightySnoo@lemmy.world 10 points 1 year ago

One thing I don't get. Custom emojis can only be created by an admin, but you're saying an admin's account here got compromised because of that and not the other way around. Does that mean that an evil instance set a custom emoji with the injected JavaScript and propagated it to the federated instances?

load more comments (1 replies)
[-] cantevencode@lemmy.world 9 points 1 year ago

Does an admin account have any permissions to view email addresses or data of registered users?

Did MichelleG not have 2FA enabled?

Now that this has happened, it's be worth pushing this issue through as high priority. If HttpOnly was enabled, then an admin takeover would not have been possible.

https://github.com/LemmyNet/lemmy-ui/issues/1252

[-] nosut@lemmy.world 21 points 1 year ago

The JWT exploit bypasses 2FA requirements. It basically steals your active session and allows a third party to use it.

[-] cantevencode@lemmy.world 8 points 1 year ago

Good point. I suppose the only way to fix that particular issue to disallow cookie authentications from a new location

load more comments (2 replies)
load more comments (1 replies)
[-] giant_smeeg@lemmy.world 9 points 1 year ago

Was the vulnerability known before hand and not applied to this instance or is it new?

[-] AlmightySnoo@lemmy.world 18 points 1 year ago

see the GitHub repo, it's new

[-] ruud@lemmy.world 15 points 1 year ago

It's not fixed yet in the current version

load more comments (1 replies)
[-] ulu_mulu@lemmy.world 8 points 1 year ago

Amazing how you quickly reacted to this!! Bravo!!

TIP: if you can't login after what happened, clear out your browser cache including ALL cookies, that fixes it (it did for me at least). I believe it's also advisable to change lemmy password.

[-] 00Xero00@lemmy.world 7 points 1 year ago* (last edited 1 year ago)

I can't log into my account anymore, this one is a new one I've just made. I tried to reset my password but nothing came in the mailbox. I can still see comments and posts from that account though.

It's this one:

And I don't know why but I can't save the profile pic for this account.

Edit: Nvm, I use another email to sign up for Lemmy and forgot about it

load more comments (2 replies)
[-] 001100010010@lemmy.dbzer0.com 6 points 1 year ago

If I'm not from lemmy.world and visited a lemmy.world community via my home instance, does the hacker gain access to my account?

If I, while logged in to my home instance, accessed lemmy.world in another tab, does the hacker gain access to my account?

Does this hack infect devices used to access lemmy.world?

Sorry for noob questions, I'm just worried.

load more comments (2 replies)
[-] godless@lemmy.world 6 points 1 year ago

With the JWT secret rotation, shouldn't everyone be forced to re-login? I'm posting with my existing session without any changes.

load more comments (2 replies)
load more comments
view more: next β€Ί
this post was submitted on 10 Jul 2023
827 points (99.4% liked)

Lemmy.World Announcements

28381 readers
2 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 1 year ago
MODERATORS