Suboptimal ways to respond to a public security incident (lemmy.stuart.fun)

submitted 1 year ago* (last edited 1 year ago) by andrew@lemmy.stuart.fun to c/selfhosted@lemmy.world

12 comments fedilink hide all child comments

This issue is already quite widely publicized and quite frankly "we're handling it and removing this" is a much more harmful response than I would hope to see. Especially as the admins of that instance have not yet upgraded the frontend version to apply the urgent fix.

It's not like this was a confidential bug fix, this is a zero day being actively exploited. Please be more cooperative and open regarding these issues in your own administration if you're hosting an instance. 🙏

all 15 comments

sorted by: hot top controversial new old

[-] TragicNotCute@lemmy.world 2 points 1 year ago

IMO it’s not a good idea to be discussing attack vectors publicly when a number of other instances are unpatched and the exploit has been in the wild for less than a day.

I agree that admins need to work together, but discussing it in public on Lemmy so soon after the attack isn’t the way. There exists a Matrix channel for admins, that’s where this type of thing should go.

[-] entropicshart@lemmy.world 4 points 1 year ago* (last edited 1 year ago)

When a vulnerability at this level happens and a patch is created, visibility is exactly what you need.

It is the reason CVE sites exist and why so many organizations have their own (e.g. Atlassian, SalesForce/Tableau )

It is also why those CVE will be on the front page of sites like https://news.ycombinator.com to ensure folks are aware and taking precautions.

Organizations that do not report or highlight such critical vulnerabilities are only hurting their users.

[-] TragicNotCute@lemmy.world 0 points 1 year ago

It is common practice to notify affected parties privately and then give full details to the public after the threat is largely neutralized. Expecting public disclosure with technical details on how to perform the attack in less than 24 hours goes against established industry norms.

[-] Dark_Arc@lemmy.world 4 points 1 year ago

That only stands true when the issue is not being actively exploited.

[-] Puzzle_Sluts_4Ever@lemmy.world 0 points 1 year ago* (last edited 1 year ago)

[removed by mod]

[-] SeeJayEmm@lemmy.procrastinati.org 1 points 1 year ago

I strongly disagree with some of your points.

Yes, the vulnerability is out there. Maybe the root cause actually introduced a LOT of vulnerabilities. The fix is being pushed at a frantic pace. To expect the devs to take time out of the mad rush to notify those impacted to do a proper writeup is just insanity.

It's not insanity. It's called incident management and it's something the development team needs to build a proper procedure around, given the expanded scope of this project. I agree that the devs working on identifying, mitigating, and fixing the vulnerability should not be expected to also handle the communication. They need to designate someone for that role.

A 0-day was actively being exploited in the wild. There was confusion, misinformation, and a general lack of information.

You need to:

Indicate that you are aware of an ongoing problem and are working to identify it. This let's people know there is an issue and that you're aware of it. You can do this without giving specific details on how to replicate the exploit. This includes server admins publicly acknowledging that they are aware of the issue and will provide updates when they have them, to alleviate the concerns of their user base.
Once a mitigation are known, you publish that, in as many channels as you need to get that information out to the people who need it. So that server admins are aware of what they need to do to reduce their risk.
Once a fix is in place, you publish that, same as above.

The way I see it? This (hopefully) got fixed pretty much instantly and there is active work to get the fix applied by the people who need to apply it. That is what should be done.

And how do you know this since it's not been communicated? Most of the information I (as a person running a lemmy server) have been able to glean is from random threads spread across random communities.

Give it a week or two to see how they handle the public disclosure side of things.

A couple of weeks for a postmortem. Sure. A couple of weeks for an active, in the wild, 0-day, to officially communicate that the problem exists and how to mitigate/patch it. Absolutely not. I still don't see a security alert on the GitHub telling me I should be updating to to patch an active exploit and it's been how many hours now?

[-] Puzzle_Sluts_4Ever@lemmy.world -1 points 1 year ago* (last edited 1 year ago)

[removed by mod]

[-] SeeJayEmm@lemmy.procrastinati.org 1 points 1 year ago

Is the project small? Yes.

Did it explode in popularity leaving the devs overwhelmed? Certainly.

Do I expect them to strictly follow established ITIL incident management? No.

Do I expect them to communicate in a consistent way when an incident happens? Yes.

I agree the primary developers should be left to fixing the problems but there are enough active members of that project that someone could have handled communication in a more concise and official way. I don't consider random posts in asklemmy or selfhosting by random users just guessing to be a substitute for that.

If the project is going to persist and grow it needs to get better at that. Pointing it out isn't shitposting.

[-] Puzzle_Sluts_4Ever@lemmy.world -1 points 1 year ago* (last edited 1 year ago)

[removed by mod]

[-] SeeJayEmm@lemmy.procrastinati.org 1 points 1 year ago

I mean, there is a reason reddit hired so many people over the years. And if you are going to jump down the throats of people who prioritize fixing an issue and counting on “active members” to notify users over writing up the reports that many of those users won’t even look at? You want a production quality piece of software. That means Reddit or Threads or Bluesky.

Why are you getting so defensive? The only throat getting jumped down is mine, by you. I'm expressing my opinion of gaps in the communication of the project and how I think it can be improved. In a conversation thread on selfhosted no less. I'm not out in !lemmy@lemmy.ml bitching them out, submitting issues, or otherwise harassing the devs. Pointing out a gap and suggesting solutions is neither shitposting nor jumping down someone's throat.

I’ve been through similar a decent number of times on the corporate side. Something has gone very wrong. People want answers. A good manager assesses the situation and responds back “Look, we know what is going on and all hands are on deck to fix it. Making a powerpoint is not fixing it. We’ll do a proper write up for next week but we can either have So and So fix it or report on it.”

I think you're the one confusing this with a large corporate project. Not me. There's no managers here, there's no powerpoints, and at no point have I asked for a detailed write-up. I asked for someone on the project, who isn't actively working on identifying and coding the fix, to be the "point man". Post a simple sticky at the top of !lemmy@lemmy.ml xposted to !lemmy_support@lemmy.ml that indicates there's a problem, they're aware of it, and a fix it being worked on. Once mitigations are identified or fixes are published, update the post with that. Ideally, a github security incident would be also be published with the same info so people not watching lemmy at the moment can notified via that channel.

But people very much don’t seem to understand how small this project is. Spend time with passion projects and “open source” projects that AREN’T on the scale of a small-medium sized company and you understand that standards are going to be lower because people have day jobs and so forth.

I get it. I have pretty low standards. I'm just saying that a consistent communication strategy going forward for this project would be beneficial.

[-] tko@tkohhh.social 1 points 1 year ago

I'm with you. I figured out through various comments that I should update my UI to 0.18.2-rc.1, and also run an update statement on my database to fix the modlog. Only after that did I find the matrix channel. Eventually I also found !lemmy_admin@lemmy.ml which is great, but the only thread there on this issue doesn't even mention updating the UI. I think if we can get to the point where critical information that admins need to know is consistently posted in one place, it'll make everybody's life easier. I don't think that's too much to ask.

[-] andrew@lemmy.stuart.fun 1 points 1 year ago* (last edited 1 year ago)

If this was not a zero day being actively exploited then you would be 100% correct. As it is currently being exploited and a fix is available, visibility is significantly more important than anything else or else the long tail of upgrades is going to be a lot longer.

Keep in mind a list of federated instances and their version is available at the bottom of every lemmy instance (at /instances), so this is a really easy chain to follow and try to exploit.

The discovery was largely discussed in the lemmy-dev Matrix channel, fixes published on github, and also discussed on a dozen alternate lemmy servers. This is not an issue you can really keep quiet any longer, so ideally now you move along to the shout it from the mountaintop stage.

[-] GuyFleegman@startrek.website 1 points 1 year ago* (last edited 1 year ago)

This issue is already quite widely publicized and quite frankly “we’re handling it and removing this” is a much more harmful response than I would hope to see.

Hi, mod of a community on the instance in question here. Why is this response harmful? What should we have done instead?

[-] andrew@lemmy.stuart.fun 1 points 1 year ago

I feel like it's up for discussion here and you very well may stand by the response there, but IMO with how prevalent this issue is, a specific response of "we've disabled custom emoji" or "we're upgrading to 0.18.2-rc.1 today" would have been more constructive and reassuring to users. Removal of the question and lack of details gives me a lot less confidence that the issue and fix are understood and doesn't leave any room for that discussion.

[-] popemichael@lemmy.world 1 points 1 year ago

It's strange that they would try to bury this information.

The number 1 tool against future hacks like this is education.

this post was submitted on 10 Jul 2023

10 points (85.7% liked)

Selfhosted

40406 readers

249 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz