1202
submitted 4 months ago* (last edited 4 months ago) by rxxrc@lemmy.ml to c/technology@lemmy.world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...

(page 2) 50 comments
sorted by: hot top controversial new old
[-] lustyargonian@lemm.ee 39 points 4 months ago

Linux and Mac just got free advertisment.

[-] LordWiggle@lemmy.world 26 points 4 months ago

The words 'Mac' and 'free' aren't allowed in the same sentence.

load more comments (5 replies)
load more comments (2 replies)
[-] UncleArthur@lemmy.world 37 points 4 months ago

Annoyingly, my laptop seems to be working perfectly.

[-] Valmond@lemmy.world 22 points 4 months ago

That's the burden when you run Arch, right?

load more comments (2 replies)
[-] kamenoko@sh.itjust.works 35 points 4 months ago

AWS No!!!

Oh wait it's not them for once.

load more comments (1 replies)
[-] alphacyberranger@sh.itjust.works 34 points 4 months ago* (last edited 4 months ago)

One possible fix is to delete a particular file while booting in safe mode. But then they'll need to fix each system manually. My company encrypts the disks as well so it's going to be a even bigger pain (for them). I'm just happy my weekend started early.

load more comments (11 replies)
[-] spacesatan@lemm.ee 32 points 4 months ago

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas.

Never trust a texan

load more comments (4 replies)
[-] jedibob5@lemmy.world 29 points 4 months ago

Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half...

[-] veam@lemmy.world 27 points 4 months ago

oh joy. can’t wait to have to fix this for all of our clients today…

[-] iturnedintoanewt@lemm.ee 19 points 4 months ago

You have no idea how much fun its being.

load more comments (3 replies)
load more comments (1 replies)
[-] Nachorella@lemmy.sdf.org 26 points 4 months ago

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I'll be working from home for a few days before anything is fixed.

[-] badbytes@lemmy.world 26 points 4 months ago

Stop running production services on M$. There is a better backend OS.

[-] dan@upvote.au 31 points 4 months ago

The issue was caused by a third-party vendor, though. A similar issue could have happened on other OSes too. There's relatively intrusive endpoint security systems for MacOS and Linux too.

[-] Swarfega@lemm.ee 21 points 4 months ago

That's the annoying thing here. Everyone, particularly Lemmy where everyone runs Linux and FOSS, thinks this is a Microsoft/Windows issue. It's not, it's a Crowdstrike issue.

load more comments (12 replies)
[-] stringere@sh.itjust.works 23 points 4 months ago

Crowdstrike did the same to Linux servers previously.

load more comments (3 replies)
[-] Pudutr0n@feddit.cl 23 points 4 months ago

This is a better article. It's a CrowdStrike issue with an update (security software)

load more comments (2 replies)
[-] Mikina@programming.dev 21 points 4 months ago

I see a lot of hate ITT on kernel-level EDRs, which I wouldn't say they deserve. Sure, for your own use, an AV is sufficient and you don't need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.

The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.

Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.

I'm not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn't warranted here.

Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.

load more comments (1 replies)
[-] umami_wasbi@lemmy.ml 19 points 4 months ago

No one bother to test before deploying to all machines? Nice move.

[-] huginn@feddit.it 21 points 4 months ago* (last edited 4 months ago)

This outage is probably costing a significant portion of Crowd strike's market cap. They're an 80 billion dollar company but this is a multibillion outage.

Someone's getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.

load more comments (6 replies)
load more comments (1 replies)
[-] Treczoks@lemmy.world 19 points 4 months ago

I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.

[-] wizardbeard@lemmy.dbzer0.com 38 points 4 months ago

It's not a flaw with Windows causing this.

The issue is with a widely used third party security software that installs as a kernel level driver. It had an auto update that causes bluescreening moments after booting into the OS.

This same software is available for Linux and Mac, and had similar issues with specific Linux distros a month ago. It just didn't get reported on because it didn't have as wide of an impact.

load more comments (9 replies)
load more comments (1 replies)
load more comments
view more: ‹ prev next ›
this post was submitted on 19 Jul 2024
1202 points (99.5% liked)

Technology

59454 readers
2021 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS