Friday, July 19, 2024

Thoughts on the Crowdstrike Outage

By now you're probably aware of the massive outages caused by a bug in Crowdstrike. The bug is the result of an update pushed out overnight, which caused Windows systems to become unusable. Unfortunately, Crowdstrike is very widely used by large corporations so the impact is immediate and widespread. A Symantec Enterprise Antivirus update had a similar effect about 12 years ago, but this one is much more widespread.

We are lucky that only Windows systems were directly affected. Crowdstrike also makes endpoint protection software for Mac and Linux. We use it on everthying. Several of my coworkers who use Windows were dead in the water this morning. The fix was to reboot into safe mode and delete the affected file. PCs with Bitlocker disk encryption enabled needed a recovery key from IT. I use a Mac so I wasn't directly impacted but because user authentication servers run on Windows I had trouble accessing some systems.

This could have been a lot worse. If this had affected Linux systems, it potentially could have taken out DNS and NTP servers, and the Cisco Network Registrar systems used to provision cable modems. That would have killed voice, video, and Internet service for any customer's device which rebooted. Recovery would have been greatly impeded because IT personnel wouldn't be able to communicate or get the fix easily.

AIUI, it's taken out several airlines, shipping companies, banks, and various point of sale systems.

It's a perfect example of the dangers of IT software monoculture where one bug can take out a huge swath of systems.

Consider it a sign to increase preparations for grid (power and Internet) collapse.

No comments: