One little security patch…
Took down half the world yesterday morning.
A major cyber outage has grounded flights and disrupted businesses and media organizations throughout the world.
The technology glitch caused chaos on Friday morning with Delta Air Lines, American Airlines and United Airlines issuing statements saying that their flight operations had been impacted. Emergency response systems were down at police agencies and healthcare providers in Phoenix, Arizona, forcing some police and ambulance providers to dispatch cars manually, per The Arizona Republic.
Banks and financial services firms from Australia to India and Germany warned customers of disruptions.
Full article, HERE from Fox News. And HERE from the AP.
So, Windoze machines worldwide got the infamous blue screen of death this morning. Servers and who knows how many more machines???
And it’s not an easy fix. MS has to be booted in safe mode, then ‘cleaned’, THEN the new patch reapplied. That is not a 30 second issue.
Meanwhile, folks were without credit cards, ATMs, bank operations, etc. Soooo, if they didn’t have cash on them, they were…broke. Most folks ‘I’ know carry at least a $20 stuffed away somewhere in their wallet or purse as a ‘just in case’. I know I’ve had to use that a couple of times when travelling and needed gas and the station’s credit machines weren’t working. I’ve also seen it happen at a major grocery store chain before too.
Much less the issues with airlines worldwide having to resort to…’paper’ to do things, and lines hours long with little/no results (and no hotel vouchers, so they were stuck at whatever airport they were in), with little hope of getting a flight anytime soon.
Microsoft has always had ‘issues’ with the kernel, which is usually what causes the blue screen of death, but to have a security update do it is… sad, because MS has NEVER been a graceful failover system since the DOS days.
Thankfully, MAC and Linux machines didn’t have the issue, otherwise, I believe we would have seen and entire world stoppage!
And I can’t help but wonder what some folks would have done with that…
This MS has to be booted in safe mode might not be possible, depending on how the Windows machine is set up. If the filesystems are encrypted with BitLocker, to boot it into “safe” mode, you need to know the BitLocker key, and as an employee, you might (won’t?) have it. It’ll really affect repairing remote workers.
Replacement laptop via Fedex?
Yup, it was, and still is a big ‘ol mess.
The Tech News thread over at AoSHQ has been quite informative the past couple of days. Pixy (who runs that thread daily and is that site’s administrator, as well as others) is all over this.
What you said about cash: Yup. That. I always have enough to fill the vehicle, or buy groceries, or both. I had to use the cash in the wallet as a fallback twice this past week, once at a Circle-K and once at a local gas station. Both were capable of continuing to transact with cash though, so that’s a plus.
Some businesses can’t do transactions (even cash) if their network or their point-of-sale server goes down. That’s bad design, and I’m seeing more and more of that. *shaking head* Talk about a nightmare scenario if everything goes that way…
Two coworkers of mine were effected. One has an electric payment due yesterdaybut Chimeis not paying out. The other coworker has ason’s visiting girlfriend due to fly out this morning but am not sure the airport will be boarding passengers. That same coworker depends heavily on other payment methods than hard currency and never has any cash on hand.
Talk about a single failure point. “Hey I’ve got a great idea, let’s make the world’s economy dependent on a single company’s software.” Makes the dire predictions about the “Y2K” bug look like a tempest in a teapot.
It’s worse than that. A single point evil MicroSoft failure.
+1
I predict a great shifting of IT staff, as poor employers demand and good employers entice.
I was shocked to NOT be affected by it at work
Have you seen the CrowdStrike ad touting their DEI efforts? Hilarious in the context of what happened. Now I’m not sure if it was real or photoshopped, but it made a good point either way. Of course, now I can’t remember where I saw it and can’t find it again!
And John Deere announced they are diching their DEI efforts – guess they, like Tractor Supply, realized their customer base was not really into ‘diversity’, etc.
John Deere is still screwed, though, as they fired a bunch of Americans and moved their plants to Mexico. Lots of pissed off farmers out there.
And annoyed that they can’t just fix the darn things.
And this decades old. I recall a story of a fellow getting ribbed by neighbors for his “east bloc” machinery. But his reply was, “I can fix it with my toolkit. I don’t have to wait for the service tech. to show up.”
Retired IT … don’t miss it for a moment. The “Cloud” is nothing more than someone else’s storage/app server on the web. Too many decision makers got hypnotized by the “glamour” of outsourcing, even as it siphoned the service and knowledge needed to run their organizations. It’s a great single point of failure, and this is one dandy failure.
And people wonder why I hate “The Cloud” for storage, and hate subscription programs.
If you don’t physically store it and own it, it isn’t yours.
+1
+1000
I am now in my seventies and I remember my Grandpa teaching me as a little kid to always have some cash on you. I’ve always got an “emergency” C-note tucked away and also five twenties in the billfold. Once, at a business conference, when our crew took a cab to the airport the cabbie had a ‘system fail’ and could not use cards for payment – I covered the whole cost with my cash as nobody else had any on them.
Remember the old “Maverick” TV series? The guys had an emergency $1000 bill pinned inside their waistcoats.
I remember when an Alcatel one line of software update managed to shut down the entire east coast phone service. “It’s just one line of code, no need to do any regression testing.” Heh…
I had a library file go south years ago on my wife’s computer. Booting required third party software, swapping files from the DOS command line, crossing my fingers and being rewarded with a restored machine.
When Windows crashes, it’s never an easy fix, and due to it’s relatively inexpensive price, companies risk substantial amounts of money by using it.
Cash on hand. My cell phone is in a protective case. Tucked behind the phone is some cash and a SIMS tool. Ziplock bags with quarters are stashed in several locations in my car. A thirty year old 12 volt air pump is always in the trunk along with a minimal bug out bag.
Funny, the company that failed bigly is into election integrity.
Preview of coming attractions in November?
I had an extra chunk of cash because I already knew I was going to need it for a few-day hotel stay. Between transport and food delivery, I don’t like to do cards.
Remember that Crowdstrike was the ones that assured everyone that the DNC breach absolutely, positively, pinky swear! was the result of the Russians. (And not Seth Rich with a thumb drive.)
So, yeah, a lot of background.
And to make it more interesting, while a freshly rebooted machine is waiting for the Crowdstrike fixed update, it’s most likely wide open to any random(?) malware just waiting for machines. Hell of a Friday update, guys.
“Sufficiently advanced incompetence is indistinguishable from malice.” Not gloating, but all the Casa RC machines are Linux. Did NOT need any additional drama this week.
All- Thanks, glad I’m not the only ‘old fart’ that believes cash is king…and doesn’t care for single points of failure.
100 and a 50 burried in my wallet. Also have at least 200 in 20’s in my wallet. I also have 300 hidden in my car. Just put 10 bucks a week in a drawer. 520 at the end of a year. Simple
This is basically downstream of malformed ideas we have about basis for trust, that are widespread in our society. And, slightly adjacent at least to universities.
We are trying to regulate ‘good’ technical decisions. Crowdstrike’s apparent de facto monopoly is said to be downstream of some of that.
We have professional lawyers heavily involved in a lot of regulatory activities. They basically judge what a lot of people can do without getting into a great deal of trouble.
Understanding technical problems, particularly technical problems of great complexity, is maybe a different sort of background. From what I can tell, each technical background is suited for a very narrow slice of technical problem.
As I have learned more about how engineering doctoral programs seem to work, I have become increasingly convinced that the general trend of them is to train engineering PhDs that I would consider to have some holes in their foundation. A lot of the time, said holes may not screw up most of the work that the PhD does.
Telling the JDs the exact way that the JDs have screwed up, and also about what to do about it, may be a point of short coming where the modern engineering PhD is concerned.
Or maybe I am simply insane, and fooling myself. 😀
Hmmmm….. Crowdstrike… Where have I heard that name before….
oh yeah. The attack on Trump and the Steele Dossier.
Who was recently attacked and had the media attention drawn from the attack on him and all the questions concerning it by Crowdstrike’s failure?
i wonder what they used this outage to get past us this time. they got rid of j6 video/documents last time there was an “update.” and days after the attempt on trump. things that make you say hmmmm.
All- Thanks, and no good answers here… Bob- Good point!
For at least two decades I’ve said we are building a civilization dependent on toy computers running flakey software. I rest my case.
I have no idea if it’s true, but supposedly SW Airlines was not effected because their machines are running Windows 3. Too good to check. /snerk
Microsoft Releases Recovery Tool for Windows Machines Hit By Crowdstrike Issue. This avoids having to boot into Safe Mode or a requirement of admin rights on the machine, because the tool is simply accessing the disk without booting into the local copy of Windows. If a disk is protected by BitLocker encryption, the tool will prompt for the BitLocker recovery key and then continue to fix the CrowdStrike update.
So you will still need to know the BitLocker key before you can fix the problem. Remote users?
A couple of others inferred it above, but no one outright said it:
This wasn’t a Microsoft issue, it was a Crowdstrike issue. They installed an “update” to their software which broke Windows.
That’s a product of insufficient (nonexistent?) testing and wasn’t the fault of Microsoft (for a change). That’s why it didn’t affect everyone with Windows servers or personal computers (Crowdstrike is pretty exclusively used by big companies).
I work at a tech company in the transit industry. Several of our customers were affected by this because they pretty much all use Windows servers and some of them use Crowdstrike. They had to fix the underlying issue on their servers and then we had to go back in and get our applications up and running again.
It was a scramble, but we had everyone going again by about 10am Eastern time on Friday.
We don’t use Crowdstrike so we weren’t affected at all other than helping our customers fix their issues.