2024 Weeknote 29 : Apparently ‘CrowdStrike’ is a verb

July 15 – 21

My “weeknotes” capture events, thoughts, and other items from the past week, mostly focused on work.

[1] We got CrowdStruck

Yep. It happened to us, just like it happened around the world.

For me, Friday started at 2:30am as I discovered an incident in progress via Teams, and I confirmed a long list of offline servers in our monitoring tool. I hopped into the conversation and offered to handle communications for the technical resources, hoping this was something that could be resolved in a matter of a couple hours.

By 4:00am it was clear this was a bigger problem than a few servers down. And by 4:30am it was clear this was an international event—systems we didn’t even manage were down for us, and media reports and social media were pointing the finger at CrowdStrike. Then we knew we were in trouble.

The rest of the day went the way it went for countless other organizations. Everyone was impacted, most importantly our server and desktop engineering teams. We closed county government offices to public business. Most county employees were sent home, with the work day canceled. (Later, this would turn out to be a mistake, because we couldn’t fix problems for users that weren’t present.)

My official day ended around 5:30pm on Friday, after 15 hours of straight work, and with the knowledge that this coming week will similarly stink. We don’t know the exact math, but it’s likely north of 2,000 PCs are affected with this issue and will require manual intervention, including disabling Bitlocker drive encryption by manually entering a 56-character unique key to get PCs to boot into Safe Mode.

[2] Civilla tour

On Wednesday I took 3 colleagues on a pilgrimage to Civilla, the nonprofit in Detroit that has made a name for itself working with the State of Michigan and others on developing human-centered digital (and non-digital) services. We drove up in the morning, then got a 2-hour studio tour and introduction from 2 of the 3 co-founders: Adam Selzer and Michael Brennan.

This was a stunning introduction to what they have achieved in the last 10 years. I want to go back and learn more, and I will definitely be looking into their (free) online learning courses. We couldn’t believe we got to meet 2 of the founders, and that we even sat around their boardroom table for a bit.

Civilla has a history that we in government cannot exactly replicate. But there are so many lessons in their approach we can absolutely use. They achieved their amazing results on a shoestring budget, with lots of people pitching in to help drive a unique and powerful mission. They talked to real people in the real world using government services to find the gaps and then share those gaps with government leaders, far from the offices where services are delivered to the public. They routinely short-circuited the communications between the aspirational ideas of leaders and the gritty details of delivery, and that made all the difference. We can do that, too.

If you don’t know Civilla, you should. Check out their highly-produced video that recaps their crowning achievement: the revamping of public benefits applications in the State of Michigan. I dare you to watch this without tearing up, at least a little:

If you work in government services, and especially digital services, and you get a chance to do the Civilla tour, jump on that chance.

[3] How an “abundance” mindset may help fashion a response to modern socio-political woes

I don’t have a lot to say about this now, because I need more time with these ideas, but I’m liking where the folks writing about an “abundance” mindset are going.

If you’re upset about how politics in our democracy are failing to meet our needs as a civil society, you probably owe it to yourself to check out this piece: Abundance as a Sense-making Framework.

Shared via Jennifer Pahlka, this piece explores the idea that polarized explanations for why things are going sideways don’t really work. That is, Democrats can’t just blame the GOP, and vice-versa. This also echoes an idea by Anil Dash—formalized in this May 29 post—that the results of any system are, by definition, perfect results. That is, whatever results you’re getting, they are products of a system that was setup to give you those exact results. Whether the system designers intended that outcome is another matter.

Bottom line: The results we’re getting from our politics today are an emergent property of the abundance of our modern world, and we need new ideas to change these systems to get better results.

[4] Miscellanea

Up until this week, I was saying, “If I get asked for one more chatbot, I will scream.” But now I have an alternative! This week U.S. Digital Response posted a brief explainer on chatbots that I will now send around to any agency or team that asks for one. Highly recommended: What is a chatbot, anyway?
We completed 4 second-round interviews for a new Project Manager. That’s 13 interviews total—plus additional screening calls, emails and so forth—and we think we have a winner, but we’ll see in the days ahead. (Actually, we had a few winners, but sadly we can only hire one PM right now.)
Our leadership team completed our Business of People training sessions this week. Highly recommended. Best leadership / management training I’ve encountered in my career.

[5] Watch This

It’s always interesting to see how the mainstream media covers tech stories that have global impact. Seeing how it’s covered by my favorite news service — DW News — is even more interesting. This is very balanced worldwide coverage from a day after the CrowdStrike story started.

[6] Internet Funnies

Following last week’s assassination attempt, the GOP convention, and Biden getting COVID-19 this week, it’s been a little tougher to find amusing bits. But if you keep looking, you can always find them. After all…