This is a transcript of episode 310 of the Troubleshooting Agile podcast with Jeffrey Fredrick and special guest Gene Kim.

Phoenix Project author, Gene Kim, is back on Troubleshooting Agile to discuss the groundbreaking theories of organizational management described in his new book, Wiring the Winning Organization. In this episode (part two of three), Gene describes the “Danger Zone”, and the first of three mechanisms for exiting the high-risk zone: Slowification.

About Our Guest

Gene Kim is a Wall Street Journal bestselling author, researcher, and multiple award-winning CTO. He has been studying high-performing technology organizations since 1999 and was the founder and CTO of Tripwire for 13 years. He is the author of six books, The Unicorn Project (2019), and co-author of the Shingo Publication Award winning Accelerate (2018), The DevOps Handbook (2016), and The Phoenix Project (2013). Since 2014, he has been the founder and organizer of DevOps Enterprise Summit, studying the technology transformations of large, complex organizations.

Show links:

Listen to the episode on SoundCloud or Apple Podcasts.

Danger Zone

Listen to this section at 00:14

Jeffrey: Welcome back to Troubleshooting Agile. This is Jeff Fredrick and I’m joined once again by guest Gene Kim. Gene, thank you for coming back!

Gene: Jeffrey, I’m so happy to be back. And I’m looking forward to another mutually exciting discussion and co-creation.

Jeffrey: Absolutely, co-creation! We talked last time about kind of your motivation of working on your new book, just published, Wiring the Winning Organization - a fantastic read. I was very happy to read it in advance, provide a quote, you know, and what I summarize it as, it distills so many of the lessons that I’ve learned, in no joke, 30 years of studying software, and it does just an amazing job of abstraction. And you simplify it into this very simple framework. And so let’s start there, let’s introduce it, tell me, for someone who doesn’t know, let’s talk a little about the book and how you described getting to the ‘winning zone’ and ‘wiring’ and your three main plays.

Gene: Yeah. Okay. Great. Yeah. I think the premise of the book is to say that the leaders are responsible for enabling people to do their work easily and well, and the way they do that is by creating the right social circuitry, which resides in what we call layer three. So layer one is, you know, the work in front of us, whether that’s code and IDE, the Linux environment that we’re trying to configure, the thing that we’re trying to assemble, the engine, we’re trying to design.

Gene: Layer two is the tools that we use. So it could be the IDE itself, the platform you’re running on, you know, the microscope to, you know, see the specimen.

Gene: And then layer three is the social circuitry. And I had mentioned last week the notion that, how do you get good at creating layer three social circuitry? And we can use the same intuitions and experiences that we use to create great layer one and layer two circuitry. But we do it for the social parts of the sociotechnical system. And the job of layer three is really to make sure that in order for people to do the work easy and well, everyone has to have what they need, when they need it, from whom they need it from, in the right format, right place, right time, and so forth.

Gene: And, you know, if we can investigate what the other side looks like is that no one has what they need, ever! And when they get it wrong time, wrong place, wrong person, wrong format, right? And so, um, I think we’ve all had experiences where we see systems that fully liberate people’s creativity and problem solving capabilities versus those that constrain or extinguish it. And so, the ways to characterize those two extremes is, one is the winning zone, which we call- let’s do the danger zone first. Life is bad when you have to do highly consequential work where everything’s tightly coupled. Everything when something goes wrong, causes everything to go wrong. You can’t undo, can’t learn, can’t experiment. And so that’s a losing formula.

Jeffrey: Real quick! Now, you and this is part of the book, and part of I think the genius of it, is that you and Steve managed to come up with an illustration of this fraught environment in a seemingly mundane example of refurbishing-

Gene: Oh, yeah!

Jeffrey: -a hotel. So you didn’t need to go to something really a touchstone of like a complex software project or an auto factory or some global enterprise. I think you can illustrate that this can happen with just a few people on a relatively simple project. And I think this really speaks to the universality of the principles you’re dealing with. So, this what you’re describing, these three levels, you can illustrate it with painting a room and moving furniture, which is brilliant.

Gene: Yeah. In fact, if I can tell you a little bit more of that story and where it came from, I’m not sure if you know this. So that was actually inspired by something that our mutual friend, Elizabeth Hendrickson-

Jeffrey: Oh, yeah!

Where You Draw the Lines

Listen to this section at 04:24

Gene: -said she said leadership, for her, is all about where you draw the lines. And she said, you know, you can imagine drawing a line that’s like completely not ideal, where two people have to work together and they can’t. There’s something about you, the way you draw the line that is like supremely unsuited. And so that thought experiment really came from that kernel of an idea. And the notion was, ‘hey, what if you have a project where you have to renovate an old Victorian hotel, you know, uh, scores of rooms and there’s three interdependent steps. You have to remove the furniture, paint the room, and put the furniture back in.’

Jeffrey: Seems simple.

Gene: Seems simple, right! ‘So imagine creating two silos. Movers and painters. And the only way that you can complete the room is through a schedule-‘

Jeffrey: Of course, how else could you do it?

Gene: Right, hahaha. And so like what ends up happening? I proved this in a simulation that I wrote was you easily get into a situation where the mover, you know, painters show up and the furniture is still there and the movers show up and the painters are still painting, and it causes a chaos and disruption and a tightly coupled system where no one can actually work independently. Um, and so we go into a lot of elaborations of what can go wrong, including movers mad at painters, painters mad at movers. We start measuring the painters on number of walls painted measuring movers based on-

Jeffrey: Very natural consequences of chaotic output from what actually sounds like very reasonable starting points. And I think that’s the genius of it is it the place that you drew the line initially in a way, seems kind of natural. It doesn’t seem like it’s going to lead to these unexpected consequences, and yet it does so. But then you have the way of like, okay, great. Now what do we do? We how do we move? This is the danger zone!

Gene: Yeah.

Jeffrey: You know, where do we want to be?

Gene: Well, in fact, if I can just add one more thing, right. Is that something really, genuinely surprising happens that you can- As if that weren’t bad enough, you can add one more rule: if a room gets behind, you expedite and you have-

Jeffrey: Of course!

Gene: -movers break the job and go somewhere else.

Jeffrey: Yeah.

Gene: And it turns out that actually causes everything to get worse because where you had one problem, now you’ve created two.

Jeffrey: Yes.

Gene: Because now you have the room that’s in trouble, and now you’ve broken another room team to help out. And so you’ve now problems spread. And this is a mark of coupling. So. Yeah.

Gene: And so anyone in the DevOps community might see the punchline coming. It’s like, ‘okay, instead of dividing by functional silos and integrating through a schedule, how about we just divide them up into room teams? So movers and painters work on a room independently of each other. Right. And we’re not allowed to steal movers and painters from another room.’ This allows a independence of action.

Gene: And what’s incredible is that just like modularization allows independence of action between room teams, you can also create independence of action between the moving painting, moving operations by defining where the handoffs between those sequential steps are. And that’s actually the secret of the Toyota production System, so the fact that modularity, and what we’re calling linearization- modularization and linearization are actually orthogonal to each other, they’re both mechanisms of simplification.

Gene: Boy, what a what a profound Aha! Moment, right? Economists even have a term, they can even measure just how vast amounts of value can be unlocked when you go from a system where no one can get anything done, because everyone’s stuck to one where teams can actually work independently, experiment independently, improve independently. Anyway, how am I doing?

Jeffrey: Fantastic, although I will say we’ve kind of jumped ahead, so you come at your your changes, your recipe for getting to this blissful state. You have three elements. And we’re kind of talking about the middle one here like the simplification. And I think the challenge a lot of people face is they’ll hear about these wondrous interventions, but they’re like, ‘well, we you know, we’re not sure how to apply it here. I mean, it’s great you’re able to figure it out for painters and movers. But there’s a step before what, you know, what allows you what gives you space and time,’ and you have this great, this great phrase, which is slowification.

Gene: Yeah.

Slowification

Listen to this section at 09:02

Jeffrey: What is slowification? How did you come up with the term? Where did this come from? And can you explain what you mean by slowification?

Gene: Yeah. So the slowification is trying to describe this concept that there’s actually no English word for that we could find, but we have a lot of adages for it, which is like slow down to speed up. Or slow is smooth, smooth is fast.

Jeffrey: Yeah, exactly. One of my favorites.

Gene: And, uh, yeah. And essentially what it’s saying is that you take you make a short time investment for a longer time gain. You stop sawing to sharpen the saw, there’s another one. And so the Germans have a word for this is, verbesserung.

Jeffrey: Oh!

Gene: But we couldn’t find one from English, in fact, I put a blog post out about 40, 80 words that ChatGPT came up as candidates, but none of them really worked for us. The notion is that at some point, you want to pause the game, right, so that you can reconfigure and improve. And so, one, you can say, ‘all right, with the moving and painting as bad as things get, you just keep going,’ right? You know, add more incentives or whatever, or you freeze the game.

Gene: You can activate a slower, more deliberate mode of analysis, and you reconfigure the system. You rewire the system. So the Navy Top Gun project was an example of slowification. You know, when too many US Navy pilots were being shot down over the skies of Vietnam? When in professional basketball, like when the coach calls time out, right? That’s a way to pause the game, to reset.

Jeffrey: But actually, I’ll say this in the book, you have an amazing, the most dramatic, story of slowification I’ve ever come across, and it was in the yacht race, right?

Gene: Oh, yeah!

Jeffrey: So let me see if I have this right. So this was this was college teams doing yacht racing somewhere in Europe. And one of the universities sent a team where the people hadn’t sailed before!

Gene: Right.

Jeffrey: They’d not sailed! They’d not done it! And so you might predict, they would do terrible. And initially they did, but then they but they had this weird quirky rule, which was when things would go wrong, they would stop and diagnose it… In the middle of the race! What was your response to this? I was just stunned. I thought, ‘this is amazing. I want everyone to know this story!’

Gene: Oh, it’s so great! So this is Adam Traina, and he actually spoke at the DevOps Enterprise Summit two months ago, now Enterprise Technology Leadership Summit. But it was this crazy story of all the business schools competing in a boating regatta. And the majority of the teams, as he described, you know, one of them, just had circumnavigated the globe. One of the coaches was an Olympic level you know yacht racing, something, something and in comparison, in the MIT, this is actually the LGO program, progenitor of the program that Steve was a part of. I’m sorry, a descendant of the program that Steve was part of. 80% had never been on a boat before.

Gene: And so the main principal, as you described was when you have a problem, call it out and pause. And he described, like all these kind of strange kind of non-conventional things that emerge, like most many of them, English was not the first language. And they had to create a vernacular so that certain terms would not sound like each other. Right.

Jeffrey: Okay.

Gene: They reconfigured the system to better transmit signals from the helm of the boat to other places. Including the extravagant, extreme example of, like, when a sail got wrapped around a mast or something, right? They they paused and you know, try to figure out how do we get in the state.

Gene: So the the punch line is, they were dead last in the first heat, I think I’m using the right term.

Jeffrey: Yeah.

Gene: They had actually placed near first in the second heat, but, you know, as a law of averages, right? They couldn’t keep up. By the second one, they were in first place. And what’s astonishing is, they had a winning streak that went on for years, including when Adam Traina and the other captain, the only other person with sailing experience, graduated and left the team.

Gene: And so they’re now on decades of a winning streak, where it shows that because they were so good at creating standard work that it survived the transition of leaders. It’s just this incredible, incredible story. We use that as a counter example of daily workarounds, like, instead of solving the problem, instead of pausing production. Right, we’re just going to make do.

Jeffrey: Yeah, normalization of deviance.

Gene: Normalization of deviance. Yeah.

Jeffrey: Exactly. Yeah. There’s a nominal process, but the process doesn’t really work. So we have workarounds, and the workarounds build up. And we kind of, like you said, make do. And you have to stop. Diagnose. And then yeah, like you said, they created this body of knowledge, right? They created a binder that they can then bequeath to the subsequent crews. And those people could pick up, and have all the distilled knowledge of the past, and add! And the principles of how to- not just ‘it wasn’t dead’ at the time you passed it on, right? What made it alive was the principle of ‘no, we stop and we diagnose as we go.’ And that cultural element, that’s the wiring of the socio technical system, right? That’s the social aspect that we’re going to be a learning group as we go. And it doesn’t make us slowe Right? I mean, it does, it does, in that one race, in that one heat! Yep! You’re slower! And then how many heats are you going to run?

Jeffrey: This is not a one time game! This is an infinite game. And so we get the benefits, you know, for forever, going forward. So I thought that was a fantastic example. And probably the most extreme example of slowification that I could think of.

Case Studies for Communication

Listen to this section at 15:22

Gene: Yeah, I love it. In fact, he spoke at DevOps enterprise two months ago, I’ll give you the link. And he’s actually going to be fielding Q&A at the, uh, unfortunately named GeneCon that’s happening next week. Anyway, just where we get to ask more questions about, like, tell us more about this amazing story and maybe just to, uh, riff with you, just to use this as a counterexample to the horrible, heartbreaking examples in healthcare where the Mrs. Morrison versus Miss Morrison, where they operated on the wrong patient despite 14 clear signals that they had the wrong person, including the patient saying ‘you’ve got the wrong person!’ It’s just as a kind of a very strong example of what happens when you don’t pause in production.

Jeffrey: Yes, absolutely. So fantastic and grounded in the kind of visceral examples, both of these, that people can appreciate regardless of background. And again, this is what what I really enjoyed about it, was to ground it in things that were not technical, that were not domain specific to, you know, lean manufacturing, to software and DevOps and safety culture, and, you know, the other places. It really did, I think, made it very human in a way that I think a lot of people couldn’t understand. It’s not a software book. It’s about human organizations, human problem solving.

Gene: Yeah. Okay. And one more thing on that.

Jeffrey: Please! Yeah, yeah, yeah.

Gene: I love- So there’s like 26 case studies of which, uh, I think 5 or 6 are technology related. So like, they’re the Amazons, the Googles and so forth. But you know, I think the breathtaking example for me of slowification was the famous Google Disaster recovery team. So it’s basically, these are the people who said, ‘all right, when things go wrong that could potentially take out entire data centers at Google, like, how do you make sure that we can actually, you know, function and that we have business continuity?’ And so they actually rehearsed and planned, you know, entire data centers in an earthquake. And one of the things that they found was that there are certain key people that they still relied on, like the Brents in the Phoenix Project. And so they had this, um, they upped the drills by saying, all right, aliens are now descending upon the earth, and they’re kidnaping key engineers! Now recover! Hahaha!

Jeffrey: Hahahahaha!

Gene: So now just like the example of Adam Traina and the MIT sailing team, you don’t have, you know, Brent around. You have to rely on the tools that Brent left behind. And by doing that, this is where they found those latent defects and the vulnerabilities in the system. And it takes a lot of courage for leadership to be able to say, ‘if we really care about this, we have to do drills like this and maybe even in production.’

Jeffrey: And the principle that you’re describing here, especially from the beginning, you just added, is that you’re learning, you have deliberate learning that’s set aside, your carving time out to learn that is not always in production. How do we learn from a deliberate experience as opposed to whatever happens to occur in the routine part of our jobs? Right. Because I have heard people say that like, well, don’t we just learn? In fact, there was a blog post not long ago from I think it was Jason Fried from Basecamp- 37signals. And he’s said basically like ‘ahhh retrospectives… there’s too many retrospectives. If you’re smart, you just learn from experience.’

Jeffrey: Um. You know, I kind of appreciate what he’s saying, and there is learning, but it’s not the same kind of learning that you get from this kind of deliberate practice, and this carving time out. It feels like you’re going slower because you’re making the time to learn rather than do something, quote unquote, productive.

Gene: Right. And to tap a phrase that I’ve heard you say so many times, quoting Spinal Tap, you know, ‘so what does it look like when you turn it to 11?’

Jeffrey: Yeah.

Gene: And, you know, that’s of course, Netflix Chaos Monkey. Right. It’s like, okay, we are now going to, uh, deliberately turn off VMs in production during office hours so that developers are going to get very good at ensuring that the services that they put into production can survive VMs just disappearing, right? Which is how they survive the first big EC2 failure of the AWS East One Availability Zone. Um, so when you really care about it, you don’t just plan and prepare, you actually do drills in live production to see if you are as good as you think you are.

Jeffrey: That’s right. And now and, I want to connect back. People in our audience may have heard about these things before. They’ve heard about Chaos Monkey, maybe even Chaos Gorilla, and they thought, ‘oh, it’d be great to work in an organization where leadership supported that.’ And I think this is where I hope that this book can come in and play a role of abstracting the conversation so that you don’t need to go convince, you know, your head of product or you know, the head of sales. ‘You know, what we need is we need time for Chaos Monkey.’ You know that that doesn’t I haven’t found that to be universally effective. It works some places, but not everywhere.

Jeffrey: But to say, ‘look, we need to slow things down, to have some deliberate learning so that we are resilient, so that we’re effective so that we can improve our performance. We want to go faster.’ And here, you know, this is what I find, when you tell people ‘we want to make sure we’re more effective and faster so we can deliver more.’ Now you have some people’s attention and you say, ‘look, and here’s evidence about how slowification-‘ Now this non-technical non-jargon. It’s not from software. It’s not from lean word, and language, I think can really help that conversation. That’s my hope that for our audience they’ll be able to take that forward.

Gene: And I’m right there with you. It would be my- nothing would make me happier than to have slowify, to be kind of the vernacular, to be a shorthand to say, ‘oh man, look what’s happening. Like we need to slowify,’ and have that be just a shorthand to say, ‘oh, I know what you’re thinking. Uh, absolutely. Uh, because this is not tolerable.’

Jeffrey: Yeah, fantastic. You know what? Um, Gene, this has been so fun, but I’m looking at our clock, and we’re getting getting about time. I know this is a lot to ask, would you be willing to come back one more time for one more episode? And we can, I think, get through simplification and amplification. Would that-?

Gene: Absolutely.

Jeffrey: All right.

Gene: Absolutely.

Jeffrey: Fantastic. Thank you, Gene, and listeners, hope you’ve enjoyed it as much as I have. We’ll be back next week. Same bat time, same bat channel. You can find us a agileconversation.com, where you’ll find our Twitter and our email address. And you can get ahold of us, find past episodes, and of course, you’ll find us in all your favorite podcast listeners and hope to see you next week.