This is a transcript of episode 360 of the Troubleshooting Agile podcast with Jeffrey Fredrick and Douglas Squirrel.
How do you make sure that critical information is preserved over time?
Listen to the episode on SoundCloud or Apple Podcasts.
Reverse Corporate Espionage
Listen to this section at 00:14
Squirrel: Welcome back to Troubleshooting Agile. Hi there, Jeffrey.
Jeffrey: Hi, Squirrel.
Squirrel: So we have a great story this week. It’s one I found, as usual, poking around on Hacker News. And you had some really excellent examples, from both of our experience, at a company that was acquired of exactly this story occurring, and we have some interesting challenges. Plus I have clients who have had this happen to them. So let’s talk about that story, okay?
Jeffrey: Yeah. Let’s do that! But I do want to give our listeners a warning up front that unlike a lot of episodes where we introduce a problem or a symptom that people are experiencing and then say like, ‘oh, that’s funny. Now here’s how you can troubleshoot it and diagnose it.’ Here’s one where I’m not convinced we have like a full solution, haha.
Squirrel: Nope.
Jeffrey: So if people do, as they listen to this, have one. I would love to hear it. But let’s go ahead and talk about the problem of the loss of institutional knowledge. And it’s very funny phrase here, in this story, of “reverse smuggling” or “reverse corporate espionage”. The story was dealing with a petrochemical plant. And the person writing this in 2011, I believe, they talked about how they had retired and everyone else who had worked on this particular plant in the 80s, that made a certain polymer had retired. The company decided that they wanted to go and, you know, have more of that polymer and was like, ‘well, how can we go and expand it or duplicate it or something like that?’ So they owned a plant that was functioning, but they had lost the institutional knowledge of how they built the factory, the plant that they have. And there are amusing details in here.
Squirrel: And if this sounds like some software systems that are our listeners might have worked with or that their engineers might be complaining about, you’re right! You got the idea! But this is with physical stuff. It’s like an actual giant tank that’s full of, I don’t know, selenium or something. And there’s something else that’s liquid helium, and you mix them together, and you get helenium or whatever.
Jeffrey: Yeah.
Squirrel: This plant is actually working and making stuff and no one knows how.
Jeffrey: Yes. That’s right. And it’s kind of the funny that people often say like, ‘well, software engineers are so undisciplined, they’re not like real engineers.’ But I think this is a case of like, no real engineers and software engineers suffer from the same problems. We have these systems that function, but we don’t always understand how because the knowledge gets lost.
Jeffrey: And so the person was in the amusing position of having to smuggle knowledge back in. He was hired essentially as a consultant. And, by chance, the company had lost a lot of the documentation for the plant, despite efforts to digitize and standardize their information storage.
Squirrel: But don’t worry, this guy actually had kept some of the documents, which he wasn’t supposed to do.
Jeffrey: That’s right, exactly! It was kind of a funny thing, it was a workaround for the institutional problems at the time. As they were going through various mergers and putting libraries together and digitizing, the engineers at the time were struggling sometimes to get copies of the documents they needed, so they, strictly against their rules, created local copies. When he retired, he took some of them with him. So they ended up in a funny position where officially the company had the documents, and he did not. But in reality, he had the documents and the company did not. So they were kind of a challenge of like, how do we smuggle this stuff back into the company? And beyond that, it wasn’t just documentation. There were also trade secrets that officially the company knew, and no one outside the company should have. But in fact, he knew because he had helped create them. He had his name on some patents, but the company had lost knowledge of those trade secrets. So how do you get the knowledge back into the company from the outside?
Jeffrey: I thought this was fantastic, not the least of which because I have experienced this problem in many places, and it goes along with one of my favorite phrases in software, which is— people are often complaining about “legacy code”, and that’s the description —a legacy is something valuable that you inherit. So in a sense, it’s only successful products and projects that will ever experience the problem of this institutional decay. Most of them will never survive or be valuable enough to suffer this problem. It will only be the really valuable ones that do, so if we’re really lucky, the things that we’re working on today might one day suffer this problem. But I just thought it was a very funny story, and I immediately thought of many different examples in software that I’ve experienced personally that were similar to the story here.
Lucky Saves
Listen to this section at 05:13
Squirrel: Oh, absolutely. And if listeners would like to think of some other physical examples, we have loads and loads of old radio and television programs that have only survived because either an engineer took the discs home when they were supposed to be wiped. Of course, no one thought anyone would want old episodes of Doctor Who, or Howdy Doody, or Fibber McGee and Molly, or some other ancient program, but, of course, they’re extremely valuable today. They turn up on eBay in places and at auction because somebody happened to keep one and take it home, or some intrepid listener actually recorded it, which I don’t think you were supposed to do. And it was really hard. It was like reel to reel tape and things. So people went to great efforts to preserve these things.
Squirrel: We built up this software, well, in our world it’s software, but this kind of archeological record, which is really incomplete. We’re trying to discover what was happening this very short time ago, within human memory, there are people still alive who watched the first Doctor Who, but they can’t recreate it! Those people have the memory, but the physical artifact is the only thing from which we can reconstruct the actual entity, the actual thing. And we kind of have it like we do sherds of pots buried in the ground from thousands of years ago. There’s this very similar rediscovery and imperfect reconstruction of something that you used to have. But those things are tremendously valuable!
Squirrel: Jeffrey, you were telling me about a program that I didn’t even realize people were still using, that your company still has, that is still running! And I was dumbfounded that it was still going. But what happens when it breaks?
Jeffrey: Well, that’ll be a real challenge for us because it turns out we have very few, I think one or two, people at the company who actually spent any time working on this product. Which, at the time that I joined the company, 13 years ago, it was one of the two major products of the company, and half the company was working on it.
Squirrel: I remember, I remember when it got started, I helped write the first version!
Jeffrey: Now, 13 years later, it hasn’t been under active development for, oh, 11 or 12 of those years. As a result, the institutional knowledge has really decayed. And for me, what makes this interesting is, I worry a lot about and think a lot about, how do we have generative organizations with information flowing and information going to the right place in the present tense? How do we collaborate now? But part of that is thinking a bit about where does this information go in the future?
Searchable Archives
Listen to this section at 08:29
Jeffrey: So there are steps that I try to have people do now, which is, for example, moving conversations out of email, which is tied to individuals inboxes into common repositories. So I tend to be advocating that if you have Slack or Teams or something like that, that this is a better searchable archive. But there is the challenge that what is the searchable archive changes over time. And so part of it is, if I look back over these 13 years, there’s been a progression of going from like a wiki with items in file shares, and links to file shares, to links to Google Drive, to the wiki no longer being used, and everything from Google Drive being migrated into SharePoint.
Jeffrey: So if I’m looking for old information, ‘why did we make that decision eight years ago?’ I might be able to find a wiki page on it. It’ll say, ‘oh, here’s the reason documented in this document,’ with a URL to a Google Drive that’s no longer active. And that file does exist somewhere in SharePoint, but it can be very difficult to find. The article, that we’ll link to in the show notes, that talks about the petrochemical plant, is talking about the loss of knowledge over multiple decades. But I’m thinking about things that I personally wrote or authored or edited that just have been sort of misplaced in the evolution of different storage technologies over a decade. It’s a challenge here, we how do we design our information systems around this process. So that’s why I thought this was worth talking about.
Squirrel: Absolutely. So I’ll just note a couple other examples. When I was talking about Doctor Who episodes and other things, there’s also old video games and things of that kind, old data tapes from the era when big mainframes had data tapes going around inside them and so on, you’ve seen pictures, perhaps. When you say formats, Jeffrey, you’re right. Sometimes the actual physical, magnetic record is not something we can read anymore because the machine which that tape would go around, or the player that would play the cassette tape, or the videotape, or whatever it was that was being used in 1954 or something is no longer extant. There are archivists who have to work really hard. You could look these up. There are inspiring stories of retro computing and ancient format archaeology to build hard drives and projectors and other things that can actually extract this information just based on, maybe, pictures of the device itself. Because the device no longer exists, no one has actually got the format. And these are all for things that are under a century old.
Squirrel: The other thing to remind listeners about is this is not a new problem. For example, listeners might be aware that we have something like three books from the Mayans because a whole lot of people came along to conquer the Mayans and burned all their books because they were heretical. The Library of Alexandria was burnt. We have lots of records of ancient Greek philosophers, only because somebody decided to quote them in some other book, or write on the back of a piece of paper or a piece of papyrus that they happen to write on.
Squirrel: All that kind of thing has been going on for a very long time, and that actually gives me some hope, because humans seem to have figured stuff out! And we do have some propensity for rediscovering stuff, even things like Greek Fire, which we haven’t understood for a long time, at least there are good guesses for what these are for and how people might have created them. So yes, there is knowledge that gets lost over history, but an awful lot of it can be reconstructed and often rebuilt and rebuilt in a better way. So that’s my optimistic take. But I’m with you, Jeffrey. I’m not sure what we could do today. I don’t think I’m going to convince a whole bunch of my clients suddenly to stop using Slack and to carve cuneiform into tablets so that it’s durable a thousand years from now, their discussion of how to set up the checkout page. I would like to convince them of that, but I think durable storage and permanent knowledge retention is a problem that human beings haven’t solved yet.
Jeffrey: Yeah, absolutely. And I agree with your optimism for the species, that this won’t be a problem for them. But I will say from very recent experience, I could feel this sort of cliff of memory where, doing some archaeology earlier in the year, in some cases, we were able to find, relatively easily, video recordings or teams recordings of meetings where certain decisions were made and listen to the discussion. And so this question of like, ‘who made this decision and why and when?’ We had an absolute record and that was very, very useful! And then items just a couple of years before that, the same equivalent artifacts had existed and were lost. And while the species will be okay, if it’s my job to try to fix that software, and I want to know why it is the way it is… I’m thinking of my future self in a couple of years! How can I make my life better for myself, in a couple of years? The species will be fine, but I just want to have fewer headaches.
Squirrel: Well, and I certainly would like you to have fewer headaches, Jeffrey. Maybe what we could do is, as we always do, ask our listeners for their ideas. What are you doing today to make your life better, or your successor, or your successors’ successors’ successors life better in your organization with your software? So decisions and reasons for code being the way it is, and thoughts about why you make certain product choices, how are you preserving all of those for future generations? Perhaps future generations of yourself who will have forgotten everything that you know today but still really, really want to know. Are you going to rely on smugglers to bring some information back that they shouldn’t have retained? If that’s your preservation plan, I would recommend changing it. But we’re interested in what you might have, what you might be doing today to deal with that. Get in touch with us, agileconversations.com is the place to go. And of course, the other way to keep in touch is to come back again next week when we’ll be here with another episode of Troubleshooting Agile. Thanks, Jeffrey.
Jeffrey: Thanks, Squirrel.