This is a transcript of episode 160 of the Troubleshooting Agile podcast with Jeffrey Fredrick and Douglas Squirrel.
A reader asks whether following the “bad idea” advice from last episode can lead to an unrecoverable local maximum, sentencing the team to endless patches and obscure bugs. Squirrel and Jeffrey have different takes on this based on their ancient greybeard stories.
Show links:
Listen to the episode on SoundCloud or Apple Podcasts.
Introduction
Listen to this section at 00:14
Squirrel: Welcome back to Troubleshooting Agile. Hi there, Jeffrey.
Jeffrey: Hi Squirrel.
Squirrel: Boy, are we going to be tired in March. There’s so much going on, Jeffrey you have something very exciting coming really soon.
Jeffrey: Yeah, I’m really excited on Saturday, this coming Saturday, March 6th at five pm GMT, I will be on Clubhouse with my friend Julio and we’re going to be talking about Psychological Safety, Does Your Team Have It? We did the first Clubhouse event last Saturday on kind of short notice. It was a lot of fun and I think it’s something I’m going to keep experimenting with so next up Saturday, psychological safety. But that’s just the start. As you know, we’re going have a lot more than that, I think coming up next week. We also have VSM DevCon and people can find the link about that on our events page and they’ll find other things there like, we have a conversational dojo next week and then later in the month will be at a few different conferences. And at the end of the month, the last minute we currently have scheduled is a workshop that you’ll be running on decoding tech talk. Can you tell us about that, Squirrel?
Squirrel: Sure. So if you know people who or if you yourself are a non-technical person who finds technical people very difficult to talk to and you listen to our podcast to find out how to have better conversations with them, this workshop is designed for you. It’s to help you figure out what are all the things that would really help you to build trust with your tech team, to have a profitable conversation that leads to revenue based results and how to have an interaction that is not frustrating and confusing, which I have a lot of non-technical people telling me. I just had one earlier today telling me how difficult he found it to talk with his CTO. So if you’re the CTO, you might refer to your CEO and say, “hey, you might want to come to this.” If you are that non-technical person yourself, you might find it interesting and helpful. It’s on the 24th of March. You’ll find details on that one on DouglasSquirrel.com And you’ll find all of these, including that one on conversationaltransformation.com/events. OK, there’s our parish notices everything that’s happening. And we’ll try to let people know about more of the exciting March events as they come up.
Listener Comment
Listen to this section at 02:30
Squirrel: We wanted to turn to a listener question this week, we really like getting listener questions. We really like hearing from people who are listening carefully and have thoughts about what we said. And we had a listener named Eric who had a particular concern about last week’s topic, which was The Power of a Bad Idea that maybe it would be a good idea to start by doing something that wasn’t optimal, that wasn’t a good thing that you knew had flaws and defects and see where that took you do iterations from it to improve.
Squirrel: And here’s what Eric says, and he gave a lengthy example in banking and then he said, ‘well, actually, I’m not in banking, but my team has done this’ and what he said “was we keep bumping into out of order processing of records, but we always find a way to patch it. And we go on to add new features and new data flows.” I’ve been here, Eric. I hear your pain. I’ve been there. “Fundamentally, our system doesn’t handle concurrency. We’ve talked about adding properly support transactional concurrency as an upcoming feature from the product owners perspective, they just want to get some data flowing.” So I can really empathise with that product owner who wants to see continued results. And I can really empathise with Eric, who says, “Oh my God, I have to deal with another slew of concurrency bugs and out of order updates and oh my gosh, how am I going to handle that?” So what do you think about that, Jeffrey? How could we respond to Eric? How could we suggest that he might use the power of a bad idea? Or maybe this really was a bad idea.
Jeffrey: One thing I just want to credit Eric first, not only do we like people writing in, but we especially like it when they tell us that, yes, they’ve ordered the book and yes, they did actually put a sheet of paper and summarise a conversation. So I think it’s worth noting that, you know, he did kind of bribe us to get his question answered by saying that he’d actually done a conversation analysis. But putting that aside, he brought this up as an example where looking back at our conversation last week, we talked we made a distinction between iterative development and incremental development, which is to, whether you’re starting with something rough and then reworking it and refining it, refining it, as opposed to where you have a plan and you’re incrementally building pieces as opposed iteratively reworking it.
Jeffrey: And he said, “I really like that. The problem is that we can end up in a local maximum solution where that local maximum that we hit, that we’ve evolved our system to, is fundamentally not good enough.”
Start-Ups and Hill Climbing
Listen to this section at 05:01
Jeffrey: The example he gave us, was one around the transactions and what this reminded me of right off was a very interesting story that I heard from a product manager. I worked. Without a Start-Up many years ago, and he was at the Start-Up because he’d been essentially brought in by the VCs, he was introduced by the VCs and they introduced them because they had a good experience with him at a prior company. And the good experience was very relevant here. It was a problem where the team had entered. It ended up in a local maximum, as he described it, and they had done this iterative development and ended up with a solution that was working. And but the problem was it just funds. It was not scalable enough. It was it would not scale. It would not handle the load they needed. And in this situation they were at is in this procurators position was like this was a fatal flaw, that fundamentally the product was not suitable for the market. Architecturally, it was it was never going to work. And, in fact, that the VCs ended up shutting down the company. And that sounds very extreme. It is very extreme. But they were they were responding in their minds. They were able to respond early and cut their losses because they had this tip off that this approach was fundamentally not going to work. So that’s that’s the story that came to mind for me as I read Eric’s email.
Squirrel: Right, well, I disagree with those VCs I’m not convinced at all. Now, of course, I wasn’t in the situation. It could be that this team got onto such a local maxima that they really couldn’t recover. But I’m doubtful about that. And I would double down on the power of a bad idea here because of some of the experiences I’ve had. So I can think of an example. As you were talking, I was remembering being the CTO of a Start-Up, well, it’s a little further along than a Start-Up. We were an E-commerce company and we had built our own version of Shopify or Magento, and we had some special circumstances, we had very, very fast turnover of stocks. So those systems at that time were not capable of handling it so it wasn’t completely stupid to build our own, but it was mostly stupid because it was pretty non-scalably and there were an awful lot of problems with it. And we didn’t have, for example, any sort of framework that would allow us to consistently train people to have consistency in our pages and so on. And I remember turning up and the team said, “hey, we’ve kind of hit this local maxima.
Squirrel: By the way, if people aren’t familiar with that term, it comes from a mathematical strategy called Hill Climbing, where you look for where you are and you have a little score of like whether I’m better or worse and you’re trying to go up a hill and you draw a map of where you are and you say, “well, this thing that’s a little bit distant from me is a little bit better so I’ll keep going up the hill. And you can imagine if you were actually climbing hills trying to get to the top of Mt. Everest but you start out on the hill that’s near my house. That’s maybe 100 feet high. You’ll wind up at the top of that hill and then you can’t get to Mt. Everest because every direction would be down. So you actually hit the local maximum, but you can’t go down and then back up again to get to the tall mountain. So that’s the idea of a local maxima. You get trapped because you can’t iteratively improve because every direction from where you are is downward and worse. And the team said “this is where we are. We have this horrible spaghetti system and we really can’t work with it.” And I said, “So what are you doing about it?” I had just arrived. So I wanted to understand the team strategy and they said, “don’t worry, Squirrel, we’ve got it all under control. We’re fixing this system and we’ve made a whole plan.” And I said, “great, how long will the plan take?” And they said, “one year.”
Squirrel: “Wow, that must be one heck of a plan. How are you doing on the plan?” And they said, “well, we’re one year into the plan.” And I said, “great, so we must be almost done.” They said, “no, we’re a year from being done.” I said, “hang on, this isn’t going to work.” And this is probably what the VCs were thinking. They were thinking to themselves, well, we could go and try to make this scalable and throw away more of our money, or we could just close it down and send this guy to the next Start-Up and let’s see if that works. But the thing was, we didn’t have that option. We weren’t VC, so we didn’t have a bunch more money and we didn’t want to shut down the e-commerce business. So we came up with a much better solution, which is the sort of thing that I really think could apply in Eric’s situation. But he’s welcome to come back to us, maybe he can come on the podcast and we’ll discuss it with him. If you disagree, Eric, tell us or if other listeners do, please tell me but what we found was that we could incrementally and iteratively move toward the better solution. So in our particular circumstance, we had a nice division of our application.
Squirrel: There were essentially pages on which you could buy stuff and Those pages were rendered in a particularly ugly and horrible way. But we could introduce a framework. We happened to use the symphony framework, but you could have used any mechanism. And what we could do is we could make it so that all the pages were rendered in the same way they used to be. But with the symphony framework around them and then our process, we called symphonisation. So we were able to take each page and say, OK, this page, we’re going to symphonies, we’re going to make it run through the symphony framework. So at least we understand what the heck it’s doing and it will be consistent and standard. And we could do that page by page as we worked on each one. And that led us iteratively move our system from ugly horrible spaghetti, can’t understand any of it, to,ugly horrible spaghetti that at least runs in a framework to, this page is now cleaner and more understandable and this next page is cleaner and more understandable. And of course, the way that we did that is we worked on different pages as people asked us to work on them, as the features came in and were prioritised. And that meant we touched the ones that were most important first because people kept asking for improvements.
Squirrel: I don’t know if we ever touched the about page, which told the story of the founders, that one worked OK, we didn’t have to change it. We didn’t add new founders every week, so that didn’t matter. But say the handbags page, we had new handbags every week and people wanted to buy them and we wanted to improve the page so we would make improvements on that page. And that was one of the first to be symphonised. So it meant that we made the improvements in the parts of the system that were most affected first and that kind of iterative improvement. Of course, I haven’t seen Eric’s system, it may be absolutely impossible in his situation. And I certainly believe if you were, say, building a rocket ship, that this might not work for you. Most of us are not. Most of us are building things that are somewhat more loosely coupled and less interdependent, and an approach like the one I’m describing could help you iteratively move out of the local maximum by working with pieces of the system to move them ahead. And so I wouldn’t have shut this start-up down. What do you think, Jeffrey?
Taking an Iterative Approach
Listen to this section at 11:46
Jeffrey: I think it’s a great idea. Hearing you describe it, reminded me of Martin Fowler describe this thing that used to be called the Strangler Application or the Strangler Pattern.
Squirrel: Oh, yes.
Jeffrey: Now renamed the Strangler Fig Application.
Squirrel: I hadn’t caught up with that one yet. OK, that was a 2019 renaming.
Jeffrey: So fairly, fairly recent, link the show notes, of course. And in general, what I really liked about your story here was the technical creativity. I tend to agree with you that almost always there’s a way for us to pull ourselves out of this. Now, we could argue, I think one important part of what you were doing is you’d say, like, “were you hill climbing?” I imagine you might be putting some effort in that for a while doesn’t give you an immediate ROI.
Squirrel: Yes, that’s true.
Jeffrey: And so, if you could strictly say “OK, we’re making investment.” But look, that’s fine. There’s nothing about the iterative approach that says it should be a hill climbing algorithm, it often is but you stuck at a local maximum, guess what? Find your way out of it. I think that’s for me, the short version of it, of what you’re describing. The other thing I will mention, people who’ve seen my picture know that I’m a greybeard. And actually that’s in a sense your fault, Squirrel, because when I came to work for you, you said you were looking to hire a greybeard. So I said, “well, I need to grow a beard.”
Squirrel: Yeah. And it was great, which was very convenient.
Jeffrey: Yeah, exactly. Now, the thing is, what that means in practise is I’ve actually been around in the industry for a long time. And in particular I’ll say I was in the software industry in the 90s and this is one of those ‘kids these days’ things.
Squirrel: Yeah. Get off my lawn.
Jeffrey: Kids these days, they read what we all, these old timers, wrote back in the late 90s and early 2000s about things like YAGNI and don’t do big upfront design and things like that. And the thing is, it’s one of those things about context. What we were talking about in those days was when you had the design phase,the design phase of your schedule was going to take two to three months and then you were going to go from the design phase to the requirements phase or something like that were these phase gate processes where the design process was very long. I do think there’s a case, I will say this, where you can harm yourself by not looking ahead to kind of what’s obvious. And so the other part that resonated with me in Eric’s story is people who set off saying, “oh, no, we don’t do any kind of design anymore. We refuse to go to a whiteboard and talk about architecture.” This may seem like an exaggeration, but I will tell you, from personal experience I’ve had people tell me, “no, no, that’s big upfront design, we should just start with a test.”
Squirrel: And people like that often then hire me later to get them out of the mess. So keep doing that, folks. Great! It’s a super business model. No, it’s not. Please, please don’t.
Jeffrey: So I will go back and say, when we’re talking about, fans of Agile and we’re fans of iterative development. And I would also say, that it helps to understand a bit of context. And in the past we’ve talked about, you know, “non-functional ideas” and the idea of understanding a bit about what we expect the system to do in the future and to say, you know, do we have a reason to believe that what we’re doing is reasonable? Now, it’s quite possible that you will end up on a local maximum anyway and if you work long enough in the industry, I guarantee it will happen to you. So you’re not going to avoid all problems, but you will avoid a certain amount of them. The good news, as you just described, is if you do, you can probably find your way out. If you’re willing to go think about how you would incrementally approach it. Now there’s a different problem, which is, you know, how do we build and have the discussion about now is the time to make that change. But that’s going to have to be, I think, a problem for a different episode.
Squirrel: Indeed. I just wanted to add one more thing about the Eric situation, that I think the iterative approach I was describing could really fit well for. He says there’s this tension and it goes exactly to this point you’re making Jeffrey about ‘how do you know when to do it?’ There’s this tension between what the product owner wants, which is get some data flowing and developers who say, “oh, my God, could we just get some transactional concurrency?”
Squirrel: The way we squared that circle in this case was to take the transactional concurrency in our case, it was the kind of spaghetti code that the problem that we were trying to fix and find a way to atomise it and do little pieces of it alongside keeping the data flowing and selling handbags. And that’s an art, that’s a tricky thing to do. Maybe that’s something that we should talk about again. So we’ll we’ll ask listeners to to tell us if you’d like us to cover that in a future episode exactly how. That that’s the ultimate trick, if you can manage it, is that you can do a little hill climbing, so you’re still trying to improve in a certain area, but you’re also making an investment in the long term improvement. If you can split it up fine enough, you can manage to do both without as much pain. So we’ll we’ll leave Eric with that idea. He and others are very welcome to come back and argue with us.
Jeffrey: Of all the topics we’ve done, this is one that I really expect people to argue with us and say, “no, no, you don’t understand, here’s a situation that could never be done any other way.” And other people who said like, “well, of course, we can start with just a test. We don’t need that whiteboard design stuff.”
Squirrel: Exactly. So perfect. We’d love to be attacked from both sides. If you’d like to do that, you can find us on Conversationaltransformation.com. That’s where you’ll find the events we mentioned at the beginning. On the events page. You’ll find our book, Agile Conversations. You’ll find our email and Twitter and I don’t know, carrier pigeon, anything else you can find us with. Feel free to get in touch with us that way and let us know what you think and how much you disagree with us. That would be fantastic. And we’ll see you again next Wednesday.
Squirrel: Thanks, Jeffrey.
Jeffrey: Thanks, Squirrel.