This is a transcript of episode 212 of the Troubleshooting Agile podcast with Jeffrey Fredrick and Douglas Squirrel.
Squirrel and Jeffrey continue discussing what to do with a giant backlog of work for your tech team, and suggest ways to think about paying off your tech debt and thin your features for faster feedback.
Show links:
Listen to the episode on SoundCloud or Apple Podcasts.
Introduction
Listen to this section at 00:11
Squirrel: Welcome back to Troubleshooting Agile. Hi there, Jeffrey.
Jeffrey: Hi, Squirrel. This is part two of a great listener question we started last week. We previously discussed the backlog of portfolio items with 9 to 12 expected months of work and five years beyond that!
Squirrel: A five year backlog! What are we going to do with that!? Delete it, I say.
Jeffrey: After, of course, having difficult conversations. That’s something you and I tend to come back to again and again for some reason, these “conversation” things.
Squirrel: Somehow we manage that. But our listener had a second half of their question, and I’ll refresh us on that question.
Jeffrey: Please do.
Squirrel: ‘The team were forced to cut corners to meet a deadline they did not commit to, and this has caused a huge amount of technical debt. The debt never gets dealt with because of the first problem.’ That being the five-year backlog. The first problem means that they get more and more work so they don’t deal with the technical debt. And then the technical debt makes them go slower, which makes the first problem worse. So our listener understandably says I’m stuck between a rock and a hard place. What can I do to break out of these bad patterns? Remind our listeners what advice we gave last week, because I bet we’ll be building on that with this second question about cutting corners.
Jeffrey: Last week we talked fundamentally about getting aligned across the team. Importantly, the team was not just the technology team. It includes the senior staff who are committing to work and bringing in the person we called the gold owner: the person who has ultimate personal responsibility for this beyond the team and the people who are putting demands in. They might be as high up as the CEO of the company.
Squirrel: You need to be able to discuss things like, ‘Hey, our backlog is five years long, is planning for that really an effective use of our time? We seem to be getting slower as a result of all that extra planning and new work coming in. Maybe we should stop the madness? Maybe the emperor has no clothes. What do you think?’ Once you can get that discussion at an appropriate level, then you can start to break out of the bad patterns. That was our advice last week. But this week we’ve got this further problem, which is kind of a consequence of the first one where as a result of this tsunami of work coming in, there is now this big extra burden of technical debt, which is making the problem worse. What should we do about that?
Cutting on Credit
Listen to this section at 03:26
Jeffrey: You get this sort of vicious cycle where the technical debt is making things slower, which puts more pressure on and draws more pressure from the senior staff, which likely makes them more firm about setting deadlines, which probably causes more technical debt. You get this terrible pattern.
Squirrel: A downward spiral.
Jeffrey: Exactly. When I read this, I was wondering what kind of corner-cutting is going on, because sometimes corner-cutting can be good. It doesn’t necessarily mean having technical debt. We’ve talked before about feature-thinning, when we’re going to cut corners by doing less, but the things we do will still be done well. I wonder how much of this corner-cutting that was being done was in coordination with the people who own the deliverables, the people who are making the requests. We’ve had great success collaborating with business partners to present some trade-offs, so we can give them the simplest version of what they want, and it will work. But they won’t be able to do all the other things they would normally expect; that kind of cutting corners, that deliberate choice of prioritization can be a really powerful tool and a great way of reducing scope to meet deadlines. When I hear people talk about cutting corners I worry about unilateral decisions made by the development team without involving the business stakeholders. What was what was your thought when you heard this second problem?
Squirrel: Well, I’m a big fan of cutting corners. You and I might disagree on this, Jeffrey, I might be willing to cut corners in a way that does create technical debt, in a way that creates real problems and isn’t scalable or a good solution. We talked a few weeks ago on the podcast about a case where I told an organization to return TRUE from their authentication routine. Every person in the world could log in with any password and get in. Before our listeners fall off their chairs, the situation was one where logged-in users could read a little data and make an offer of work. So the worst that a malicious entity who got in would be able to do is to make a large number of offers of work, which they of course couldn’t or wouldn’t fulfill. So they might have created a denial of service? But that consequence—agreed to very carefully with the people who would be working with the operations team and the customer service team, the people who who would be handling that problem—was very low in risk and impact, and it was something they fixed within a few days. Cutting that corner allowed them to get to users. Those users were not terribly concerned about authentication. They were concerned about making offers of work. That really broke a logjam for that firm and was an effective way of cutting a huge corner: creating a massive technical debt that you would never want to carry for very long, like an emergency loan from your bank. This debt analogy I think can be misleading or confusing sometimes when people use it inappropriately. But here it applies. There was a case that happened to me once where I needed 18,000 pounds for the span of three days. We were borrowing it for three days against a house that we were about to buy. There was zero risk for the bank because they were organizing the mortgage, but we needed this money for the transition period. We were willing to take on that debt which we could never repay other than by making the transaction happen, because we knew the transaction was happening, and so did the bank. We paid them a small amount of interest. We had the loan for three days. We paid it off immediately when we bought the house and everything was fine. But there would have been no transaction, no mortgage for the bank, and no house for us if we hadn’t done that. I’d count this as an example of a massive corner cut with a massive technical debt, agreed upon with the stakeholders for a defined period and which you will pay off. The exact opposite of the situation our listener is in!
Jeffrey: That’s an interesting story! I don’t mean to quibble, but I would have said that return TRUE case is much like feature thinning. You had that negotiation with the business about what you’re doing, rather than the unilateral decision from the engineering team. In that sense it was a creative solution like, ‘Hey, we’ll let you log in. But it will be fake.’ I think when people talk about cutting corners and technical debt, that often means the team feel what they’re doing is unsustainable code: that they’ve done things badly. It’s not tested, they’re violating their design principles. Things won’t scale, won’t take them forward. As a result, when they come back to it and work on it again, there’s going to be a lot of cruft there to deal with, so the next thing that they do will add to that burden. Whereas having your code that returns TRUE probably doesn’t add a lot of overhead! I would guess that doesn’t add a lot of conceptual complexity.
Utiltiy of Debt
Listen to this section at 09:46
Squirrel: No, it doesn’t, but it’s exactly what technical debt is! I’m going to rant a little bit here, if you go back to the original definition of technical debt, it is an analogy to exactly the kind of financial situation that I was describing, where you might consciously take on the debt and have a plan to pay it off. We have this negative view of debt, right? Debt is a very useful thing. I’m about to buy a very substantial piece of property, and I’m going to go into a huge amount of debt to do it. I wouldn’t be able to do it if I didn’t have the bank there ready to provide the cash I don’t have. That sounds like a great arrangement. I’m going to be very happy about that debt. I think we have this negative view of debt as if it were all unsecured credit card debt at massive interest rates and that therefore all debt is terrible. I want to defend technical debt. When it is gone into thoughtfully as a responsible party, it isn’t a bad thing at all. Technical debt is not incompatible with feature thinning! Last week we had the analogy of widening a road but needing to close a lane to do that work. ‘Is everybody OK with having less capacity for a while until I can make the road wider?’ Similarly, is everybody OK with us having this technical challenge, this thing that’s kind of waiting to be fixed? We’re paying interest on that. We’re having customer service requests about that. We’re having bugs as a result of that. Are we OK with that for a while in order to get a bigger benefit? I think that kind of debt is a good idea.
Jeffrey: I agree completely, I think the heart of both our scenarios is when you’re having that explicit conversation it’s like co-signing for a loan. You’re saying ‘we together have a plan to pay this off.’ But what happens when the technical team does this independently, it’s more like they’re taking a credit card advance but not telling anyone. The debt is hidden from other people. It’s not a negotiation. There’s no repayment plan. It’s going to be there and accumulate in an unsustainable way.
Squirrel: Let’s try to help our listener. Unfortunately, it seems they’ve incurred this unsecured credit card debt. It’s like when some somebody in a marriage comes along and says, ‘You know what, honey? I borrowed a lot of money and I didn’t tell you about it.’ That’s going to be a very difficult conversation. The other party is probably going to be pretty unhappy about it, but I suspect we might tell our anonymous listener to start there. Is that right, or do you have a different idea about what we could do about this pile of secret debt?
Jeffrey: I’ve often worked with teams where there is technical debt. One of the non-intuitive things about it is that very often paying off technical debt can be free…In the sense that if the debt is slowing you down and that slow pace is baked into your current estimates, fixing some of it early can actually have the project go faster. I know we’ve talked in the past about a mutual friend of ours, Paul Julius, who as CTO told his team that they had 50% time: they could spend half of their time paying off technical debt and the other half working on features. This was a big shock to people.
Squirrel: This is a big expansion on the Google standard, which is like 10% to work on whatever you feel like. If I know PJ, he would have been riding them very closely to make sure that they were working exactly to the stat. ‘Let’s deal with this ugly, horrible code that somebody wrote in a panic late at night.’
Jeffrey: He gave this policy after his team had come up with their committed projects for the quarter. They already knew ‘this is the work we’re committing to.’ To then be told ‘by the way, spend half your time on technical debt,’ people were shocked! How could that possibly work? But in fact, they finished all the work they’d previously committed to because they saved so much time from paying off the technical debt. I’m saying technical debt here in an expansive way. It wasn’t just code, it was systems, putting automation in and around delivery and deployments, things of that nature. By improving the system, they were able to recoup their investment very quickly such that they were more able to deliver on all their deadlines despite spending the time doing this. So technical debt can be a lot cheaper to pay off than people realize.
Squirrel: It’s very, very helpful to make sure that the rest of the organization knows that you are doing that payoff. I’m sure PJ also told people, ‘by the way, I’m telling the team to spend 50% of their time here,’ and the benefit of that is the rest of the organization can then participate in the decision and be aware of what improvements are happening so that if somebody hears in the lunchroom, ‘Yeah, I was just fixing this code,’ they won’t think, ‘Oh my God, they aren’t working on my feature.’ So we return to communication here: it’s very important to let others know what your intent is.
Theory in Action
Listen to this section at 16:17
Jeffrey: I suspect our listener is familiar with most or all of what we’ve been talking about here. So what do you do as a technologist, when you already understand all these dynamics. As you said, they feel they’re stuck between a rock and a hard place. They know what they need to do, but they don’t see a way out of it. What’s different about what we’re saying? In a sense, we’re coming back with pretty familiar elements in the Agile toolkit. What’s different about what we’re saying versus what you find with people who are frustrated, who feel stuck in their organization?
Squirrel: Fundamentally the thing that we’re saying is, have the difficult conversation, make the topic discussable, share where your challenge is and what your intent is. The problem is of course, that’s very threatening. I’m reading between the lines here from our listener, but this doesn’t sound like an organization in which that’s prevalent. We’ve got the senior staff off committing to work, imposing deadlines, making promises, certainly not modeling what we are describing. You, dear listener, have to be the first person to address that. You have to be the first one to violate the norm and perhaps create a new norm. That’s threatening. That may not work out. We’re not promising that will suddenly make everything better and the whole organization will suddenly begin functioning differently. But the alternative is being stuck between this rock and this hard place. If you were to deal with the fundamental problem of communication and trust and interaction with these external forces that currently seem unchangeable, then I think you would find either that they really are unchangeable—in which case you were screwed anyway—or the situation is changeable. There is something that you can do about it. The good news I have is that almost alawys, this is successful. Things are more changeable than you think they are. Otherwise I wouldn’t do this in my consulting engagements.
Jeffrey: I completely agree. What ultimately stood out to me about this question: ‘what can I do to break out of these bad patterns,’ was that I part. If you look at just the technical side, ‘I’ can’t do anything. I can’t make senior staff stop committing to more work. I can’t make them not want to outsource. I can’t make them stop asking for these estimates. There’s a lot of things here where the world would be better if it were different, but I’m sorry, you have the world you have and suffering comes from arguing with reality. What you can do is is have these conversations and find out if, as a group, as a team, bringing in these senior stakeholders and gold owners, if as an organization, you’re willing to have the conversations about where you really are, what’s really possible. ‘Can we deal in reality?’ If you’re able to do that, then you can move towards much better dynamics, improved actual teamwork between all the different participants.
Squirrel: You can get to 50% time, as with PJ.
Jeffrey: Exactly. You may come up with all these improvements in the relationships and better dynamics, and that’s the payoff. That’s wonderful. And if that’s not possible, then at least you’ve learned that. You’ve done what you can. Then we’re at that classic line from the early days of Agile. Change the organization you’re part of or change which organization you’re part of.
Squirrel: Our listener did refer to a tight labor market, so there’s other opportunities. I think there’s plenty of opportunity here. For example, you have senior stakeholders who care enough to continue to make promises on your behalf. There’s some motivation here, energy and action and probably some things that you can do.
Jeffrey: I agree. I think a lot of people are too quick to quit out of bad situations. That’s a fair choice. If people don’t want to have these conversations, they want to work in a organization that already has that figured out? I don’t blame people for that, but I think it can be really fun and really rewarding to work through these issues in an organization and to feel that you’ve made the system better. That maybe explains why I end up doing this consulting stuff.
Squirrel: That’s why we record the podcast every week to help you guys to do exactly that.
Jeffrey: Yeah. So it may be challenging, but I think there’s great possible rewards, both as an organization and personally a level of satisfaction when you’re able to change the dynamics in the organization. That can be a tremendous payoff. I think it’s worth the risk.
Squirrel: Wow. Ok, well, I think we’ve filled the ears of our anonymous questioner. Very happy to hear from any and all of you with follow-up questions, disagreements, so on. Thanks, Jeffrey.
Jeffrey: Thanks, Squirrel.