This is a transcript of episode 181 of the Troubleshooting Agile podcast with Jeffrey Fredrick and Douglas Squirrel.

If you’re already struggling to hit targets, don’t add more! Recommendations for improving performance without overcomplication.

Show links:

Listen to the episode on SoundCloud or Apple Podcasts.

Introduction

Listen to this section at 00:13

Squirrel: Welcome back to Troubleshooting Agile. Hi there, Jeffrey.

Jeffrey: Hi, Squirrel. So Squirrel, as we talked about last time, you’ve been tweeting a lot more recently.

Squirrel: I have, but I’ve not been on Twitter! It’s wonderful. I’ve discovered a fantastic thing: you can tweet from the command line, so you never have to look at Twitter. Twitter is now write-only for me, and so much less distracting.

Jeffrey: Well, you do look at it sometimes because that’s your topic notebook, but it’s like a notebook you don’t need to open to enter in. It’s been fun for me because instead of just knowing what you’re thinking about from when we catch up on our phone calls, I kind of follow them along. Maybe not in real time, but today I was on the train and I went to Twitter and got to scroll through some of those posts. I found one that was really exciting that I thought would be great for the podcast.

Squirrel: Well that’s why I’m tweeting them: so we can pick them up and write blog posts and podcast episodes and stuff. So which one are you interested in?

Jeffrey: Link in the show notes, I’m going to read it out and then you can tell us about the scenario and your thoughts about this. ‘If your tech team has a set of detailed business metrics, like 10% conversion or NPS up by 12, and is missing them comprehensively: don’t add more metrics. The complexity of the targets may be confusing them. Try ‘make this specific user happy’ instead.’ This sounds like a great story. What inspired this?

Struggling for Control

Listen to this section at 01:51

Squirrel: Well as usual, there’s real clients behind it, but there are multiple clients in this case. Sometimes clients say, ‘oh, that was about me’ and it’s actually about someone else. So if any of my clients are listening, it probably isn’t you specifically, these are common errors and challenges. In this case, I have multiple clients who have been trying to get greater control. It’s the same as the tilted slider, the same mechanism where people say, ‘I’m not getting what I want. Let’s tighten down. Let’s turn the wrench further,’ and they wind up stripping the nut. Right, just like when you’re you’re trying to tighten a nut on a bolt. When they try to get that tighter control and focus on the metrics—’let’s try to increase NPS by 12.2, how about 12.27?’—what they see over and over again is that the team has greater trouble hitting the metrics. As you get the metrics more specific, as you try to have greater control, you get worse outcomes. It was that observation that made me think, ‘gosh, this is one I should tweet about and talk to Jeffrey about, because I’ve seen it over and over again. Tighter control leads to worse results.’

Jeffrey: You mention the idea of adding more metrics, something like ‘we’re missing the metrics, therefore we need to add more.’ Is that the dynamic?

Squirrel: Certainly! Sometimes rather than just adjust the target it becomes ‘we don’t need just NRPs. We also need a customer survey and a number of clicks per day.’ They add more thinking that will give better guidance to the development team. Universally, it does not.

Jeffrey: Sound like there are two behaviors here. One is just adding more with the idea that more metrics mean more control. The other is a false precision: ‘oh, instead of 12 it should be 12.2 or 12.27’ when really the score is something like 3. It’s nowhere close.

Squirrel: Exactly.

Jeffrey: Part of what excited me about this is I’m actually really a fan of using metrics. I’m actually on board with this. The move towards OKRs I think has been in many ways positive. One of things that it’s done is it has brought a lot of business metrics into the frame for development teams. The teams are trying to impact business metrics directly. But I do see that having the kind of impact that you describe, where people are adding on more and more business metrics-

Squirrel: Because if one business metric is good, wouldn’t 12 be better? The team could help us with all our business metrics and we could just go to the beach?

Jeffrey: Exactly. This idea that more are better is a common problem with metric use in general, not just business metrics. I often find when teams are trying to be data-driven, they look to add more and more metrics. Sometimes it’s development metrics: things in production, ‘let’s add more logging.’ But when I start working with teams I’ll ask, ‘how are you using these metrics today?’ And then there’s often the sort of awkward pause-

Squirrel: That becomes a long silence.

Jeffrey: Exactly. They are looking at the metrics that they think would be important, but they don’t make them part of their decision-making process. I agree with what you said here, don’t add more metrics. Instead, start using the metrics you have. More broadly, use the data you have! As a company, you already have data. Have you thought to use it? Have you looked at things like the calls that are coming in? What are the topics that people are calling about? Who are the people who are calling?

Appropriate Data

Listen to this section at 06:22

Squirrel: When some people start to look at the sorts of things Jeffrey is saying, ‘count the number of calls’ and ‘find out who’s calling and when,’ they wind up at, ‘we need to have at least 12 calls a day where people cite this feature.’ If you already have this disease, then it’s easy to take what Jeffrey saying and warp it into what we don’t want. That is why at the end of the tweet I said, ‘why don’t you try asking some users?’ In fact, I have one client I know is probably going to listen, and I’m not trying to beat her up for it, she did a great job in her environment, but she made three spreadsheets with pivot tables and everything, these massive things, in order to figure out the top five things that the team should work on. Just overcomplication on top of overcomplication. When you have the tendency to take what’s actually a fairly simple problem, ‘what would make our users really happy,’ and you’re creating these complex metrics, mechanisms, algorithms, fancy things, just stop! Hang on! Maybe, for example, you could go and ask the people in customer service, ‘what do people call the most about?’ And then when they tell you the 20 things you say ‘and which one is the most common? Say that a little more slowly?’ And you write that down.

Jeffrey: That’s exactly where I was going with this. You have the data, and sometimes it can be as simple as asking people. I had a particular case in mind where we realized we didn’t really know where to go, so we decided to gather data. Rather than trying to do an exhaustive data mining effort of looking through all past things, instead, the product manager just started to have the team group what the options were about, and used that to say, ‘OK, well, now we have some data.’ Using that imperfect data is better than spending a long time trying to create perfect data.

Squirrel: Exactly! That’s what I wish this client of mine would do: to spend time understanding the users well enough that they don’t have to create three spreadsheets with twenty pivot tables in order to determine what is needed.

Jeffrey: I think there is a place to get more sophisticated over time. This is something you can grow into once you’ve shown your ability to actually bring that data into the decision-making process.

Squirrel: Oh, absolutely. There are companies that do a tremendous job of very, very deeply understanding their users in a very sophisticated way through often very complicated algorithms. They use data science to analyse the input that they get. People like Google and Facebook and folks like that, they’re also kind of good at delivering. They’re also kind of good at getting that work done and getting the feedback actually handled. So the problem is folks on this quest for control that you see so often, when they feel the team is out of control, say, ‘if I could just get better numbers, more numbers, then I could be like Google or Facebook. I could be like these organisations that do these complex analyses. My being more complex would be better.’ It’s not.

Jeffrey: It sounds like you’re very much focussed on step one: make sure that you’re able to deliver anything. Step two: make sure you’re delivering something you have reason to believe would make at least one user happy. Only from there, you can get more sophisticated.

Squirrel: Sure. Get super sophisticated, once you understand your users. Once you have that baseline from which to work, by all means. The problem is the vast majority of organisations that I see have trouble making one user happy, much less a chi squared P relevant least squared regression of all users. Let’s try making one happy.

Jeffrey: So today you’re the one focussed on the users. I completely agree with everything you’re saying. But to think less about users for a moment, make sure you’re using the data you have before you start looking to collect more. Both of our messages are actually the same: Don’t have more metrics. Get the basics right.

Squirrel: Stop the madness.

Jeffrey: That’s true from both of our perspectives, even though we’re coming from different places in considering this.

Squirrel: Thanks, Jeffrey.

Jeffrey: Thanks, Squirrel.