This is a transcript of episode 244 of the Troubleshooting Agile podcast with Jeffrey Fredrick and Douglas Squirrel.

Squirrel describes how he tells clients to handle overly specific, too-coherent directions: consider what an adversary would do to comply with the direction and still get bad results. He and Jeffrey discuss illustrative examples and applications from military history, economics, and pizza delivery.

Show links:

Listen to the episode on SoundCloud or Apple Podcasts.

Introduction

Listen to this section at 00:11

Squirrel: Welcome back to Troubleshooting Agile. Hi there, Jeffrey.

Jeffrey: Hi, Squirrel. You were telling me about this great technique you’ve been using with clients who have a certain problem, which is that they’ve been given objectives that sound more like marching orders than actual objectives. And you have a great name for it too: adversarial analysis! What is this and why do we need it?

Squirrel: The situation I see over and over again is that somebody will give a direction to somebody else, very commonly what I encounter is the CEO giving a direction to the head of product or the CTO, but it could be the same dynamic anywhere in the business. Often, the direction is kind of one level down from where it should be, it’s not a strategy, it’s a tactic. And the thing that has been useful is something that I practiced and learned about in the analysis of algorithms. Listeners might be familiar with algorithms as tools used to determine whether or not a website or some other service is secure. In the typical example there’s Alice and Bob, and they want to communicate, say, a credit card number.

Jeffrey: Oh, but there’s another, Eve.

Squirrel: Eve is the adversary. Eve is coming along trying to do something bad. It’s very helpful to imagine that somebody is out to get you, that they’re really going to try to do the wrong thing as effectively as possible. They’re practicing malicious compliance. So Eve might be visiting your website and because she’s visiting your website, she has to click on buttons and enter names and addresses and things, but instead she enters some SQL code that deletes your entire database, because she’s evil.

Jeffrey: Little Bobby Drop Tables shows up.

Squirrel: That’s the example. You’re using this imagined evil being because if you think about somebody whose only mission is to make your life miserable, it unearths lots of things that you might have been missing if you make the happy assumption that we often like to make, that everybody’s got our best interests at heart. This adversarial thinking catches all the people who are doing things by accident or through maliciousness. Let’s take an example, for one of my clients the instruction was “never fail this specific kind of transaction.” So when the user is trying to do X, never fail it, make sure that the user always gets a positive result, that they always have the booking, the purchase, that whatever it is that they’re doing succeeds.

Jeffrey: That sounds great!

What Could Go Wrong?

Listen to this section at 03:35

Squirrel: Yeah, absolutely! “What if everybody who came to buy something from us succeeded?” And I said, “Well, let’s think about this adversarially.” Because one of the things that you might do is accept orders and requests that are obviously fraudulent. They have bad credit card numbers, they come from people who have scammed you before. “Whatever your anti-fraud division is doing, ignore it.” The advantage of that would be you would make sure that you reduce the number of failures to a small number. But that’s not actually good. You’re losing money.

Jeffrey: When you told me that it reminded me of Goodhart’s Law, which is the idea that any measure ceases to be a good measure when it becomes a target. So when you’re looking at your success rate on this transaction and it becomes a target, you never fail. You get 100%. But it no longer is a good measure of the business. You get these distortions, you don’t get the benefits you were hoping for. The happy path case of “Yeah, more successful transactions are better.” But now the adversarial approach shows the problems with just going along that way naively. When you make this a target, you open yourself up to kind of various problems.

Squirrel: Absolutely. Another classic example is the famous Pizza Arbitrage. A delivery company in the US was offering delivery for a restaurant without that restaurant’s permission and without scraping the restaurant’s website very well, so they were mistakenly selling a pizza that costs $24 for $10, or something like that. So the restaurant’s owner went off and bought pizzas from himself and made money no risk at all. And the point is that there is somebody who would be really happy about that because they’d say, “great, there’s more pizzas being bought! This is super.” That person wasn’t taking into account the possible adversarial action. In this case there was a real adversary actually costing them a lot of money. It’s probably not even fraud, right?

Jeffrey: No, it’s not! You’re just ordering. They set themselves up to do this.

Squirrel: Yeah, and somebody at the delivery company is saying, “look, we have more orders coming in!” But they haven’t considered “are we actually making any money on those orders?” In fact, they were losing money because they were buying it for one price and selling it for a lower one, which is a good way to go out of business very quickly.

Jeffrey: That’s right. So this is for those cases where the problem is you have a simplistic target you’re given, you’re told to go do this thing that isn’t well considered and doesn’t really have constraints on it. This vulnerability is what your adversarial analysis is pointing out: the implicit assumptions being made that should have been there but weren’t specified. Is it something like that?

Why Would You Want This?

Listen to this section at 06:25

Squirrel: Exactly. And that’s what I did with my with my client. We had a look at this instruction “Never Fail” and said, “actually, if we think about it, there are cases where we’d really like to fail. It’s not just fraud.” There are cases where a customer had just completed an order, and my client isn’t selling a product you really want two of very often. It’s not “we might want two pizzas,” you probably don’t want to order two cars at once, or something like it. So if it’s a double order you want to say, “Hey user, are you really sure? Maybe you don’t want to do this?” And the CEO here said, “Yeah, actually there are lots of cases like that and we want users to be happy because we failed for them in a way that was productive and helpful.” And I said, “Oh, we want happy users. That’s what we really want.” So what we decided was that the target should be something like “everybody leaves our page happy,” and they might leave happy that they did not purchase that second product. We want them saying, “great, this is a super company. I want to come back and purchase more stuff from it” because either “they sold me a great thing” or “they didn’t sell me it for a really good reason.” And of course, we also want the financial controller to say, “yes, I’m happy that we actually made some money on the transaction.” So that’s the sort of thing that I’ve discovered when I apply this adversarial analysis with my clients, it really helps you to unearth very quickly what are the things that could go wrong with this and what is the underlying interest? Because the interest here wasn’t get more orders, the interest was make more happy customers who come back and order more. Once we understood that, it was much easier to align to that, and then of course it’ll be easier to build actual features in the software that help with that: really clear error messages and helpful detection of problems earlier in the transaction flow and all of those sorts of things suddenly flow out when you give the target, “make the customer happy” rather than “complete more orders.”

Jeffrey: This is great. There’s three things that come to mind from the story and what we discussed. The first is it struck me that this is right out of Art of Action, which we’ve talked about on this podcast before, and the idea of briefing and back-briefing. There were two concepts in there that both seem to apply: people usually aren’t very good at giving orders or directions or setting the intent, depending which language you’re using. In that book, they described a very nice model of briefing and back-briefing, and the point it made was that the executives, the leaders, the officers, generals, whoever you want to put it, are not giving good briefings because they’re not talking at the right level. They’re usually getting down one level too low. And it sounds like that’s what you’re seeing here. So a good briefing gets you elevated to the right level, and your adversarial analysis in part pushes back and gets you back up to the right level, which I really like. And the second is a good briefing has constraints in it. So again, your adversarial analysis is helping to bring out those constraints that are important. The last thing is, this kind of adversarial analysis sounds like a really useful conversation. Multiple parties exploring “what’s really the interest that we’re after.” And so to be able to have that conversation as opposed to just taking what you’re given, “go increase this by 10%, lower that by 20%.” You’re saying “Really? Let’s talk this through,” and this framework depends on the psychological safety that the person you’re challenging understands that you’re doing it as a way of improving the outcomes. So you need a certain amount of trust to have that conversation, but if you do, you end up in a much better place. So those are three things that came to me. I love the dynamics going on here.

Sating the “Obvious”

Listen to this section at 11:02

Squirrel: Yeah. It would be very important to make clear that you do not, in fact, intend to go out and order a bunch of pizzas in order to make free cash. But you are thinking about how someone could do that and whether that then means that actually we’ve got the target a bit wrong here. What do we actually mean? But you would definitely want to give that disclaimer before launching into it with your boss.

Jeffrey: The disclaimer of “look, I’m on your side. I’m going to ask some questions here to clarify the constraints. This isn’t me trying to shoot down your ideas. I’m trying to better understand it.” That’s actually a really critical thing. So often when I’m my coaching, the first lesson is “your boss needs to know that you’re on their side.” Because so often what happens if the boss is saying things that you agree with, people say nothing and they only raise objections. So the boss is left thinking this person doesn’t get it. They’re not aligned. They don’t understand because all the things you agreed with, you didn’t say. So it’s worth reminding yourself when you’re going to go have this conversation that your boss can’t read your mind and know that you’re taking this adversarial approach in a positive sense, in a constructive sense. So it’s worth making that explicit.

Squirrel: There we go. Thanks, Jeffrey.

Jeffrey: Thanks, Squirrel.