This is a transcript of episode 240 of the Troubleshooting Agile podcast with Jeffrey Fredrick and Douglas Squirrel.
We look at why “prop trading” companies like Jane Street have phenomenally profitable technology operations, and how we can learn from them and improve, even in organisations where IT is a “cost centre”.
Show links:
Listen to the episode on SoundCloud or Apple Podcasts.
Introduction
Listen to this section at 00:11
Squirrel: Welcome back to Troubleshooting Agile. Hi there, Jeffrey.
Jeffrey: Hi, Squirrel.
Squirrel: So you sent me a fascinating article. Why don’t you tell our listeners about it?
Jeffrey: This came across on Hacker News, and it was on understanding Jane Street. For people who have never heard of Jane Street, they’re what’s called a market-maker, which means they are happy to buy and sell any stock at any time. They’ll just set a price for it. So they “provide liquidity to the market.”
Squirrel: Which means they trade all the time, every single second of the day.
Jeffrey: That’s right. After spending ten years in London at a company that created signals that people would use for automated trading, I came to be a little bit more interested in this area, but that’s not why I wanted to talk about it.
Squirrel: So what could our listeners who are in something completely different learn from this?
Jeffrey: What really stuck with me was this line in it:
So one reason Jane Street spends so much energy on finding hiring the right people, running job ads on programming theory blogs or posting puzzles, holding guest lectures and after-hours chess tournaments, etc. is that every company reaches the point where “do something strategic!” is an itch that’s worth scratching, but in prop trading there are very few legitimate ways to scratch it, so companies are forced to excel.
Jeffrey: Really what stood out to me is that the kind of environments that demand excellence in what you’re doing technically, in your coding and software development practices, are such great environments. I was thinking about those elements that make it such a rich learning environment and what that means for us, for people who aren’t necessarily in those environments. So that’s why I thought it was worth talking about.
Squirrel: I’ve certainly been in environments with fast and slow ability to respond. How do you look at this? How did you compare this to other situations that our listeners might be in? Because we probably don’t have that many listeners who are high-frequency traders.
For the Rest of Us
Listen to this section at 02:34
Jeffrey: That’s true. The first thing that stood out to me is that for these companies, the money they make is a function of the software they write very directly. They’re working on a hypothesis of how they’re going to make money, and if it doesn’t work, if they’re not able to implement it correctly, if their software isn’t high quality, then they lose money. There’s two things about that. First, their software team is looking to make profit. That’s one of the big divides across my career as I’ve worked with lots of companies, this difference in thinking between profit center versus cost center.
Squirrel: Anathema to me. My bristles go up when I hear “cost center” applied to technology.
Jeffrey: That’s right. One of the tests we used to share with people is “if your organization calls you IT, it means they probably look at you as a cost center.”
Squirrel: Wait a minute, some of our listeners might not know these terms. A profit center is a part of your business where you expect to get more out of it than you put in. In a typical business, sales would be a profit center because you shouldn’t be paying your salespeople more than they sell. If you do that, you’re doing something incorrect. Whereas say, the provision of the physical environment for your employees, renting the office space, with very few exceptions is a cost center, because you wouldn’t expect to get more out of it. You probably don’t expect to get anything out of it. You’re just going to throw money at this thing, and if you could somehow make your employees all work at home—gee, some people seem to have done that recently—then you would eliminate that cost and you’d be really happy. Whereas if you eliminated your sales, you would eliminate your business, there’d be no profit.
Jeffrey: That’s right. That’s really explains why you had a lot of companies who were very excited about outsourcing their IT operations because they looked at it as a cost center. “This is a cost of doing business, if we can make that cost lower, that’s better. If the quality suffers, maybe it doesn’t matter as long as it’s adequate and you get the job done, that’s all that matters.” So a cost center is really defined by “can we do an adequate job for the lowest possible price?” There’s little value in doing a better job because in a sense, so what? Whereas profit centers are the opposite. You do a better job, you’re making more money. That’s a very different mindset. If I think about prop trading, I think the very most opposite case I’ve ever dealt with was dealing with the IT departments at insurance companies. This is going back 15, 20 years mind you.
Squirrel: I know this intimately. So painful.
Jeffrey: I visit these traditional insurance companies, and the problem was they were so fundamentally profitable in their core business that what they did in software, in effect really didn’t matter all that much. So there was very little incentive for them to improve. The mindset was just “it just needs to be adequate.” They weren’t really competing on their technical capability. Now of course, there’s been a lot of changes in a lot of industries, this is part of “software is eating the world.” More and more companies have to be actually competing on their ability to deliver software with quality. It’s one of the big drivers behind things like the DevOps enterprise summit that IT Revolution runs: talking to these enterprise companies where traditionally IT was a cost center and they need to really change to become a profit center. Very different mode of operating.
Squirrel: It’s my purpose in life. I make technology teams insanely profitable and if I’m going to do that, they better be a profit center first. That’s usually the mindset shift that I have to help somebody go through, to shift their thinking from “I’m throwing money at this” to “I can get more out of it than I put in.” It’s a huge shift. If you don’t have that, and I bet a lot of our listeners recognize this and say “that sounds like my company,” we’ll have more to say about what you can do about it, but it’s very useful to know where you sit on this axis.
Jeffrey: That’s right. The second thing that quote made me think about is about slow feedback versus fast feedback. One of the things about high-frequency traders and firms of this type is that when they are going to go and test a theory in the market, they’re going to get feedback very quickly. They’re going to be able to put something out and start making trades, and it’s either going to start performing according to their models or it’s not. That fast feedback is another really key element for learning. In terms of rate of learning, it’s probably more important than the profit center versus cost center, and it’s also more accessible to people. I think you have some really great examples of the opposite, of places that have slow feedback and kind of limited ability to change that.
Behavior at the Extremes
Listen to this section at 08:01
Squirrel: I can give you an example not from my direct experience but from a very clever person of the the kind of extremes to which somebody like Jane Street goes. Dan North talked a lot “Spike and Stabilize,” which he observed when he went I think from Google to a prop trading firm. He watched the engineers as they worked and he had been an agile advocate and was suggesting people do things like unit tests. And he noticed that these engineers were playing with millions and millions of dollars and didn’t have any tests at all, and would actually put something live with what’s called a stop loss, something that would stop the automated trading bot from losing more than a certain amount. And they’d set it at some eye-watering number for you and me, like $1,000,000. Then they’d run it and they’d come back half an hour later and they’d say, “Oh yeah, actually there was a bug. It lost the whole million dollars. Well, I’m glad I found that out in half an hour. Let’s fix the bug and try it again.” His jaw was on the floor, he’s falling out of his chair saying “My God, why didn’t you just write a unit test?” But as he started to understand it better, he saw that the million dollars for these folks was a small price to pay for getting really immediate feedback. So the method that they were using he christened Spike and Stabilize because they would not write any tests at all, not even check if the thing worked, put it live, see if it made money, and if it made money then they would stabilize it and say, “Oh, I understand there was actually a bug in here, but it was making us more money. Let’s put a test for that bug.” So it’s kind of extreme “test in production” philosophy, which you can get if you get extremely fast feedback. If you’re trading like a Jane Streets, thousands of times a second, then it doesn’t take you very long to get a real read on whether what you’re doing works. However, as you were saying, I have an opposite experience. I have a sideline in biotech firms, so there are various companies that have me help them with their technology, which also includes things like wet labs and so on. I’m no expert in these things, but it turns out that a lot of the tools that I use make just as much sense for these folks as for engineers, and there are plenty of software engineers in these companies, too. I remember one of them, many of them have this characteristic, but one was really prominent because they said, “we really do have to wait for the next iteration of what we’re doing.” And I said, “Well, gosh, can’t you speed that up in any way? Maybe you could run it in parallel.” And they said, “Squirrel, you can’t have two mice have a baby in three weeks when it takes one mice six weeks to have a baby.” They were waiting for the next generation of mice. So there are situations in which there are laws of nature and biology that force you to have quite a slow response time. You cannot have feedback instantly because you have to wait for something in the environment. That’s the opposite of where Jane Street is, of course.
Jeffrey: Yeah. I think these are two axes we could chart companies on. Slow to fast feedback on one axis and cost center to profit center on the other axis, and the prediction I would have is that companies in the upper right that are fast feedback environments that are technology profit centers would tend to have more learning and they would tend to have above average performance of their technical teams. The consequence of that is a twofold: we can learn from those people, it’s worth understanding what people are doing in those environments because they’re going to have a lot of opportunity to learn. So what they’re doing is going to be of interest to people who are in other environments and also want to be effective at learning. Second is to look where you are and ask “how can we be better? By changing our environment? Can we get a shorter feedback cycle and can we improve our visibility of our profitability? Can we measure the impact of our profitability for development work?” You mentioned earlier that this is often getting people to change their mindset and think in terms of ROI. Can you talk about your first step for approaching that with people?
Have the Conversation
Listen to this section at 12:56
Squirrel: First is to try and measure where they are on these axes, try to get an understanding of where they are coming from and then work out what kinds of steps, what kinds of conversations could lead them to change that mindset. If you have somebody who gives you a limited budget and you look at that as immovable, you’re not going to be able to change from a cost centre to a profit center because of course in a profit center you spend more and you make more. So the philosophy isn’t limited budget, it’s “how much can we spend in an efficient way.” If that’s not the mindset, then yes, you have a classic situation where you’re a cost center. But if that really is immovable, why am I there? So I don’t view that as immovable. My experience is that mindset can change, and it’s going to require a conversation couched in the language of money. Of course it helps that I’m coming from the outside, but you can do this internally as well. If you go to someone who has previously been immovable and say, “I’d like you just to join me on the profit center side for a moment, I think there might be more profit that we can make here. Here’s what that would look like and here’s what the return would be.” I often find that that makes a huge difference in the conversation with that person. For instance, that person might say, “The board of directors won’t let me do anything about that.” Then you say, “How could we go to the board of directors?”
Jeffrey: Yeah.
Squirrel: Exactly the same thing happens when I talk to the development teams. There it’s more a technical barrier where they say, “Oh we couldn’t possibly get fast feedback.” Now the folks with the mouse model? Fine. I recognize we’re not going to speed up the mice. But all other cases I ask, “what would it mean for us? If we really had to, how would we get faster feedback?” Even in my biotech cases I’ll get answers like “Well, if we did that we’d have to go around compliance. Compliance would have to change their process.” And I’d say, “Great, where do they sit?” Or they say, “We’d have to invest in more servers, we’d have to have an additional staging area, we’d have to automate a lot of our testing.” And I say, “Great, who’s in charge of the server budget? Who’s in charge of QA?” So what you often get with these mindset conversations is if you can get to them to the point where they say, “If I were to do that, then this would have to change.” You want to listen to the second half of that very carefully and then say, “Great, how would we do that?” That usually blows their minds because they have thought that’s not possible, and I find that often it’s much more possible than they think it is.
Jeffrey: That’s fantastic. The other thing that comes to mind for me about this is people might say, “well, we’re doing this work and it’s supposed to have an impact on future sales, and it’s going to have an impact on renewal rates” or something like that. “So we won’t know the impact on profitability until the renewal cycle comes up.” And my view is well, actually you have a model, right? You have a model saying “we think when people use this new feature, it will give them value and that value will be recognized by us on the renewal rate.” So you can use that usage as a proxy metric and say “if we see usage of this functionality go up, then we predict it will have this future impact.” So you can start using it as a proxy metric to get faster feedback on your impact on profitability. I remember a good example we had at TIM, we had a new UI and one of our product managers would be looking at the usage and when people weren’t using it, he would pick up a phone and call them and say, “I saw you use the new feature and then you went back to using the old version. Why is that?”
Squirrel: Some of them thought we had installed cameras. It was really quite frightening for some of them. He had to reassure them. This was when software as a service was a bit newer so people didn’t always understand it wasn’t on their own computer and they accused us of spying on them. But once they understood we were just looking at server logs, they found the feedback very helpful. Of course we found it utterly insightful and very valuable for increasing profit quickly and we didn’t have to wait for them to renew next year.
Jeffrey: That’s right. So these are things that people can do to change the mindset. Last week we talked about having a business mindset and talking about things in terms of ROI, so there’s another example of how you can talk in business terms.
Squirrel: That sounds fantastic. Thanks, Jeffrey.
Jeffrey: Thanks Squirrel.