If you’ve read a lot of articles about decision making, you might think that good decision making is simply a function of applying the right decision-making frameworks to the world, and then reaping the benefits.
For instance, if you’re lucky enough to be picking between three career paths, you might decide to do an expected utility calculation to figure out which of the three to pursue. Or if you find yourself an observer to some internecine office politics, you might decide to fall back to Bayesian analysis to figure out what’s really going on.
I’ve talked a lot about these ideas on this blog, and I’ve usually had good things to say about all of them. But in this piece I want to take the other side for a bit, and talk about the limitations of these techniques. These limitations, as you’ll soon see, apply to just about every technique drawn from the judgment and decision making literature. They should be obvious to you if you’ve ever attempted to put these ideas to practice, or if you’ve dug deep enough to discover the origins of the ideas.
Somewhat surprisingly, the bulk of the criticism may be captured in a single sentence: “action produces information.” Keep this aphorism in mind; it’ll come in handy in a bit.
Some Examples
In order to understand the limitations of classical decision making frameworks, it helps to go through a bunch of real world examples. Here are two.
1. Picking Careers
A friend called me recently to talk about a difficult career decision he was about to make. He was finishing college, and had many options available to him.
“So I want to do a startup.” he said, “But I’m balancing that against the possibility of joining a FAANG company, or perhaps a prop trading firm.”
“Why the prop trading firm?”
“Oh, the money’s good.”
“Have you ever worked at one?”
“Nope, but we have mutual friends who have.” (We did).
And so it went on in this manner, for a bit. After a couple more questions, we worked out that my friend had:
- Interned at two major tech companies.
- Interned with two large startups (>50 and >150 people respectively)
- Not had any experience at an early-stage startup (<10 people, or pre-product market fit) and not had any experience at a prop trading firm.
My friend was really smart; it was obvious that we could have done something like an expected utility calculation to work out his options. An expected utility calculation is basically a fancy version of the ‘pros and cons’ analysis that most of us are familiar with. Instead of listing all the pros against the cons, however, we list down a bunch of ‘utilities’ — that is, things that we value e.g.: money, smart teammates, challenging problems, etc — and then we evaluate each option (startup vs FAANG vs prop trading firm) with a score for each item in this list. Finally, we write down the probability that we would realise the utility for each option available to us. (If you want a more detailed example, you may find one here).
But of course we didn’t do that. Expected utility calculations assume perfect information about each choice. That wasn’t the case here.
“It sounds like you don’t have enough information,” I said, near the end of our call. “You already know what it’s like to work at a FAANG company, and you're currently interning at a latter-stage startup. But you don’t know what it’s like to do an early-stage startup, and you can't easily figure it out, because you don't have a good idea for a startup right now. Plus, you don't know what it’s like to work at a prop firm. So why not spend a year or two doing each of those things? Just to get that information?”
“Hmm,” my friend said, thinking.
“I say this because I think the decision will be easier a few years down the road, once you have more information. And a year or two isn’t much, given the long arc of an average career.”
2. Bayesian Analysis in the Office
Bayesian updating is the method of ‘holding a belief, and then updating that belief as new information emerges’. It is a fantastic technique to use when making judgments of the world. I’ve written about this approach before, in my series on putting mental models to practice; I’ve also described how Superforecasters use Bayesian analysis to come up with well-calibrated forecasts of the future.
But good analysis and good forecasting isn’t the same thing as effective action.
Let’s imagine, for instance, that you have a colleague who is constantly coming in late. The boss doesn’t appear to care. You don’t really understand what’s going on; you think that it’s a little bizarre that nobody talks about this. You conclude that your colleague is lazy but that there is something political going on, which makes it acceptable for him to come in late but not for you to do so.
The Bayesian model will tell you to examine your priors, to calculate the probability that your chosen explanation is the right one. It then tells you to watch for new pieces of information that might shift your % confidence in that belief. This is all well and good, but note that Bayesian updating was built for the integration of new information; it doesn’t say anything about the generation of new information.
Test subjects in decision experiments may sit back and expect to have information revealed to them over the course of said experiment. Geopolitical forecasters cannot expect to influence the events they are forecasting. But if you are a real-world decision maker, you are in a different situation from these people. You may choose to analyse, or you may choose to act. In other words, there is a real trade-off that you're making here: time spent doing Bayesian analysis might actually be better spent acting, in order to unearth new information!
Let’s return to our example of the perpetually late colleague. Can you do something to generate new information? The answer is yes, you can. You may, for instance, arrange lunch with said person, and then ask head-fake questions to determine what’s going on in their life. Perhaps this generates more questions than answers, in which case you may drop back down to Bayesian analysis. But perhaps it results in an answer that’s so unambiguously true that you may save yourself the trouble: “Oh, my wife and I just had a baby, and I’m opting to come into the office because Jon wants me around for this deployment; once this is done I’m going to take my paternity leave and vanish for a month.”
The Cost of Decision Analysis
The main problem with the decision making frameworks you read about in books and blog posts is that they all come from the field of judgment and decision making, which in turn has its roots in rational choice theory, which in turn comes from economics. In these academic disciplines, the assumption for rational choice is that you have a bunch of options laid out in front of you, and you must pick from one of them. This picking usually happens in an environment of perfect information.
If you pick well (meaning that you’ve maximised your ‘utility’), you are said to have ‘acted rationally’. If you’ve picked badly, you are said to have acted in an ‘irrational’ manner.
It is tempting to read this research and then conclude that the frameworks and models are immediately applicable to the real world. But in reality, the models are somewhat limited because the real world doesn’t share all the assumptions that the models must make.
For starters, you often have more choices available to you than are laid out in front of you. Some of these choices may be occluded by uncertainty, or are reachable only by creative problem solving. Other times, they are hidden due to lack of information. For instance, perhaps my friend has better options available to him; perhaps he was limiting himself to only three career choices based on what he currently knows.
Second, decisions in the real world are often time sensitive — the sooner you act, the more value you realise from having acted (but often the precise amount of that value is also occluded by uncertainty). Most decision-making frameworks don't take time selection into account, since there is no need to model time sensitivity in decision experiments. But in the real world, the utility of each choice may sometimes depend on the decisiveness with which you act on your analysis.
Third, and most importantly, action often generates new information, which then allows you to make better decisions. In other words, there is often a cost associated with doing decision analysis, but the frameworks do not take that cost into account.
In fact, this third observation — that action generates new information, which then allows you to make better decisions — is a pretty powerful one. As it turns out, none of the decision making frameworks from the judgment and decision making literature will tell you when you should stop analysing, and when you should act instead. The reason they do not do so is because these frameworks were originally designed for use in economic modelling. In decision experiments, you do not typically expect a participant to act aggressively in order to gain more information from their environment — you expect them to do the analysis!
I should note that this is not a particularly novel set of observations to make. The three critiques I've laid out above are old criticisms of the judgment and decision making field; psychologist Jonathan Baron takes great pains to include them in his seminal textbook of the subject.
And the observations are also quite obvious if you know where to look. Watch any group of entrepreneurs for a long enough period of time, for instance, and you would notice that the best entrepreneurs aren’t necessarily the best calibrated Bayesian updaters or expected utility calculators. Instead, the best entrepreneurs tend to have a mix of bias-to-action and fast adaptation in response to new information.
(A Chinese businessman we had done business with once put it to me like this: “Why you think so much? Just act first! Then you watch and see what happens. Maybe the customer don't like it. Or maybe your competitor do something to you because you do this. But then you know more than if you just sit here and think think think!” )
Why is this the case? Like well-adapted predators, good entrepreneurs are able to adapt their behaviours to match the contours of reality, and the contours of reality in business seem to be:
- A sizeable portion of decisions in business are reversible decisions.
- The information that comes from action is often more valuable than the insight that comes from analysis. This is especially true if there is a high level of uncertainty in your industry.
To put this simply: analysis has its limits. Expected utility calculations may tell you how to pick the best option from a suite of limited options, in an environment of perfect information. Bayesian updating tells you how to update your beliefs when presented with new information. But neither technique has anything to say about how to act to generate the best options, or the best information. It shouldn’t surprise us, then, that sometimes the people who act quickly and remain adaptive are more likely to win than those who perform the best Bayesian analysis in the world.
Heuristics for Acting
The title of this blog post comes from Brian Armstrong, the founder and CEO of Coinbase. In his interview with investor Patrick O’Shaughnessy, Armstrong says:
It doesn't even matter what you do as long as you do something, because that's my other favourite quote, is “action produces information.” So at a certain point, you got to stop pontificating about this stuff and just try something, anything. You're going to be embarrassed by the V1 until you go out there and you create. That's part of the product development process, is just dramatically scaling back kind of the ambition and the feature set and everything to rapidly iterate and prototype these things, but go do anything. The first thing you try is almost guaranteed not to work. So don't give up, just go try the next thing, and the next thing, and the next thing. That's the only way that new products and companies ever get created in the world. You got to put a lot of shots on goal to get one to eventually work.
If the frameworks from decision science are limited in their usefulness, then perhaps we should look to actual practitioners to see how they manage the tension between analysis and action.
As it turns out, there are many examples from businesspeople and product managers and practitioners. You merely have to know how to look. Here are three.
Scott Berkun, on Product Bets
In The Year Without Pants, veteran product manager Scott Berkun talks about a major bet he had to make with his team at Automattic:
Part of the reason that perfect decision formulas can't exist is that you never know if you're buying too much or too little insurance. Did you see the right doctor for your elbow? Did you ask the right questions? You can make the correct decision in the wrong way. One risk with our plan B was that two weeks wasn't enough. We might need to spend months to improve even one weak spot. Fear of this uncertainty motivates people to spin their wheels for days considering all the possible outcomes, calculating them in a spreadsheet using utility cost analysis or some other fancy method that even the guy who invented it doesn't use.
But all that analysis just keeps you on the sidelines. Often you're better off flipping a coin and moving in any clear direction. Once you start moving, you get new data regardless of where you're trying to go. And the new data makes the next decision and the next better than staying on the sidelines desperately trying to predict the future without that time machine.
In the book, you'll learn that Berkun flipped the coin ... and got lucky. But even if he didn't, that would have been ok — as Berkun points out, his team would have course corrected anyway, when it became clear it was the wrong path for their project. The point was to make the coin flip, and to move on.
Gary Klein and the Zone of Indifference
This begs a natural question: how do you know when it's better to 'flip a coin and move in any clear direction’, as Berkun puts it? Psychologist Gary Klein, who works primarily with the military, has a heuristic he calls ‘accepting the zone of indifference.’
The zone of indifference is when you cannot tell which decision is the best option. Klein writes:
If you had to compare two options, one of which is outstanding and the other of which is terrible, you wouldn’t need to do any analysis. It would be an easy choice. As the two options get closer and closer together in their attractiveness, the decision gets harder. (...) In the example of purchasing a used car, we can see that the three options are all very close—they each have comparable strengths and weaknesses. There just isn’t much that differentiates them. The options were so close together that simply flipping a coin would have been sufficient. (...) I call this the zone of indifference problem.
Recognising that certain decisions are in the zone of indifference is a good way to save decision-making time: imagine, for instance, that you are leading a meeting where you have to decide five things in 30 minutes. A brutally effective way to get through as many of those decisions as possible is to use the zone of indifference to rule out certain decisions, which would then allow you the time to focus on only the most important, most tractable problems.
Klein continues:
We usually think that the goal of decision making is always to pick the best choice. There are few decisions more important than on the battlefield or on the fireground, where lives are at stake. Yet military leaders and fireground commanders recognize that it is better to make a good decision fast and prepare to execute it well rather than agonizing over a “perfect” choice that comes too late. We can rarely know what is the best choice, and the quest for a best choice can drive us to obsess over inconsequential details. How often do we get ourselves trapped into splitting hairs, to find the very best option out of a set of perfectly good choices? Better to make your goal one of selecting a good option that you can live with. If one option emerges as the clear winner, fine. If two or more options wind up in the zone of indifference, that’s fine too—just pick one of them and move on. If you can accept the impossibility of making the “right” choice, you can free yourself from unnecessary turmoil and wasted time.
Jeff Bezos’s Reversible And Irreversible Decisions
Amazon CEO Jeff Bezos has a remarkably similar, if simpler formula: if the decision is reversible, act quickly, and delegate decision rights. If the decision isn't, then analyse all you want.
He describes this heuristic in his 2015 shareholder letter:
Some decisions are consequential and irreversible or nearly irreversible — one-way doors — and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don’t like what you see on the other side, you can’t get back to where you were before. We can call these Type 1 decisions.
But most decisions aren’t like that — they are changeable, reversible — they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.
As organisations get larger, there seems to be a tendency to use the heavy-weight Type 1 decision-making process on most decisions, including many Type 2 decisions. The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention*. We’ll have to figure out how to fight that tendency.
*The opposite situation is less interesting and there is undoubtedly some survivorship bias. Any companies that habitually use the light-weight Type 2 decision-making process to make Type 1 decisions go extinct before they get large.
I can’t make it more concise than that.
Get the Action Sheet
Download the actionable summary for Action Produces Information here →
Originally published , last updated .
