Operational Excellence is the Pursuit of ‘Knowledge’

Note: this is Part 3 of the Becoming Data Driven in Business series. You should read the prior two parts (one, two) before reading this essay.

In How to Become Data Driven, I introduced process behaviour charts, and wrote that they are best thought of as a way to introduce a particular type of thinking about data to your organisation. At the end of the piece, I followed that up with a Donald Wheeler excerpt, where he says that introducing these charts to your company are how you “get to the kind of thinking that enables continuous improvement”.

At this point, it’s probably clear that adopting XmR charts help you understand variation. But the obvious next question you might ask is: what is the type of thinking that it leads to, that Wheeler and others talk so much about? What is this promised land and how does it work?

Wheeler has a deceptively simple thing in Understanding Variation where he says (note the bold emphasis, which are mine):

In any study, success will often depend upon knowing what questions to ask. The process behaviour chart is unsurpassed for focusing data so that the user can formulate the interesting and important questions.

So what’s the difference between the traditional, limited comparisons and the process behaviour chart? The difference between superstition and knowledge.

The traditional approach inevitably results in wasted effort and missed opportunities. The use of traditional approaches guarantees an excess of both kinds of mistakes people make when interpreting data. More often than necessary, managers interpret some bit of noise (my note: routine variation) as a signal, and thereby waste time and resources in looking for an explanation that does not exist. On the other hand, many signals (my note: exceptional variation), and the opportunity for improvement which these signals represent, are missed by the traditional approach.

The process behaviour chart filters out the probable noise in order to detect the potential signals in any data set. By filtering out the noise, the process behaviour chart will minimise the number of times that you will interpret a bit of noise as if it were a signal. By causing the potential signals to stand out, the process behaviour chart will also minimise the number of times that you will miss a signal. Thus, process behaviour charts are the beginning of knowledge because they help you ask the right questions.

Everyone needs that kind of help.

It turns out that Wheeler is using a very specific definition of the word ‘knowledge’ when he makes these statements. One of the more annoying things about Understanding Variation is that it is not clear that he’s doing this if you’re just reading the book. You’ll have to step back a little and read the broader body of work around Statistical Process Control (SPC), and in particular W. Edwards Deming’s thoughts on business management. The word ‘knowledge’ is related to two ideas in the broader SPC literature; both bits point us towards the type of thinking that Wheeler is talking about.

The first idea is something we’ve covered before: SPC practitioners believe that ‘management is prediction’. The underlying argument is that in order to run your business properly, you need to be able to predict — within limits — the effects your actions have on business outcomes. You need to know things like “oh, so if I run such-and-such marketing campaign, my leads should go up by around X% over the next quarter”, and “if we launch this feature, we should expect to see higher engagement from our core users, which may be measured using the metrics A, B and C” ... plus you need to be correct about those things.

I am painfully aware that this is easy to say, but difficult to do. It is rare in the vast majority of businesses I’ve been involved in. Hell, if I’m being honest, my growth decisions around this very blog are more gut-driven than rigorously tested, which is a little embarrassing to say out loud. And perhaps you can relate: how many of your business decisions are driven by feel, made on loosely correct beliefs about your business — which is a more polite way of saying ‘you manage by superstition’? In most cases, though, the things you do somehow work out, and perhaps the business is able to grow — but you’re never really sure if your actions or your predictions are as good or as causal as they can be.

SPC practitioners take this observation to its logical conclusion. Deming argues that if management is prediction, then what you need to seek as a business operator is not truth — for truth does not really exist in business — but instead what you should seek is knowledge, where knowledge is defined as “beliefs or theories that enable you to make better predictions”.

Putting these two ideas together gives us a better idea of what Wheeler means. When Wheeler says that usage of process behaviour charts is the ‘beginning of knowledge’, what he means is that such charts should lead you — and your organisation! — to ask the kind of questions that will slowly build into a causal model of the business in your heads. Over time, consistent and repeated investigation of exceptional variation (plus removal of said variation) should lead you to a better understanding of the various subsystems in your business. Which in turn means that you should be able to understand how all those systems fit together — so you may predict how modifying one aspect of that system would affect all the other aspects that are connected to it.

All of this, of course, is ‘knowledge’. And so we may restate Wheeler’s assertion a little more concretely: the type of thinking that usage of XmR charts should lead to is an org-wide commitment to the pursuit and use of ‘knowledge’, which is defined as beliefs or models that enable you to predict business outcomes more accurately. This sounds deceptively simple — almost too simple to be useful. But as with most things in SPC, simple axioms tend to lead to interesting practices. Let’s dive in.

The Process Control Worldview

The good news is that we’ve already looked at one such example of this worldview in action: this is, essentially, the goal of the Amazon Weekly Business Review (WBR). (If you haven’t already, I recommend clicking the link and rereading my description of the WBR in practice, but with this worldview in mind).

When ex-Amazon executive Colin Bryar calls the WBR a ‘process control tool’, what he’s doing is referring back to the style of thinking that Wheeler is talking about. In all of my interactions with him, I’ve noticed that Colin takes a particular view on companies: he thinks of a business as a system with inputs and outputs, which may in turn be decomposed into smaller processes — and that if you want to improve a particular process, you throw something like the WBR at it in order to force watchful improvement. And the reason the WBR forces watchful improvement is that it compels its participants to identify and focus on the controllable input metrics that drives desired business outcomes. In other words, the WBR forces the pursuit of ‘knowledge’.

One way of looking at all the tools, mechanisms, and org design experiments that Amazon have done over its nearly three decade history is that it has mostly been in service of this process control,  operations-centric, ‘pursuit-of-knowledge’ mindset. The bad news is that this style of thinking is unnatural and unintuitive. The good news is that this style of thinking may be learnt. One thing I discovered whilst going down this rabbit hole is that Amazon founder Jeff Bezos didn’t come naturally to this worldview — in an a16z podcast with both authors of Working Backwards, Colin Bryar hints at Bezos’s growth when he was shadowing the founder:

How Jeff changed during that time and just to become a better CEO — one thing he said is, he learned how to become a better operator, becoming more operationally efficient. Fortunately, that is a teachable and learnable skill. And Jeff had some great people at Amazon, like Jeff Wilke, you know, who’s a great operator. He’s the CEO of the Consumer Business. And now in his own right, he’s this great operator and insists on the highest standards — it’s one of Amazon’s leadership principles.

In other words, Bezos learnt this style of operating from Wilke — who came from an operations background, and whom Bezos called his ‘tutor’. “Becoming a better operator” is just code for “getting better at predicting business outcomes”. Which is probably why talking about this worldview is just as important as discussing any of the specific mechanisms that lead up to it.

Stated concisely, what is the process control worldview? I remember being surprised by Bryar’s assertion that the WBR is “just a process control tool”, but it turns out that he’s publicly talked about things in this manner long before I met him. In a First Round Review interview from February 2021, Bryar says the following:

"Just think of a business as a process. It can be a complicated process, but essentially, it spits up outputs like revenue and profit, numbers of customers, and growth rates (emphasis mine). To be a good operator, you can't just focus on those output metrics — you need to identify the controllable input metrics. A lot of people say that Amazon doesn't really care about profit or growth. I think that the data say otherwise, but what is true is that the main focus is on those input metrics," says Bryar.

"If you do the things you have control over right, it's going to yield the desired result in your output metrics. The best operators I've seen very clearly understand that if they push these buttons or turn these levers in the right way, they're going to get the results they want. They understand that process through and through. (emphasis mine)"

The reason I’m calling this out explicitly is because most people I’ve talked to who are familiar with SPC think SPC is about controlling and then reducing variation in manufacturing. They then assert that SPC is only applicable in manufacturing, and not applicable elsewhere. This is true — variance reduction is an important aspect of manufacturing excellence, and SPC is primarily used in production lines. But what I got from talking with Colin is that the process control mindset is really the more interesting, valuable thing that may be taken and applied to businesses outside of manufacturing. This mindset was really the thing that Amazon adopted and then perfected during his tenure there.

I’ll give you another example of the mindset in action, from early-ish Amazon.

Anyone who has followed Amazon might be familiar with the company’s focus on ‘absolute dollar free cash flow’. The most famous example of this is perhaps Bezos’s 2004 shareholder letter, where he makes the case for free cash flow (FCF) per share, and then walks readers through an example of a hypothetical business with terrible FCF characteristics.