Making Sense of Deming

The piece on the Weekly Business Review is going to take a bit more time to get done, since Colin Bryar has final edit rights. I want to spend this week talking about W. Edwards Deming, the proximal source of the ideas in the Becoming Data Driven series. This piece will serve as a comprehensive introduction to Deming, so that you don’t have to read multiple books on the man (like I did); you can skip the theory and go straight to real world instantiations of the ideas. About which, more in a bit.

This piece is long. I want to do some setup so you understand the justification for it.

A couple of weeks ago, a reader pinged me and asked if I was going to talk about the rest of Deming’s ideas. This reader had been digging into Deming independently, after learning that the man was one of the key sources for the data driven stuff I’d been investigating over the past 1.5 years.

“Absolutely not.” I said. “The Statistical Process Control stuff is hard enough to articulate, can you imagine what would happen if I told people what Deming thought of OKRs?”

He laughed.

But I’ve changed my mind in the weeks since: I do think it’s worth it to talk — even if just a little — about the man’s work. Not because I think you’re going to adopt them, much less accept them — many of Deming’s ideas are controversial. But the blast radius of these ideas is so large and so impactful that you cannot unsee them once you know what to look for; if you are interested in operational excellence in business, at some point you do have to tackle Deming head-on.

The problem with Deming is that his ideas are so … subtle, so odd, that they’re a little difficult to get into.

Fortunately, what little Deming I’ve been feeding you by way of the Data Driven Series is actually foundational to all of his other ideas. This makes this essay a bit easier to write.

Here’s what I’m going to do: I’ll give you a high-level overview of Deming’s work. Fair warning: you are going to find many of these ideas difficult to stomach. We’ll go through the major tenets of his philosophy, and then I’ll talk to you about the real world impact of these tenets — at least, the impact that I’ve discovered so far — before finally telling you what I think about the man and his ideas, including what bits I want to test in practice.

Deming 101

W. Edwards Deming is typically described as an engineer, consultant or a statistician. For many years — right up to his death — his business card read ‘Consultant in Statistical Studies’. While this is not wrong, I think he is better described as a business philosopher. Deming took the ideas about variation that we’ve explored in previous parts of the Data Driven Series and extrapolated from them an entire coherent philosophy of operational excellence.

There’s a particularly revealing exchange in Christopher Leonard’s Kochland that tells us Koch Industries CEO Charles Koch understood the essence of Deming’s ideas:

Just what was continuous improvement, exactly? [prosecuting attorney Ken] Ballen asked.

“How much time do you have?” Koch replied.

“How much time do you have?” Ballen replied.

“Continuous improvement philosophy is a philosophy developed by a man named W. Edwards Deming, who is a statistician,” Koch said, “So he set up a philosophy based on statistics, how companies can improve their competitive position by improving the quality for the customer and your own productivity.”

Koch went on for a long time, talking about this guy named Deming, whom Koch seemed to truly admire. Deming’s ideas seemed to revolve around coming up with mathematical models for how to improve a business, and then continually driving workers to make those improvements and hold true to the plan.

“This is a long-term program,” Koch said. “As [Deming] puts it: ‘You never get out of this hospital.’ You are going to be working at this forever.”

I joked on Twitter that understanding Deming turns Kochland from a biography of a powerful American conglomerate into a playbook for gathering so much industrial power you eventually control the Republican Party. (No seriously: read all the chapters on lobbying in Kochland, and read Jane Mayer’s extremely biased Dark Money, which describes how Koch hijacked the Republican senate electoral process — and now read all of this through the lens of process control.)

The key to understanding Deming is to understand variation. Hopefully, at this point in the Data Driven Series, and especially in the wake of Becoming Data Driven, From First Principles, you have a functional understanding of variation. We’ve already looked at some of the first and second order effects of the idea: in particular how it will change the way you think about operational excellence. But that’s just the tip of the iceberg. We’re now going to look at what happens when you take variation as a given, and then chase down all the implications — especially the implications of variation on human psychology. The way we’ll do this is that I’ll first describe the set of Deming’s ideas that should be easy for you to accept. And then we’ll build towards the ideas that are more … challenging.

Deming did not give us prescriptive mechanisms. Instead, he gave us a handful of principles. Normally I would warn against reading folk like these — it is difficult to evaluate the believability of business theorists. Except that in this case we have multiple real world instantiations of his philosophy; these real world instantiations include the total industrial reinvention of modern Japan. If you internalise Deming’s principles, you may bootstrap from them an effective business operating system that you may call your own.

Let’s get started.

Deming: The Easy Ideas

Deming organised his philosophy into a system of four pillars late in this life, the so-called ‘System of Profound Knowledge’. These are:

1. Understanding of Variation — which we’ve spent the bulk of the Data Driven Series exploring together.
2. Theory of Knowledge — this is epistemology as applied to business. As I’ve explained elsewhere, Deming has long argued that ‘management is prediction’ and ‘there is no such thing as truth in business, only knowledge’. Epistemology is important when applying his ideas — if you do not have a good sense of knowledge, you will attempt to blindly copy instantiations of Deming’s principles from other companies. You would not be able to do trial and error rigorously yourself.
3. Appreciation of a System — this is systems thinking, which we’ve touched on in passing during the sequence where I introduced the ‘Process Control Worldview’. Systems thinking is in some ways an obvious consequence of Deming’s overall philosophy: you can only improve your business with process control if you know how all the processes you’re observing interlock. Similarly, you can only improve your business if you understand how employee psychology affects system design.
4. Psychology — I have mostly steered clear of this pillar, and for good reason. ‘Psychology’ describes how humans find meaning in and are motivated to work in complex organisational systems. They compose with the three other pillars in counter-intuitive ways. I believe that this pillar — of all the pillars in his philosophy — is the source of Deming’s most challenging ideas.

Deming’s System of Profound Knowledge leads inexorably to a bunch of actionable recommendations he calls the 14 Points. I will present these points in my own order, because I want to introduce the least controversial ideas first.

Point One: Create constancy of purpose for improving products and services

This is not a particularly novel point. It is also easy to say, and difficult to do. We know — sitting here and talking about business in this abstract manner — that it is not possible to run a business well if you’re just thinking for the short term. Count the number of business biographies where CEOs focus on short term cost-cutting, refuse to invest for the future, and then have the business suffer the consequences a number of years later.

Every business needs to have a constancy of purpose for improvement. This consistency of purpose shows up in two ways: first for current products, and second, for future products. Mary Walton writes, in The Deming Management Method, that establishing constancy of purpose breaks down into four elements: 1) innovation, 2) research and education, 3) continuous improvement of product and service, and 4) maintenance of equipment, furniture and fixtures, along with new aids to production in the office and in the plant. You’ll notice that the first two has to do with future products and services; the latter two with current products and the business operations needed to supply them.

I suspect that Deming had to call this out because he saw countless American businesses in the 80s underinvesting in both current and future products in the face of overwhelming Japanese competition. They cut costs indiscriminately, or thrashed about.

So far, so obvious.

Point Two: Adopt the new philosophy

The basic idea with Point Two is simply: “Adopt Deming’s philosophy. Don’t half-ass it. That’s the only way you can compete with Japan.”

I don’t think the point about Japan is as relevant today, but we should talk a little about it in order to understand the context of Deming’s work.

During World War 2, Deming helped teach Statistical Process Control (SPC) to two thousand men and women under the auspices of the Ordnance Department. Many of these folk in turn became instructors, and gave more classes on SPC to more companies and industries throughout the country. All told, some 31,000 engineers, supervisors, managers and workers went through some form of this course, taking Deming’s methods back to their factories in order to create better weapons and better supplies for the war. In Deming’s Journey to Profound Knowledge, author John Willis argued that this quality advantage was one of the factors that led to the Allied forces winning — and that Japanese production engineers had noticed.

After WW2, SPC was largely forgotten in the United States. American industry found that it had no special need for continuous improvement — they were, after all, the country with the strongest industrial base in the world. And so American manufacturing became sloppy, even wasteful. They stopped the use of control charts, and reverted back to quality by inspection — discarding defects as a normal part of doing business. In the meantime, post-war Japan began adopting Deming’s methods en masse. Japanese engineers called these methods ‘military quality control standards’ (eventually just ‘QC’) instead of ‘Statistical Process Control’ — because they saw these methods as US military production practices.

Fast forward two decades, and it was American industry that found itself on a back foot. Deming’s teachings — including an earlier version of these 14 points — had taken root in Japan, and through them various Japanese companies had bootstrapped their own unique business operating systems. The Toyota Production System came to be the best known of these. As a result, cheaper, higher quality Japanese imports had flooded America, causing serious trouble for American firms. Over the 80s and through to the 90s, many American manufacturers died, or moved offshore.

In 1980, CBS aired a documentary on the Japanese economic miracle, finally profiling an American statistician — Deming! — as the source of the Japanese threat. American companies began reaching out to Deming for help. The 14 Points comes from this period; point two is really an exhortation to drop sloppy practices and to adopt the ‘new’ — or rather the forgotten — philosophy of continuous improvement that Deming had taught US factories all those years ago.

This crisis afforded Deming a unique opportunity to push for change in his home country; this is one reason the title of his book is ‘Out of the Crisis’.

Point Three: Cease dependence on inspection to achieve quality

This is a subtle argument, one that might be difficult to understand if we haven’t already spent so much time on understanding variation. The core insight here is that in order to achieve low defects, invert the problem to focus on higher quality instead. That is: focus on what you want to see more of, not what you want to see less of.

Inspection, is, after all, a dumb idea: it is much cheaper to identify and then fix problems early in the production process than to inspect for quality at the end. If you do the latter, you will be very wasteful: you will discard finished products. Better to just increase overall quality at each stage, such that the defect rate goes down permanently.

How do you do this? Well, one way to think about this is to think of quality as a problem of variation: bad quality is the result of excessive variation during production. Therefore, the best way to improve quality is to bring variation under control within the entire system.

This is, incidentally, why these ideas have mostly been trapped within manufacturing. Deming’s tools are tools to characterise variation, which are typically misunderstood as tools to reduce variation. But as we’ve seen over the course of this series, tools to characterise variation can also be used to detect changes in variation — which means that you can run process improvement in nearly every aspect of business, not just within quality engineering.

So, if you can detect when variation changes, you can …