One of W. Edwards Deming’s favourite questions was “how would you know?”
He would often ask this question when presented with some process improvement, by some hapless manager, at some industrial company. “How would you know?” he would ask — that is, how would you know that your idea has worked?
If the manager had the misfortune to reply: “using my experience”, Deming would retort: “and with that, he has revealed himself!” — that is, the manager was proposing superstition.
Deming taught business operators methods to do better than just superstition.
How Would You Know?
It’s worth asking if you — or your boss — might do better than the hapless manager above. Think about the last time you:
- Launched a new marketing campaign?
- Changed your software development practices?
- Updated your sales enablement material?
- Or changed your sales process?
Clearly each change you’ve made — be it a new marketing campaign or introducing full CI/CD or switching to a different lead scoring mechanism — you expect to result in improved outcomes for your business.
But how would you know if the change has worked?
Did you look at some data after the fact? Did you run a two group ‘control vs intervention’ study and waited for statistical significance? The metrics you pick will wiggle; do you have a way of separating signal of change from noise? Hell, did you even pause to evaluate the impact of your changes, so that you may update your intuitions about what might or might not work in the future?
If you’re anything like me, I think the odds are good that none of the above things happened; you probably made some change and said “I guess things are better”. After which everyone moves on to the next thing.
A reader emailed me recently and said that whenever engineering leadership at his company decides on a change in their software engineering practices, the most senior engineers would vanish into a room for weeks or months and then emerge — like Moses descending from Mount Sinai with the Ten Commandments — with a new set of best practices they would roll out throughout the org.
And how would they evaluate if they had successfully changed things? Well, they would use their experience, of course!
I know I make this sound like some bad thing, but this is the way I do things too. I’m willing to wager this is the way most of us do things in our companies.
The Three Questions
If you read any number of books about Continuous Improvement today, you would see echoes of Deming’s favourite question come up again and again.
Here’s statistician Donald Wheeler, in Chapter 1 of Making Sense of Data:
Continual Improvement requires a framework. We all need some way to align our efforts and focus on a specific objective. To this end I have found the following questions to be equally helpful.
What do you want to accomplish? Until you have a clearly stated objective, you risk everyone running off in different directions, working on their own pet projects, and not cooperating for the common good. Whether it is a specific project with a limited scope, or the general day-to-day operations of your organisation, a clearly stated purpose or objective is important to help focus the thoughts and efforts of everyone involved. (…)
By what method will you accomplish your objective? While it may be necessary to have a goal, merely having a goal is, but itself, not sufficient. Remember the old saying, “if wishes were horses then beggars would ride.” Until you have a plan for achieving your objective, it will be nothing more than wishful thinking. All of our targets, all of our goals, and all of our plans are merely wishes and hopes until we have some specific method for making them come true. (…)
How you know when you have accomplished your objective? If you are going to have a goal, and if you hope to move towards that goal, then you will also need some way to measure how far you have come and how far you have yet to go in reaching that goal. Moreover, you will need a way to evaluate that measure to determine if a change has occurred.
I call this the ‘Three Questions of Continuous Improvement’. Wheeler shortens it to:
- What do you want to accomplish?
- By what method?
- How would you know?
And he prints it out at the end of every chapter in Making Sense of Data.
The Secret
There’s a secret lying at the heart of continuous improvement, and perhaps you’ve spotted it already.
Let’s back up a bit. Over the past few weeks, we’ve been deep in the weeds on becoming data driven in business: we’ve talked about the ideas of Statistical Process Control, and the ideas of statistician W. Edwards Deming (who created both SPC and Continuous Improvement). I told you how these ideas lay down the simplest path to becoming data driven in business. And last week I announced Xmrit, a free tool to create XmR charts — and a free email course that taught the basics so you can replicate that path yourself.
But at the heart of all of this lies a small set of really simple ideas. If you think about it — putting aside, for the moment, Deming’s observations about human psychology — what we have here is really three load-bearing concepts:
- The Plan-Do-Study-Act cycle, which was Deming’s frame for trial and error.
- The three questions of Continuous Improvement
- The Process Behaviour Chart.
Together, they represent a methodology for single subject studies. You now have a method to improve anything: yourself, your body, your department, and ultimately your business.
Let’s go through them quickly:
The PDSA Cycle
Trial and error sounds obvious, and simple, but as we’ve seen in a previous essay — something odd happens when you do trial and error in a business context. Most people do not circle back and ask “what, exactly, have we learnt?”
There are many reasons for this. You’re too busy. A business emergency occurs. You forget. But in so doing, you learn nothing.
Deming found it necessary to teach what we might consider obvious: you make a plan, you do the plan, you study the results of your execution, and then you act with the results in mind. Rinse and repeat.
As you do so, you must answer:
The Three Questions of Continuous Improvement
The way I like to remember the three questions is:
- What do you want to accomplish?
- How will you accomplish it?
- How will you know you have accomplished it?
If you squint at these questions, you’d realise the trouble is really with the third question. Most people know what they want to accomplish. Sales leaders want higher sales, product people want higher retention, marketing people want attribution, and so on. Similarly, these people will often have proposals for how to accomplish what they want.
Comparatively few people have an answer for the third question. Which explains why Deming gave us:
The Process Behaviour Chart
The problem with attempting to answer “how do we know we’ve accomplished it?” is variation — that is, any number you attempt to track will wiggle. This wiggling gets in the way of your saying, confidently: “yes, this metric has changed because of what we did”. Without an ability to cut through the noise of variation, you won’t be able to use data. Which means you will have to rely on qualitative judgment: you must wait longer for confirmation; we must read the ‘vibes’ — and only the vibes — which means you may all too easily trick yourself.
Process behaviour charts tell you when a metric has changed. They allow you to separate signal from noise. This is why Deming taught them — he wanted everyone to have a default answer to the third question of Continuous Improvement.
If you’ve been following Commoncog, you’ll know that we launched a free tool to plot process behaviour charts for any business metric you might care for; this is Xmrit. The simplest PBC is the XmR chart. We launched Xmrit because we want everyone to use XmR charts. And we did this because …
Why does this matter?
Why is this important?
In business, as in life, what you really want to know is what works for you.
Sure, best practices are great. But they are problematic. A best practice developed in the context of another company, at another time, for another company’s problems, might not translate well to your company, and may not solve your problems. You must always test when you apply best practices — which means asking “how would you know?” In other words, you need to do a single subject study where you are the subject.
And Deming knew this of course: he always warned against copying best practices blindly. He argued that even a little ‘Theory of Knowledge’ should make you suspicious of another company’s knowledge.
It seems a little bizarre that knowledge can be gained from studying one subject at a time. When I was a Computer Science student, I watched a famous talk called Bits of Evidence by Greg Wilson. In it, Wilson argued that software engineering was like medicine pre-science: we have very little knowledge about what works. Wilson made this argument because we did not do randomised controlled trials. We did not base our best practices on actual scientific evidence.
You can already work out the points and the counter points. RCTs are extraordinarily time consuming; they are also very expensive to run. What works across multiple companies might not necessarily work for you. But Wilson argued that until we were willing to pay this cost, we had no hope of knowledge in software engineering. I remember being struck by the strength of his argument and the quality of his talk.
But I no longer agree with him.
Consider: whilst RCTs are the gold standard in medicine, doctors know that what has been proven to work in a large study does not mean that it would — with 100% certainty — work for you. There are always individuals for whom particular drugs might not work; some people develop drug tolerances that others do not; doctors are always saying “try this, and come back in two weeks” for a reason. Pharmacology can say “we have knowledge that such-and-such drug will work for disease X for most people” but the exact efficacy and the precise side effects are only observed when you take the drug.
As a patient, you really only care about what works for you.
Continuous Improvement — kaizen, if you want to call by its Japanese name — sounds trite. I can’t believe that I’m saying that it is a secret. But in truth, I have never learnt the combination of the three ideas I just gave you; most people don’t talk about how all of them are necessary to do continual improvement.
By combining these ideas, Deming ensured that we could improve whatever we cared about. And of course initially he taught this only to the people who were willing to listen: the factory workers, industrial engineers and corporate brass of bombed-out industrial Japan, in 1950, when he started giving his first lectures there. The Japanese industrialists listened, and then they went back to tinker with their production lines. Two decades of single subject studies — ahem, trial and error — later the folks at Toyota emerged with the Toyota Production System, and folks at Honda emerged with their Honda production system, and Mazda theirs, and on and on, until the world of manufacturing was forever changed.
Well, you have the ideas now. We may have forgotten them outside of manufacturing, but I’ve just given them to you. You can improve whatever you set your mind to.
What will you change?
Originally published , last updated .
This article is part of the Operations topic cluster, which belongs to the Business Expertise Triad. Read more from this topic here→
