Becoming Data Driven in Business: A Series

What does it mean to be data driven in business? Most people think being data driven means looking at charts on a daily basis. Or they attempt to use data in their orgs, and then fall into one of the many traps that come with the territory. How do you actually get good at using data for your operations? And how do you build the mindset and the organisational capabilities necessary to do it?

In 2022, I did some work for Colin Bryar and Bill Carr, former Amazon execs and authors of Working Backwards.

The project I did for them was about explicating Amazon’s Weekly Business Review (WBR) for a broader audience. I took on the project for two reasons: I wanted to implement the WBR in businesses I was involved in, and I wanted to learn what it meant to be data driven, from folk who were present when Amazon laid the foundation for its operational rigour today.

What I didn’t expect to learn: many of the ideas behind Amazon’s metrics rigour seem to come from the field of Statistical Process Control (SPC) — a subdomain of operations research created by Walter Shewhart in the 30s and popularised in the post war era by W. Edwards Deming and his contemporaries. For whatever reason, these ideas seemed to have entered and then spread throughout Amazon — perhaps as a result of Amazon’s strong operational hires during the late 90s.

I thought SPC’s source ideas were valuable enough to dig into, given the influence they've had on Amazon’s approach to business.

This series is about those ideas, and the various instantiations of the ideas in Amazon, in manufacturing, and in other businesses. As it turns out, many of the principles necessary to become data driven in business were worked out over the course of the 30s through the 80s by Shewhart, Deming, and others. They simply aren’t as well-known.

Pre-Series

These series of articles are somewhat related to Statistical Process Control, if only because the ideas come from the founders of the field.

1. You Aren't Learning if You Don't Close the Loops — An article about the Plan-Do-Study-Act cycle, sometimes called the ‘Shewhart cycle’ or ‘Deming cycle’.
2. Process Improvement is Trickier Than You Think (members only) — The organisational and political dynamics you’ll face when you attempt to do process improvement.
3. The Disaffected PhD Skunksworks: A Story About Process Improvement — What it's actually like to do process improvement when the organisational dynamics work against you.

The Series

1. Goodhart’s Law Isn’t As Useful As You Might Think — Goodhart’s Law says “when a measure becomes a target, it ceases to be a good measure.” It turns out that the field of Statistical Process Control have long developed methods for protecting against this effect. We look at those ideas, and then examine a particular instantiation of the approach in the context of the Amazon Weekly Business Review.
2. How to Become Data Driven (members only) — What does SPC have to say about becoming data driven? The central idea it uses is deceptively simple: in order to become data driven, SPC practitioners argue, you need to understand variation. We talk about why that is.
3. Operational Excellence is the Pursuit of Knowledge (members only) — Knowledge is defined as ‘theories or models that allow you to predict better’. It turns out that the mindset necessary for operational rigour stems from the pursuit of ‘knowledge’. The beginning of ‘knowledge’, in their view, is a deep understanding of variation. This piece covers the practical implications of this mindset.
4. There is No Truth in Business, Only ‘Knowledge’ — We look at some of the more philosophical implications of pursuing ‘knowledge’ over truth, particularly in business. As it turns out, assuming that there are no stable truths in business is a pretty good hack.
5. Process Behaviour Charts: More Than You Need To Know — Process behaviour charts are the SPC practitioner's secret weapon to understanding variation. It also happens to be the simplest way to differentiate signal from noise when looking at a measure. We talk about everything you’ll need to know to be able to use them well.
6. What to Think When Looking at a Chart (members only) — In a business context, what should you think when presented with a time series? This seems to be a really dumb question that nobody talks about — but that we now have interesting things to say, given all that we've examined over the previous five parts in this series.
7. Executing on Becoming Data Driven: The Politics (members only) — Every analytics project consists of two parts. The first, technical bit, and then the ‘get the organisation on board’ bit. Why the second part is much harder than the first.
8. Executing on Becoming Data Driven: The Technicals (members only) — What the technical bits of execution look like, as applied to Commoncog (the business).
9. What’s an Operational Definition, Anyway? — The prior entries in the series focuses on data analysis, with little focus on data collection. After several months of practice, we take a look at two principles for data collection, from the field of Statistical Process Control.
10. Beck’s Measurement Model, or Why It’s So Damn Hard to Measure Software Development (members only) — Kent Beck, in collaboration with Gergely Orosz, released a ‘measurement model’ in response to a controversial McKinsey report on measuring software developer productivity. Why this model turns out to be a good lens for evaluating if a business activity is easy to measure.
11. Two Types of Data Analysis — Broadly speaking, there are two approaches to data analysis in business. The two approaches have different worldviews, and express different tradeoffs. Here's why that matters.
12. Becoming Data Driven, From First Principles — A synthesis of the major ideas of the series, in preparation for a two-parter on the Amazon-style Weekly Business Review.

Also related:

• The Deming Paradox: Operationally Rigorous Companies Aren't Very Nice Places to Work — This is only nominally part of the series, and was written as a reflection on Deming’s body of work between Parts 4 and 5 above. Includes a case study of Deming’s management philosophy as applied in Koch Industries.
• Making Sense of Deming (members only) — A comprehensive summary of Deming’s ideas, written so that you can skip reading the man himself (if you want to!) and go straight to hunting for instantiations of his business philosophy.
• The Secret at the Heart of Continuous Improvement — Continuous Improvement sounds simple, even obvious. And yet there's a profound secret at its heart that doesn't seem to get talked about.

Originally published 15 April 2023, last updated 16 April 2024.