You Aren't Learning If You Don't Close the Loops

There’s a deceptively simple idea called the PDSA loop (‘Plan-Do-Study-Act’, alternatively named ‘PDCA’ for Plan-Do-Check-Act, alternatively named the ‘Deming cycle’), which comes from the work of American engineer and quality control pioneer W. Edwards Deming. I currently think that the loop captures the heart of good business execution, but that for some reason we find it terribly difficult to do.

The basic idea in PDSA is that you:

• Make a plan
• Do the plan
• Study the results of the plan
• Act on what you’ve learnt and integrate your new hypotheses into the next iteration of the loop.

At this point I can see your eyes roll to the back of your heads, which is understandable: the PDSA loop is boring and brain-dead and common-sensical, because this is the most obvious way to iterate!

When I started taking the loop seriously my mind went back to an experience I had in university, when I helped create a new technical talk series at my school. The way we iterated towards a successful format was nearly textbook PDSA in nature: we discovered that a) more students would show up if there was free food, so we started providing free pizza at every talk, b) that attendance would drop roughly in the middle of the semester, around the time of the mid-term exams — so we learnt to schedule the least important talks then and c) we learnt that companies really wanted access to good software engineers, and so were willing to sponsor aforementioned food or send us speakers to give technical talks; we merely had to reach out and ask.

The weekly nature of our talks gave us a natural cadence to tweak the format. More importantly, though, the structure of the university term lent itself well to the PDSA cycle. At the end of each semester, I’d invite the entire team to one of the shared common rooms in my dorm; I’d cooked them a large dinner and then ask “What did we do well? What did we do badly? And what are we going to change next semester?” We didn’t do this because we’d read Deming or anything; it just seemed like the right thing to do.

As a result, the talk series — and by extension, the club — became a major success during my time as a student; it continues to be a valuable part of Singapore’s tech ecosystem today. All of us who were involved with setting it up were rather pleased with what we’d created.

And so it is one of the great ironies of my operating career that I look back on all the companies that I’ve been involved in and mostly see a graveyard of incomplete execution loops.

This was surprising. I didn’t realise there was such a disparity in execution quality between my university-level club activities and my business activities until after digging into some of Deming’s work. And now I mostly feel regret; how can something so common-sensical and so obvious be so hard to do in practice?

Distractions, Distractions, and Good Ideas

I’ve had a couple of months to mull over this question, and at this point I think I’ve got a simple answer. Back when I was in university, the semester structure forced us naturally into a PDSA loop. We planned at the start of the term; executed during, and then ‘studied’ the results of our experiments right after our exams, before acting on those conclusions the next term. But in business execution, you’ll have to be very disciplined to see your activities as a loop, much less to see a loop through to the end.

Discipline, of course, doesn’t come naturally to novice operators. Or at least, it didn’t come naturally to me.

But let’s be a bit more concrete about the phrase ‘discipline’. It’s not usually the Plan or Do part of the PDSA cycle that’s difficult for most businesses to do. It’s usually the Study and Act bit that’s difficult.

You should know this, I think: it is easy to come up with plans and then start executing on them. The problem is that:

1. Either you get distracted, mid-execution, by something else that’s shinier. Or you get distracted mid-execution by something that blows up (and in business there’s always something that’s blowing up).
2. Or you get distracted by something else that pops up at the end of the current execution loop,
3. Or you forget to do the ‘study’ bit at the end of a loop, which informs your next cycle.

It seems a little ridiculous that you can forget to do something, but consider: certain loops in business takes months to execute. A new SEO strategy, for instance, takes at least six months before results show up. A lot of things can happen in six months. Your most important customer may cancel. You start losing deals to a previously harmless competitor. Your head of engineering tells you she’s quitting. In the mess of business execution, you ignore the fact that you’ve just spent six months on an experiment, and then you … forget. You don’t ‘harvest’ the learnings from that project, and those months of execution go to waste.

A more common experience, though, is that you get distracted. (It’s no accident that the last two essays on this site have been on the topic of focus). I’ve lost count of the number of times I’ve been distracted by a shiny new thing, or (less charitably) when I’ve been working with people who are distracted by shiny new things. SEO is taking too long to work, so you go for the endorphin rush of an attempted viral blog post. Your outbound sales campaign isn’t working, so you speculate that the problem is lack of product differentiation and fill Jira with feature proposals. Your current set of features are too grindy and boring to build, so you pursue the short-term high of a side project launch. Seen individually, viral blog posts and product differentiation and side projects are all good ideas; it’s only after a handful of these that you look back and realise you’ve created a graveyard of half-finished execution loops.

Sometimes the distraction is justified. You’re in the middle of a landing page rewrite when your CEO tells you that sales has figured out a new narrative that works better than the old one, and they need marketing’s support to come up with new sales enablement material. Clearly this task is more important: it directly impacts the quarter’s revenue. So you tell your team to put down the landing page rewrite … and then you never return.

Other times the distraction is understandable but not so justified. Here at Commoncog we spent four months working on our Burnout Guide, summarising the bulk of burnout research in one easy-to-read piece. The original intention for this guide was to see if we could use a set of commonly known SEO techniques to grow the site’s rankings. But the guide went viral on launch. I was so distracted by the distribution, the positive feedback and the attention that the guide was getting that I nearly forgot about the original hypotheses that we had. It was only when I consulted the original 6-pager I wrote at the outset that I realised we needed to test certain things; the virality was nice but not the main purpose of this particular execution loop.

(This is, by the way, an explicit recommendation to write out your hypotheses before you execute. It doesn’t matter if you jot it down in a 6-pager format or a Google Doc or whatever; the point is that you’re likely to forget your original goals by the end of a loop if you don’t put things down on a page.)

So, yes, the PDSA loop is ‘easy’ and ‘obvious’ to talk about, and you’d be forgiven for thinking it’s yet another example of over-complicated management jargon. But in practice it’s terribly difficult to do. It’s no accident Deming had to turn it into a bloody framework for it to get traction amongst the companies he was consulting. And it’s no accident that I had to take it seriously before realising what it implied about my track record.

The Opposite of Sunk Cost

One of the more curious things about this entire concept is that it interacts with the sunk cost fallacy in weird ways.

I sort of hinted at it earlier, when I said “you don’t ‘harvest’ the learnings from that project, and those months of execution go to waste.” The underlying assumption is that the work you’ve spent executing on the each counts for something: those months of work are wasted if you don’t take some time, at the end, to study the results.

But let’s take a step back. The sunk cost fallacy says that you shouldn’t take into account the costs that you’ve already incurred doing some activity; you should only take into account the costs and potential rewards of decisions you have ahead of you. The Deming cycle implies the opposite. It is wasteful to execute a couple of half-assed loops: you’ve burnt all that time, and all of those resources, and for what? For nothing. You don’t learn anything, you don’t let it inform future execution loops, you just do the thing half-assed and then go after the next ‘good idea’. What do you accomplish? Possibly some things. But what do you learn? Very little.

I don’t mean to say that the sunk cost fallacy is a lie. There’s a related failure mode called the ‘plan continuation bias’, which is basically the “forced continuation of an existing plan or course of action in the face of changing conditions.” Heaven knows I’ve seen this, and I’d like to bet that you have as well, in whatever organisations you’ve been involved in. But it’s possible to go too far the other way.

Like most things from rational choice theory, the sunk cost fallacy assumes you can make reasonable guesses about payoffs, which isn’t as easy to do in real world business. Let’s say that you have a four month execution loop that’s not going well (i.e. you have a bad feeling about it) and it’s just two weeks away from completion. Should you burn the loop and focus resources on something more likely to be useful? Or should you just eat the two week cost in order to get the maximum amount of learning from the data generated? (Bear in mind the data generated may well cause you to update your feelings about the execution).

As with most such things, you cannot talk about this decision in the abstract; this has to be a context-specific case study. I don’t have neat answers for you here, I’m afraid. I’ve been puzzling over this conundrum for a bit; I have no generalisable principles.

But I’ll say this, for whatever it’s worth: in recent years I’ve been in enough organisations where distraction is the norm; I currently lean towards finishing the loop, every loop, and eating the costs in order to learn.

Wrapping Up

There’s a line in the Wikipedia article about PDCA/PDSA that goes:

PDCA (and other forms of scientific problem solving) is also known as a system for developing critical thinking. At Toyota this is also known as "Building people before building cars". Toyota and other lean manufacturing companies propose that an engaged, problem-solving workforce using PDCA in a culture of critical thinking is better able to innovate and stay ahead of the competition through rigorous problem solving and the subsequent innovations.

If you squint at the Deming cycle, you’d realise that this is essentially the scientific method as applied to business. Of course what passes for the scientific method these days is a little debatable (do you count null-hypothesis statistical testing as part of the scientific method?) but that’s besides the point: treating execution as a series of PDSA loops encourages hypothesis generation, experimentation and feedback; it forces you to assume that it’s better to be approximately right than exactly wrong.

I wish I could tell you that there’s some secret trick to doing PDSA. I wish, with every fibre of my being, that there’s some silver bullet here — so that I could use it for myself. But the trick to applying PDSA seems to be that there’s no trick: you somehow have to get your org to close the loops it starts.

Postscript

One final note, as a teaser for future essays on this topic: the PDSA loop and other, related ideas come from the domain of ‘statistical process control’, a field of practice advanced by W. Edwards Deming, Walter Shewhart and others from the 1950s and onwards. You may have heard of this field of practice by more contemporary, faddish names like ‘Total Quality Management’ or ‘Six Sigma’. You may also believe that these ideas are mostly applicable to manufacturing. This isn’t true, not really, but why it isn’t true is a topic for another day.

How I stumbled onto this field is worth a small note. Over the past six months or so, I’ve been working on a small project for Colin Bryar and Bill Carr, the authors of Working Backwards. The goal of that project was to explicate certain best practices of Amazon’s Weekly Business Review (WBR) for use in other, non-Amazon companies. I took on the work because I wanted to learn and then use the WBR for myself; while I can’t write about it yet, executing on it was what led to my current interest in Deming’s body of work.

It turns out that a surprisingly large number of ideas in the WBR came from the field of statistical process control (SPC), and specifically from ideas and principles developed and popularised by W. Edwards Deming, Donald J. Wheeler, and others in the 70s and 80s. I began to suspect that Amazon, more so than probably any other tech company in the world, had successfully adapted Deming and Shewhart’s ideas to software development and technical product management — though with a distinctively Amazonian flavour. The practices that Amazon uses may be SPC-like in nature, but the names and the references are unique to the company; you’re not going to hear words like “quality circle” or “six-sigma” or “high-performance work teams”.

This essay examined a relatively easy concept in the Deming canon: the idea that the PDSA loop is simple and obvious but difficult to do, for the same reason that exercising regularly is simple and obvious but difficult to do. Next week, we’ll talk about a more advanced concept from the same canon, which I find remarkably useful: a model that explains why organisations stop improving and slowly become shittier places to work.

Update: that essay is available here — Process Improvement is Trickier Than You Think.

Originally published 03 November 2022, last updated 10 November 2022.