The Hard Thing About Learning From Experience

The hard thing about learning from experience is that you don’t know how to make sense of your experience.

Let’s unpack that by working our way through a series of questions.

Have you ever had an experience where you failed and then concluded a bunch of ‘lessons learnt’? The answer to this is likely ‘yes’ — learning is universal, and this describes most (if not all!) of us.

Let’s take it a step further: have you ever had an experience — say, many years after that first failure, after your initial set of learnt lessons — where you had a new learning moment, and this new learning moment made you rethink the lessons from that earlier experience?

Here’s one of mine (which I've written about elsewhere): “Oh, I suppose my failed cofounder relationship with Jack wasn’t because he was a bad fit; I have a new employee today who is very much like Jack in his personality. Perhaps a better explanation might be that Jack does not deal well with uncertainty, which explains why he’s happy and successful in the large corporate job he has today. So maybe the right conclusion isn’t to avoid Jack-like people, but to test for comfort with uncertainty …”

Sometimes this new interpretation of an old memory is triggered by new data, generated by more recent experiences. But sometimes the new interpretation occurs because you have a better explanatory model with which to organise your experiences. A concrete example: I’ve begun to reread business biographies through the lens of Lia DiBello’s triad model of business expertise, and I’ve found it quite useful to reinterpret my own business experiences through the lens of the triad. One implication that falls out of this exercise is my realisation that I have a blind spot when it comes to capital — I am not comfortable with selling equity, nor do I make good cost-of-capital decisions. I am therefore not likely to reach for certain types of capital as a tool. This is clearly a problem that I’ll need to fix.

To bring this back to our discussion about learning: I think it is likely that you, too, have had such reinterpretive experiences, though I wouldn’t be surprised if you also say that you’ve never had one. Over the years I’ve learnt that introspection is likely distributed like a bell curve over humanity — as is the case with most other human attributes; in practice I’ve often had to adapt to subordinates who were more or less introspective than myself.

Now let’s go a bit further.

Let’s suppose that you’ve had a number of these reinterpretive experiences. Have you ever had an experience where you argued a position or took action that is consistent with the older set of beliefs, even though intellectually — thanks to your reinterpretation of those old experiences — you knew that they were the wrong ones? And have you ever caught yourself in the act of arguing from old belief, and — having caught yourself — sat down with your cognitive dissonance, in order to go through all the implications of your new interpretation? In other words, have you ever done the internal work necessary to destroy those internalised lessons?

I think the set shrinks a bit more with these questions. Not everyone actively works through all the implications when an old mental model is discarded. They often let it happen over time, like juice dripping out from a bruised orange. And for good reason: it’s difficult to invalidate your beliefs. It’s uncomfortable. There’s cognitive dissonance for weeks.

We can go further still with this line of questioning.

After sufficient discards, have you ever thought to yourself: “Ah, I should probably speed this process up and work through all the implications of the new mental model pre-emptively — instead of waiting for the cognitive dissonance to bite me in the ass.”

In other words, you train yourself to act on a cue-action pair. The cue: you notice something that causes you to reinterpret the lessons you’ve drawn from some old experience. The action: you schedule some time to work through all the possible experiences and book summaries and arguments that follow from the old conclusion you had; these are things that must be re-evaluated through the new interpretive lens. You schedule this work up-front, because you’ve been caught so often by the cognitive dissonance you know that it always comes around to bite you eventually.

Finally, we follow this line of reasoning to its logical conclusion:

You realise that you have to keep reinterpreting memories the older and wiser and more skilled you get, so you eventually learn not to hold on to any current ‘lessons learnt’ too tightly, but instead to treat them as contingently true things.

After all, “I was wrong about that in the past; I might be wrong about this in the future.”

(I actually go beyond this and actively wait to conclude lessons from non-fiction books, or from personal experiences. “Here is a thing that’s happened,” I tell myself, “Let’s wait and see what it all means.”)

The hard thing about learning from experience is that you never know what to learn from a recently concluded event; you don’t yet have the models necessary for proper interpretation.

In other words, the hard thing about learning from experience is that you don’t know how to make sense of your experience.

Cognitive Transformation Theory

Everything I’ve just said is a restatement of Gary Klein and Holly Baxter's Cognitive Transformation Theory. We last talked about CTT in my piece on cognitive agility, where I explained:

Cognitive Transformation Theory is a macrocognitive theory of learning. Macrocognition is a fancy word that means ‘observed cognition that is performed in natural settings, instead of artificial (laboratory) environments’. What this implies is that Cognitive Transformation Theory describes how people learn when goals are ill-defined, when conditions are messy and complex, when there is no clear feedback, when the pupil has no formal instruction, and when there exists no good pedagogical development in the domain. In other words, CTT is a theory that describes how people acquire expertise in real world environments. Amongst other things, it is a theory that tells us how people become masters of business.

I said that CTT was worth digging into because it tells us how we learn from trial and error — and let’s face it, most of us are forced to learn from trial and error in our work domains. The key activity we do is something Klein and Baxter call sensemaking:

The field of science education assumes a knowledgeable teacher attempting to convince students to accept scientifically acceptable theories. In contrast, many cognitive learning situations do not come equipped with knowledgeable teachers, and the learners have to discover for themselves where their mental models are wrong and how to replace them with more effective ones. (…)

What is hard about learning cognitive skills is that none of the traditional components of learning — diagnosis, practice, feedback, or training objectives — are straightforward. Each of them depends heavily on sensemaking. Bloom’s (1956) taxonomy includes a component of synthesis — building a structure or pattern from diverse elements; and putting parts together to form a whole, with an emphasis on creating a new meaning or structure. This corresponds to the process of sensemaking.

A simpler way of describing sensemaking is that we ‘reflect on experience’. In other words, Klein and Baxter argue that learning in real world, unstructured environments demands that you sensemake on diagnosis and practice and feedback and training objectives. These are all wicked things:

On Diagnosis: Diagnosing the reasons for weak performance depends on sensemaking. The instructor, whether in person or virtual, has to ferret out the reasons why the student is confused and making errors. Sometimes trainees do not even notice errors or weaknesses and may resist suggestions to overcome problems they do not realise they have. Even if trainees do realise something is wrong, the cause/effect mechanisms are subtle and complex (emphasis added).

On Learning Objectives: … for cognitive learning, the objectives may be to help the students revise their mental models and perhaps to reorganise the way they categorise events. For example, both Kolb (1984) and Dewey (1938) focus on learning through experience. What is important in Kolb’s reflective observation stage is how the learner transforms an experience into learning through reflection. During reflection, the student compares the new learning to what is already known and tries to make it fit with existing knowledge and sees how to leverage this new knowledge for additional learning (emphasis added).

For Dewey, the key is what the learner does with experience. Not all experiences are equal and not all experiences are educational. According to Dewey, individuals reflect on their experiences to learn what thoughts and actions can change real world conditions that need improving (emphasis added).  Dewey thought that people were constantly trying to resolve perplexing intellectual situations and difficult moral situations.

(…) We further assert that novices may not have mental models for an unfamiliar domain and will struggle to formulate even rudimentary mental models linking cause to effects. Their learning objective is to employ sensemaking to generate initial mental models of cause/effect stories, whereas experts are revising and adding to current mental models (emphasis added).

On Practice: Providing students with practice is necessary for gaining proficiency. But with cognitive skills, practice is not sufficient. For cognitive skills, trainees often may not know what they should be watching and monitoring. They need adequate mental models to direct their attention, but until they get smarter, they may fail to spot the cues that will help them develop better mental models  (emphasis added).

On Feedback: Providing students with feedback will not be useful if they do not understand it. For complex cognitive skills, such as leadership, time lags between actions and consequences will create difficulties in sorting out what worked, what did not work, and why. Learners need to engage in sensemaking to discover cause-effect relationships between actions taken at time one and the effects seen at time two. To make things more complicated, learners often have to account for other actions and events that are interspersed between their actions and the consequences (emphasis added). They have to figure out what really caused the consequences versus the coincidental events that had nothing to do with their actions. They have to understand the causes versus the symptoms of deeper causes, and they have to sort out what just happened, the factors in play, the influence of these factors, and the time lags for the effects.

To add to these complications, having an instructor or training tool provide feedback can actually get in the way of transfer of learning, even though it increases the learning curve during acquisition (emphasis added). By placing students in an environment where they are given rapid feedback, the students are not compelled to develop skills for seeking their own feedback. Further, students may become distracted from intrinsic feedback because it is so much easier to rely on the extrinsic feedback. As a result, when they complete what they set out to learn, they are not prepared to seek and interpret their own feedback.

If you find yourself nodding your head to all of these remarks, then, well, you’re not alone. Klein’s and Baxter’s paper is probably the first I’ve found to describe how we actually learn in the real world. How do we learn to love? How do we learn to find business partners or co-founders? There are no courses here, and no mentors; there is no deliberate practice. We just have … experience.

About 10 pages into the paper, Klein and Baxter finally present their core thesis:

Our central claim is that conceptual learning is discontinuous rather than smooth. We make periodic advancements when we replace flawed mental models with better ones. However, during the process of cognitive development our mental models get harder to disconfirm. As we move further up the learning curve or have more expertise, we have to put more and more energy into unlearning — disconfirming mental models — in order to accept better ones (emphasis added).

We do not smoothly acquire knowledge as in a storehouse metaphor. Our comprehension proceeds by qualitative jumps. At each juncture our new mental models direct what we attend to and explain away anomalies. As a result, we have trouble diagnosing the flaws in our thinking. Because of problematic mental models, people often misdiagnose their limitations and discard or misinterpret informative feedback. The previous mental model, by distorting cues and feedback, acts as a barrier to advancement. So progress may involve some backtracking to shed mistaken notions. In addition, flawed beliefs have also influenced the way people encoded experiences in the past. Simply changing one’s beliefs will not automatically change the network of implications generated from those beliefs (emphasis added). As a result, people may struggle with inconsistencies based on different mental models that have been used at different times in the past.

CTT has a number of interesting implications. One implication is that destroying old mental models must be a pedagogical goal if you want to accelerate expertise; the paper cites Lia DiBello’s work with ‘Strategic Rehearsals’, which first destroys a company’s shared mental model of business before introducing a new one.

Another implication is that there are increasingly sophisticated levels to sensemaking, and that you may learn them. I walked through a few at the top of this piece, and pointed out that if you are sufficiently reflective, you should eventually learn the lesson of not concluding anything concrete from a recently concluded experience.

But what if you want to go beyond that? If CTT is accurate, then it implies two more things:

1. First, that experts who become good through trial and error got there through superior sensemaking. This is straightforward: if sensemaking gets harder as we climb up the experience curve, it follows that experts must have more sophisticated sensemaking strategies compared to novices and journeymen.
2. Therefore, second: it is the sensemaking strategies of experts that we should target if we want to get better at learning in real world environments.

I’m still working out the full implications of CTT. The most surprising thing to have emerged from the theory is simply that the key to learning in real world environments effectively bottlenecks on good introspection.

And to that, who would’ve guessed?

For more, see: Klein, Baxter (2006) Cognitive Transformation Theory: Contrasting Cognitive and Behavioural Learning.

Originally published 25 August 2021, last updated 25 August 2021.