The Tricky Thing About Creating Training Programs

This is Part 6 in a series on tacit knowledge. Read Part 5 here.

In our previous post in the tacit knowledge series we took a look at Applied Cognitive Task Analysis (ACTA), a simpler technique to extract tacit expertise from the heads of experts around you.

A natural response to that post is … “ok, so after you extract that tacit knowledge — now what?” This is an understandable reaction, because the next step after doing all that skill extraction is to use it to learn — to develop a training program for yourself or for others. The main problem with this is that good pedagogy is difficult to do. And, as it happens, I’ve not spent any time talking about how Naturalistic Decision Making (NDM) researchers turn their insights into actual training programs.

Why? Well, this is simple: I haven’t found a good, generalised resource in the NDM literature on how they do so. What I have found is a place in the NDM world to dig; this post is to tell you all about it.

Some Context

I’ve mentioned previously that the entire field of Naturalistic Decision Making is built around this notion of ‘we’ll interview experts, and then we’ll extract the expertise from their heads, and then we’ll turn that into training programs or use the outputs to redesign user interfaces’. I said that the field is incredibly applied; much of the work was funded by the military or by corporations with Serious Expertise Needs (think: nuclear power plant control systems, fraud detection at Nasdaq, etc). We’ve already spent an entire series of blog posts looking at some of NDM’s techniques for extracting expertise, which is interesting for all the obvious reasons: how many of us, after all, have had the experience where we want to get better but the expertise we desire is locked up in the heads of senior practitioners? There aren’t a lot of places you can go to that specialises in skill extraction; NDM is one of the few subfields of psychology where the researchers have had success extracting tacit mental models of expertise.

What we haven’t done is to look at NDM’s techniques for creating effective training programs.

This is partly my fault, and partly the nature of the beast. It’s my fault because I’m relatively new to the literature — I didn’t, for instance, know that ACTA was a thing until a few months ago, despite the importance of the paper. And it’s the nature of the beast because I think popular NDM writing is very much focused on the ‘skill extraction’ part, and not so much the ‘design a training program’ part. Presumably this is because there are already many resources out there to design good training programs. And perhaps NDM researchers think that their approach to course design isn’t particularly interesting; certainly the individual teaching techniques they use here aren’t particularly novel.

But what is interesting to an academic is very different to what is interesting to a practitioner. I happen to think that NDM researchers have a unique approach to education design, as a result of the skill extraction they are able to do. And I am very interested in some of these techniques, because I want to apply them to my own career.

In fact, I think there’s a particular ‘style’ to the types of training programs that NDM researchers create. I want to give you a taste of that approach through the use of three quick examples. Of course, examples aren’t as good as a set of generalised guidelines, but I haven’t found any. So examples it is.

(Again: if I’m wrong here, please send me an email at cedric@this domain. I’d love pointers to papers!)

IED Defeat

IED stand for ‘improvised explosive device’. You’ve probably seen them in the 2008 award-winning movie The Hurt Locker.

In episode four of the Naturalistic Decision Making podcast, Jennifer Phillips talks about some of her work on IED defeat. She starts out by describing the skill extraction aspect of it:

Soon after 9/11, the military had troops in Afghanistan and Iraq, and if you remember, all the coalition forces had problems with these roadside bombs, these IEDs. At that time, the Department of Defence was trying to throw a lot of money at advanced technologies to detect these devices, and better protection and armour and such in the vehicles to protect Marines and soldiers, and one of the areas that we had the opportunity to get involved with was the human expertise element of this, of detecting where the devices were going be placed, and the reason this project was so interesting to me and so rewarding … it truly felt like we were in a position to make a difference in Marine’s and soldier’s lives.

We got to interview a number of young Marines who had done some really, really challenging jobs, and who had seen some terrible things in their lives, and to be able to extract their expertise and see not only what they had been through, but also, the skills that they developed. I mean, talk about richness of a domain — every single expert-novice difference that exists out there, you see in this domain of IED defeat, of being able to notice these roadside bombs and anticipate where they be located. You see these Marines seeing the invisible, recognising when things that should be there aren’t there, you see how they put themselves in the perspective of the adversary and think about where they would emplace given the terrain, given the time of day, given where the sun is in the sky — I mean, this whole range of rich factors and cues they would rely on to know that they were getting into a danger zone was just fascinating intellectually.

What is interesting about IED defeat is the existence of a clear meta. At around 28:30, co-host Brian Moon starts probing about the particular difficulty of this skill domain:

BM: What I thought was particularly challenging about (this project) is — we think about experts and they develop expertise over time, in this particular domain, because things were happening so fast … the adversary was adapting more rapidly it seemed than our folks. And so expertise in that sense, was extremely contextualised, both in time and space — and so, even as a Marine might develop expertise in a particular area, and understand the cues, those might change when they go to the next region. I always thought this was an interesting aspect of the work, because if you think about expertise in nuclear engineering, for instance, yes, there are changes in that domain, but not at the rapid pace that you saw in IED defeat.

JP: You’re absolutely right. So we were looking at Afghanistan and Iraq, two very different areas of responsibility. You would find, even within one of those countries, the different regions, the different neighbourhoods, the insurgents would have different techniques for emplacing and triggering and all those things. So you would hear stories in the interviews about … in one town, any time there was a rug that was hung over a balcony, then that meant there was going to be an attack that evening. So that’s a cue you would pull out in your interviews, but that’s not one that’s going to transfer to a different context necessarily. So what do you do with that?

So it goes back to the question of ‘how do you take this information and build something that’s going to help someone else in any context?’

We had to work pretty hard on identifying some of those commonalities. One example of a commonality might be a ‘trigger point’ — so if you’ve got a radio-controlled IED, you need to have a spotter that’s located somewhere in the terrain, who can have eyes on the road. And they need to judge where the blue force is — where the Marines are located — at what point to trigger the device based on where they’re located on the road. So in some contexts they would use a telephone pole post, in other contexts they would build their own little rock formation on the road, in other contexts they would use spray paint or something like that to mark a section on the road, so the concept, the technique, was similar across contexts, even though it might have been implemented a little bit differently, depending on what was available in that particular context.

In the end, the training program was implemented as a video game:

BM: Can you talk just for a few minutes about this idea of using games, about using video games in particular, and your mantra that I’ve heard you and Carol mention about this ‘cognitive fidelity’, and how some of the training applications ought to be focused a bit more on this cognitive fidelity.

JP: Sure, so this was so interesting. One of the big findings that came out of that CTA (Cognitive Task Analysis) was that the Marines who were really doing a great job at being able to recognise a danger zone in advance were the ones who were thinking like the adversary. It sounds like a pretty simple thing, like no kidding of course they were able to put themselves in the adversary’s shoes, but we thought this is something that young Marines, the individuals who were deploying for the first time, don’t know how to do. They’ve never had the ability — outside a training environment, and it’s not something that’s really been taught in training, at least not in a standard way — they’ve never had the opportunity to think from the other guy’s perspective. So we used the VBS technology — we teamed with Bohemia Interactive — and we built out a module in VBS where the Marines were role-playing the adversaries, and they were responsible for emplacing IEDs in the environment.

So we were essentially putting them in a position where they had to think through how they were going to be effective in placing an IED. Whether they were going to use a cellphone detonator, or … with another sort of IED, they had to think through ‘When is the blue force convoy going to come through? What is the time of day right now? How can I disguise it? Where would be a good ambush zone?’ All of those kinds of considerations that are happening in the real world.

Now if you know VBS, you know that it’s got pretty good physical fidelity, but not great physical fidelity. So to your question about the cognitive fidelity really mattering … it didn’t matter that the physical video game environment was perfect or that the leaves on the trees were blowing in the wind exactly as they’re supposed to, what mattered was that we were putting the Marines in a position where they had to think through the problem from a different perspective, and that turned out to be very successful.

Tennis Serve Detection

The next podcast episode I want to draw your attention to is episode nine, with Peter Fadde and Olivia Brown. And the example that I want to talk about is Fadde’s research with training tennis serve detection:

Expertise-based training really is a theory — an instructional design theory. We’re probably a pretty small group within the NDM community compared to human factors, cognitive psychology, or some other areas like that. And the theory really is that, hey, we can go out in the field and study expertise, study these amazing, seemingly intuitive decision-making in sports and military and other contexts, but how do we train it? Training isn’t a natural thing, training is an artificial thing. So we’re looking at something and we’re trying to understand it …

So the insight there was “ok, how do they research it?” — and that led me to .. because when I went to do my dissertation, I was looking around for a project that was doable, and I was working in sports at the time — I was working as a video coordinator with a big time football team, and we did video analysis for coaches. And I just felt that more that can be done for player training by putting video and data together, so pulling on that string led me to research that at the time a lot of it was done at Australia, especially at the Australian Institute of Sport, people like Bruce Abernathy really leading that research …

And they developed these techniques, and the techniques themselves was of interest. Because they have to take this incredibly complex, seemingly intuitive performance, that should be impossible. If you look at a goalie blocking a shot on goal, a baseball, a cricket, a softball hitter returning a 135-per-hour serve, these things are not really possible. The human perceptual and motor systems are not sufficient to have any success at this at all.

(…) but how are you going to systematically train it — how can you get more people over the bar faster? And the way you study it in the labs, is with these occlusion techniques, in particular, temporal occlusion, it cuts off in time. Presented by video.

So with a cricket baller, here’s the run-up, here’s the ball out of hand, boom, cuts to black. And the research subject has to say, well, was that an in-spinner, or whatever category it is, so it’s a categorisation task, and it’s a prediction task — is it going to hit the wicket, and if it’s baseball, is it a ball or a strike? So categorisation and prediction. So now we’re starting to really kind of narrow it down. You can say that the macro-cognitive skill we’re after is anticipation.

“Well, how do you train anticipation?” (And people would say): “Well, you can’t train anticipation, that comes from experience.”

Well, ok. Fine. What is the experience giving you? So we can reduce that back to, ok, it’s some pre-event cues and then it’s prediction. Ok — we can test prediction, and we can train prediction. So let’s take something like anticipation which is the quality that we’re after, and let’s try to operationalise it in some way that we can train, which is prediction.

So the thing was to take the video occlusion method that they used for testing, and turned it into a training program.

And this is exactly what they did (22:23):

This really started in the early 80s in Australia, with a focus on cricket and tennis. This is the tennis serve return. So you’ve got video of a server, that’s recorded more or less from the point of view of a tennis receiver. So the server tosses the ball, racket comes through, and then cut to black. And you have to say what kind of serve is that?

Now that cut to black might be right after ball contact, as you see the ball coming off  the strings of the racket, or right before ball contact, or right in it. So they cut it at these different places in here, and this is an expert-novice research paradigm, so they have advanced tennis players and less skilled tennis players (come in to be tested), and where does their performance diverge?

So for instance in that one we find that if they see even one frame of video — that’s the sharpest knife we have here — you’re dealing with about 33 milliseconds in there … if they see two of those (frames) coming off (the strings of the racket), so you’ve got 67 milliseconds, still a very short period of time, then the experts and the novices can pretty much say what type of serve that was — that was a kick serve, that was a flat serve, or that was a slice serve. Now you cut back to where you can only see a little bit of ball flight … well, their performance both goes down a bit, but the expert not as much. Now we cut to black before the racket hits the ball. And the experts are still able to say what type of serve that is — they haven’t even seen the racket hit the ball, and they can tell you 85% of the time whether it’s a kick, a flat, or a slice serve. And they can even predict backhand or forehand side on the return. And at that point the novices performance pretty much drops off. Now if we cut it back a little bit further, they’re both down to random (prediction).

So now we’ve defined our window of expert advantage. From 50 milliseconds before contact, to 50 milliseconds after contact. There’s a 100 millisecond window in there when the experts have a distinct advantage over the non-experts. So that’s the research finding.

Then if we repurpose that as a training technique, we say “ok, here’s the beginning of the window, you should be able to do that, everyone should be able to (detect the serve)”, and then we start cutting it back and you see less and less and less, until you, as a developing tennis player, are able to, like the experts, at 85%, identify the serve 50 milliseconds before the contact.

The interesting thing is that the experts are terrible at explaining what they’re looking for, so they have to look at other techniques to pick up on the cues (28:15):

The experts can’t necessarily see what they’re seeing — they almost put it in the ‘ESP category’, like with the firefighters in (Gary) Klein’s story (in Sources of Power). And if you press them enough, they’ll start making stuff up. The way that experts do, because they want to give you an answer. So you really need to start ascertaining where that is.

And so this is where you can bring in another occlusion technique, which is spatial occlusion. We know we have a window in time where that expert is picking up some kind of cues. It’s not ESP, so we know they’re picking up some kind of cues (but) we don’t know what they are. So what happens if we mask out the racket?

Well, as it turns out, we can mask out the racket, and the novice’s performance will just fall off the table, and it will hardly affect the expert’s performance at all. So that tells us that they aren’t getting their cues from the racket. But if we mask out the lower half (of the server’s body) now, the experts ability to identify that serve when it’s cut off really goes down. So that tells us that somewhere in that lower half, they’re picking up very early cues, so there’s plenty of time to help you make that — what seems like an instantaneous reaction. And you can break it down enough that you realise that if someone is hitting a kick serve, they’re going to be a little deeper in their knee bend, their ball toss is going to be behind their head, if it’s a slice, the ball toss will be a little to the side. Some of those are known coaching things — the ball toss, for instance.

And you see the reverse of it. For instance, in Pete Sampras’s biography, he talks about how — when he was 12 years old, he would toss the ball into the air, and his coach would call out the type of serve to hit from there. So he would have to learn to hit kick, flat, or slice from the same ball toss. And so later, when he played in the US Open against Agassi, considered the best returner of the time, Agassi said that he couldn’t read Pete’s serve. It wasn’t the fastest flat serve, it wasn’t the spinniest kick serve, but it all looked the same.

Fadde’s true contribution to the research literature here is that the training approach focuses on the cognitive elements of a physical skill. Fadde says (16:42):

What’s some way that you can observe some representative task in a measurable and repeatable way? Well, that turns out to describe a training task too, and it tends to be a little bit cheaper. (Because you’re re-using research techniques).

So my initial dissertation research was in applying that. And I had access to a competing college baseball team, and so I evolved a training program out of that, to train their pitch recognition. And that proved to be quite successful, there were measurable results in batting average and such, so that continued to be a focus, but also with this larger thing of saying “not only are we taking this one skill, but this can also apply to other kinds of things.” Particularly ‘in-the-action’ decision making. Not command-and-control type things. So in the news these days, things like attack recognition, defensive tactics, arrest and control by law enforcement, so that you make good, fast decisions, for the good of everyone concerned. All of these types of performances that seem too fast and too complex to tease apart.

And (my contribution) is taking these techniques from the research lab and repurpose them as drill and practice. Drill and practice: the bottom feeder of all instructional technology. What elements of extraordinary performances that we observe in many contexts — they just happen to be more measurable on TV and in sports, but they exist everywhere — how can we reduce that to something that can be drilled and practiced?

And if there’s one big insight there, these performances and the training of them tend to be very focused on what you do, and yet the NDM-type research that you come across, the higher you go in the expertise tree, the more it becomes about what you see, instead of what you do. So we’re going to push that aside, the ‘what you do’ bit — we’re not saying it’s not important, we’re saying it’s already addressed (in current training methods). What’s under-addressed and what gives us the opportunity to perhaps accelerate the development of expertise is this focus on what you see. And it turns out that applying some of these video-occlusion type things can be done on a drill-and-practice level, and they can be done on an iPhone. So we can get a lot of efficiencies out of that kind of approach.

A Tool for Police Investigations

The last podcast episode that I want to draw your attention to is Episode 20, with William Wong.

The bulk of Wong’s work over the past couple of years is “how do we develop better user interfaces for intelligence analysis?” This led him to create the VALCRI project, which is currently in use in the EU, and is described as ‘a criminal intelligence analysis system that facilitates human reasoning and analytic discourse, tightly coupled with semi-automated human-mediated semantic knowledge extraction.’

It is, to put it simply, the real world version of the crime investigation system from Minority Report.

And it looks like this:

Wong covers the story of the project in the podcast at about the 29:26 mark, explaining that “when we started the work, I was really intent about designing a system that would focus on actually creating an environment that would enable insight. Particularly the kind of sensemaking described by Gary Klein’s Data-Frame model, and particularly his triple path model of insight (…) I wanted them (in the consortium) to think about sensemaking and insight, because if you don’t have that, all you’re going to build is a search and retrieval system, which is what every other intelligence analysis system is about!”

That cognitive model for insight that they unearthed from doing CTA on criminal analysts is covered in this VALCRI video:

And the three research papers that informed the design of the project were the following:

1. How Analysts Think: Inference Making Strategies
2. How Analysts Think: Anchoring, Laddering and Associations
3. How Analysts Think: Intuition, Leap of Faith, and Insight

What I took away from the VALCRI project is really this notion of — as Wong puts it — “if you don’t have this notion of building for sensemaking and insight, then all you’re going to build is a search and retrieval system!” The history of the VALCRI project shows you how to do it right.

Wrapping Up

What can we draw from these three examples? If I may generalise timidly, I would say that all three show an approach where the researcher extracts the cognitive processes of experts, and then teaches the novice by making them simulate those same cognitive processes. The key word here is simulation:

• The IED defeat training program focused on high cognitive fidelity to the real-world task; the video game was simply a medium to help students acquire the mental model that seasoned marines use to predict IEDs emplacement.
• The tennis training program worked because it took apart this tacit macro-cognition aspect of tennis serve recognition, and then taught it to novices by simulating through an occlusion exercise. Fadde could have taught it via lecture. He didn’t.
• The VALCRI project extracted the insight generation process that criminal intelligence analysts used, and then designed an entire system around it.

You’ll notice that all three examples are drawn from the NDM podcast. This isn’t an accident! My current hack to learn the NDM method of designing training programs is to listen to as many NDM podcast interviews as possible. My goal: to collect as many prototypes of effective training interventions as I can.

‘Listen to the NDM podcast’ happens to be my recommendation if you want to do the same. It strikes me as somewhat ironic: the best practitioners of tacit knowledge extraction have yet to explicate a coherent explanation of their training methods. You’d think that they would have a generalised framework for training by now. Or at least a review paper. And, again … perhaps they have! Perhaps they have a review paper somewhere, but I haven’t found it yet.

I’ll tell you if I do. Till then, the best that we have is casual consumption of the NDM podcasts. I recommend you join me in doing so. If you're as obsessed with learning as I am, I think you’ll enjoy it as much as I do.

Read Part 7 — An Extracted Tacit Mental Model of Business Expertise.

Originally published 19 May 2021, last updated 07 September 2021.