Means and end
Richard Griffin talks to Jim Kirkpatrick about how training evaluation has changed in the last 50 years
There is no doubt that one approach has dominated the modern field of training evaluation and that is Donald Kirkpatrick's. Indeed, the word 'Kirkpatrick' has become synonymous with training evaluation. The fact that the model, albeit in a modified form, is now entering its sixth decade also means the Kirkpatrick organisation is in a unique position to assess how evaluation has changed over the years and the challenges it faces.
Now more than 50 years old, the organisation has recently refreshed its approach to evaluation. The New World Kirkpatrick model retains the original four levels (reaction, learning, behaviour and results) but focuses particularly on the transfer of training into the workplace.
Other features include:
- strong business partnerships with business leaders as the cornerstones for success
- beginning the training process with targeted, measurable business results
- introducing return on expectations as the ultimate method of demonstrating the value of training
- a packaged or formula method to maximise results.
Donald's son Jim (pictured above) and daughter-in-law Wendy now run the business. I recently interviewed Jim, for TJ, about his views on evaluation, the Kirkpatrick approach - including criticisms of it - and his thoughts for the future
I started by asking him why he thought that the Kirkpatrick approach has been so dominant. He said: "I've asked my dad that many times and his answer is that, over the years, people have told him that what he has done is break down the elusive term 'evaluation' into four practical words: reaction, learning, behaviour and results. I think the simplicity of it and, yet, the power of it has allowed it to remain the premier evaluation model in the world."
I put it to Jim that a number of commentators in recent years have argued that training evaluation is in crisis, largely because it struggles to demonstrate whether training interventions have any impact on the bottom line. He agreed but went much further. In his view, it is training itself that is in crisis: "A year and a half ago, Wendy and I wrote a book called Training on Trial. Basically, our contention was that the training industry is still hanging on to the belief, the myth, that the formal training event is powerful enough to bring about significant results for the business. In working with our many ambassadors around the world, we have found that this simply is not true. Researchers battle over this point, and unfortunately there are still many training professionals who believe it's true; they're losing their jobs and positions very rapidly.
"In answer to the question, yes, training evaluation is still in crisis. In fact, training in and of itself is in crisis because it has not been delivering evidence of value to the business."
Jim turned to why he thinks L&D professionals do not attempt to carry out impact evaluations. He pointed to the fact that most seek to assess the effectiveness of training itself rather than the effect it has on performance. The focus has been on what are described as affect reactions (satisfaction with the training event) rather than utility reactions (how useful the training is). He warned: "I know I'm going to upset some people here, but training has been rather arrogant over the past 50 years by thinking that business leaders, supervisors and executives are interested in training statistics and data, such as smile sheet scores, pre-impulse test scores, number of people trained, competencies, learning objectives etc.
"Unfortunately, that's where evaluation has pretty much been stuck: what we call Kirkpatrick's levels one (reaction) and two (learning). The higher levels, three (on-the-job behaviour) and four (results), have basically been ignored. To return to the question, they've been struggling with training impact because they've been looking more at training activity rather than training impact."
He went on to argue, as others have, that there is a need for L&D professionals to reach out beyond the training function. Too often training is seen as an end in itself rather than a means to an end and is disconnected from the rest of the organisation. He told me: "We've been so hung up on dealing with what's happening in the classrooms that we really haven't spent enough time crossing the bridge from the training world to the business world, evaluating what our graduates are doing, at level three, with what they learned at level two, and what kinds of business results are coming from the training and the reinforcement.
"So I think it's just a matter of reinventing and redefining what training's purpose is. The purpose is not to deliver training. Let me caution you here, if indeed you were going to continue on the path of training design, development and delivery: it's only a matter of time before you are replaced by an app on somebody's iPhone. You need to be working in the world of performance at level three and impact at level four. I hope and believe that this struggle is coming to an end."
In the last 20 years, the Kirkpatrick approach has been subject to considerable critical research, the majority of which has focused on the fact that positive outcomes in one level do not predict outcomes later on. Trainees might, for example, be extremely positive about their teaching but factors like workload or lack of manager's support may prevent them from transferring what they have learned into improved job performance. I asked Jim about his views on these criticisms and whether he thinks they are justified.
"Well, we see a lot of those, and typically they are not studies at all; they are just opinions and editorials by people who are trying to promote their own model. Unfortunately, many of them come from the academic world. What they're doing is commenting on the old Kirkpatrick model, kind of the Don Kirkpatrick model, which is really just a simple one, two, three, four about training evaluation. They're not looking at the New World four levels, and it is really a more powerful, more engaging, model for business people. We think that their opinions, although we respect them, really are based on outdated materials," he said.
"Most true studies, I think, actually support the model because, when they follow the New World Kirkpatrick model, they do find that value is being created through the levels and demonstrated through what we call a compelling chain of evidence, from pointed training at one end to business results on the other end."
This seems a fair response to me - the Kirkpatricks themselves have never suggested that evaluation should stop at level one or two, or that findings at these levels should be translated into ultimate outcomes. In fact, Don Kirkpatrick never claimed that his approach was a comprehensive model of workplace learning. As Jim says, there have been a number of changes made to the Kirkpatrick model in recent years as it has evolved into the latest model. These changes seek to address some of the problems we discussed, although as far as I am aware the new approach has not yet been critically reviewed.
Jim explained how the Kirkpatrick model has changed: "We have made a lot of enhancements, mostly based on research and on practitioners in the field helping us understand new ways of supercharging the model.
"First of all, with level one, we made sure it's not just a measure of customer satisfaction but, instead, really about relevance to, and the involvement of, the participants. It's not just measuring if the participants are happy with us; it's asking if they are really with us, really taking responsibility for their own learning.
"We've also supercharged level two by adding the words 'confidence' and 'commitment' to it, making sure that we don't just leave people with knowledge and skills to fight it out for themselves. We want to make sure we talk with them about confidence and commitment, and help to build bridges for them with business people so that, when they get to the other side, the business side, the work side, their jobs, there is a support network there of accountability, coaching, involvement and additional learning that will really help them be successful.
"So we've enhanced levels one and two, and we've talked about building bridges, tactical bridges to the supervisors, to the in-the-trench people, as well as strategic bridges to the executives of the business world and government.
"The third thing we've done is to enhance level three. No longer do we just look at the change in behaviour, we're looking now more closely at the drivers of those behaviours. The behaviours don't just happen because we hope that they do, because we've done good training; there has to be that performance support network that holds people accountable, that encourages them, recognises them, rewards them, reinforces them, monitors them, to ensure application of level three and then subsequent level four results.
"We've also put more depth into level four in the form of what we call leading indicators. It used to be that we would look at that flag at the top of the mountain and hope that somehow our training could impact that ultimate level four result. Well, we now have these leading indicators that are early signs, both measurements and observations, that the initiative is on track, that there are signs of an early success. This is part of what we call the early warning detection system that allows us to identify problems and barriers and fix them before it's too late and before the results are negatively impacted.
"Affectionately referring to my father, Don, we've also turned Kirkpatrick on his head. What that means is the four levels don't start with level one, with training; they start with the goals of the business. We start at level four - understanding, agreeing on the goals, the criteria for success. Then we go back to level three - what behaviours need to be implemented on a sustained basis in the workplace in order to bring about those results. Then and only then do we talk about the kind of training and learning that people need to go through in order to improve their behaviour and bring about results."
The world of training evaluation is experiencing a period of rapid change. There is, for example, a growing body of research into the factors that support the transfer of training into improved job performance. I asked Jim what changes he thinks we might see in the future.
He replied: "Yes, there is a growing body of research, practical on-the-job case studies and best practices that are showing that the key to ultimate training success is that bridge and the required drivers that will greatly increase the likelihood of application. Execution, implementation, application, performance: those are the key words to our future. As training professionals, our future depends on being tied to those actions, on serving as the civil engineers who are building the bridges from training to the business, and also orchestrating and directing the entire package or formula for training success.
"We don't just spend our time, as our predecessors have done, in the world of training events but, rather, we focus on the whole package of learning and performance, which is really where our future lies."
Finally, I asked Jim if he had any advice for TJ readers. He called on L&D professionals to think outside the box and connect with others in the organisation to become a 'strategic business partner' .
"You, too, need to get out of your own country, your own familiar cubicles of the office training world and cross the Channel, cross the North Sea, and go into the world of business. Be seen there and be talking to people, listening to the supervisors, your training graduates and executives, finding out their world, their problems, their needs, their language. Then learn to speak their language, and to be seen not just as a training provider, but as a strategic business partner," he urged.
The Kirkpatrick approach is not - by a long way - the only method of evaluating training. It is, though, the approach that most people have heard of and many organisations (and academics) use. Its recent New World development seems to echo more general changes in approaches to evaluation, some (but not all) driven by new research. These include the importance of aligning training functions with wider organisational objectives, to shift the focus from evaluations of training to evaluations of the effects of training and to recognise that there are a variety of stakeholders interested in training and that most of those won't be in the training departments.
Most importantly, I think, Jim points to a fundamental challenge - the need to demonstrate that training really does make a difference; that it is a means to an end, whether that is improved customer care, greater staff satisfaction, reduced errors or increased profits.
One of the benefits of being a TJ subscriber is full access to our decades-long archive of content - here we look back to a piece on leadership from March 2013.
Hurix Systems announced today it has been short-listed for Red Herring's Top 100 Asia award, a prestigious list honoring the year’s most promising private technology ventures in Asia.