A case of diminishing returns?

There is no more persistent topic among the members of the training community than that of evaluating training. 

Despite all the attention paid to the subject, the main form of evaluation continues to be the ubiquitous 'smiles test', a way of gauging trainee reactions to the training. Reactions, of course, constitute Level 1 of the well-known Kirkpatrick model, named for its creator Donald Kirkpatrick. Other levels of evaluation in this model include Level 2 - the learning that occurs during the training - Level 3 - behaviour changes on the job that are attributable to what was learned - and Level 4 - business results or business value stemming from changes in on-the-job behaviour that are attributable to the training (eg reduced error rates or increased productivity)¹.  

Although evaluations at Level 2 are common, the pattern in efforts to evaluate training is clearly one of successively less effort devoted to evaluation from Level 1 through to Level 4. Why is this? If trainers are out to make a difference - to add value through training - why don't they regularly and systematically go beyond Levels 1 and 2? Is the diminishing effort devoted to evaluating training somehow tied to diminishing returns? Does the return on investment in training evaluation decrease as one goes from Level 1 to Level 4? Are there barriers to evaluating training at Levels 3 and 4?

I believe there are good reasons for the pattern of diminishing effort devoted to evaluating training and I will set them forth in this article.

The chief reason we focus on Level 1 is that it presents a fundamental requirement that must be satisfied. If the trainees are uniformly unimpressed with a given training offering, if they see no value in it, if they are bored or turned off by it, that training offering is not likely to continue. The trainees are one of the primary stakeholder groups when it comes to training and their views should not, and cannot, be taken lightly.

Although it would not be fair to say that evaluations at Level 2 are rare, it does seem to be the case that such evaluations do not occur as often as many people believe should be the case. There is good reason for the low level of Level 2 evaluations: a great deal of training lacks specific, job-related learning objectives. Without them, there is little basis on which to conduct an evaluation at Level 2.

Another reason we rarely get beyond Level 2 is that behaviour - and performance - are both a function of individual and environmental factors. Kurt Lewin asserted that behaviour is a function of these two sets of variables and Douglas McGregor made the same point regarding performance. Training addresses the individual factors in the behaviour and performance equations, usually skill and knowledge gaps, but it does not, and cannot, address the environmental factors. This means that behaviour change on the job is often unlikely to occur solely as the result of training. Environmental factors must be addressed as well.

But, even if the trainees love the training, even if we have specific, job-related learning objectives and do a fine job of achieving them, even if the work environment supports the behaviour changes enabled during training, and even if these behaviour changes produce results that have business value, we are still not likely to see much evaluation beyond Level 1 or 2. Why? Because common sense says there's no need to: everyday feedback from the workplace will confirm whether or not the training is making a difference or adding value. Expensive evaluations make no sense.

Finally, in many situations it would be politically imprudent of trainers to lay claim to credit for business results. The people who actually achieved them would take a dim view of such a move. The managers responsible for those results and for providing a proper work environment would look askance at such a 'glory grab' as well. In the end, trainers would do well to pay heed to what other consulting professions have known for a long time: make your client look good and your client will take care of you.

Reference:

1 Some authors propose a Level 5 - the return on the investment in the training. But, if business value can be determined at Level 4, the ROI of the training (its value against its cost) is a relatively simple matter of calculation. In short, there is no need for a Level 5. Moreover, ROI is most useful before an investment is made, not afterward.

Fred Nickols is managing partner of Distance Consulting LLP. He can be contacted at fred@nickols.us or via www.nickols.us