Probably the most well-known and widest used measures of training impact are those set forth by Kirkpatrick (Levels 1 – 4) and Phillips (adding Level 5 ROI). These levels define the effects of training from a participant’s initial reaction to the course (typically measured immediately after the event) all the way through to its effect on the bottom-line (measured months after completion to allow for a participant to apply his/her learning and experience on the job). The five levels and what they evaluate are summed 

up below:

 

Did they like it? Level 1 measures the extent to which training participants react positively to the training. Were they engaged? Was it worth their time? Were they satisfied with the design, the content and the way it was delivered? Level 1 should also include questions about relevance to the job and a participant’s intent to apply the training. The Level 1 surveys should be no longer than 5 minutes and are typically administered immediately after the programme to capture impressions while they are still fresh. It should also ask participants for open-ended or qualitative feedback (e.g. What would you do to improve this training? or What was the best/worst part of this training experience?).

 

Did they learn anything? Level 2 measures the extent to which new knowledge and skills were acquired during the training. Are they leaving with critical knowledge and capabilities that will help them do their jobs better? This may also include an attitude change or a new way of looking at an issue or problem. The data here tells us whether they have acquired some valuable bit of learning that gives them the potential to perform better but it stops there. Although we hope that it predicts improved behaviours and performance on the job, there’s no way of knowing how this new learning affects their performance until after the employees are back on the job and higher levels of data can be collected. The Level 2 can be a knowledge check at the end of the programme, a difference score between a pre- and a post-test or it can simply be a few questions asking participants to describe their acquired knowledge or skills in some quantitative (rating) or qualitative format (What was your biggest piece of learning?)  

 

Are they doing anything different? Level 3 measures the extent to which participants are returning to their every day jobs and actually applying what they learnt in training. Do they do something better? Do they do something more effectively or more efficiently? Level 3 is probably the most critical measure because if behaviour change doesn’t happen and nothing gets applied, then all the training impact stops there. No business metric improvement could ever be attributed to the training and certainly no ROI could ever be calculated if participants don’t do anything differently. Conversely, if you can demonstrate that your training improved the way employees do something, you’re literally half way there. The behavioural or performance improvements can then be quantified and analysed to see if they predict increases in the critical metrics that matter most to the business. Level 3 typically comes in the form of a survey administered three to six months after the training concludes. Quantitative rating scales of improvement in key behaviours and qualitative open-ended questions are used to collect the necessary data.  

 

Did it impact the business? Level 4 measures the extent to which training is improving critical business metrics. That is, did the behavioural improvements and applying the new knowledge and skills actually lead to better business metrics and higher performance? What was the increase in productivity? What was the increase in sales revenue, customer satisfaction or amount of widgets produced? What were the real benefits to the business? Here is where you want to say with both correlation and causation that your Level 3 behaviour changes actually predict your Level 4 business results. Here is where you need to identify the most important metrics to the business, define how much they improved and then isolate how much of that improvement was directly attributable to your specific training programme. Level 4 data is typically collected by partnering closely with the business and honing in on only those metrics that truly drive the success of the company. This data, like the Level 3, must be collected several months after the programme to allow time for transfer and effect. 

 

Was it worth the investment? Level 5 measures the extent to which the benefits of a particular training experience outweigh the costs of that training experience. The final ROI is expressed as a percentage of the original investment. The ROI calculation can only be done when the benefits identified in Level 4 have been turned into values for the business or monetised. Once monetised, they can then be compared to the total costs which also all have been monetised.  

 

 

 

Level 6 is an evaluation that simply measures your learning climate and tells you which environmental factors are most affecting the impact of your training back on the job. That is, what are the environmental factors (things that are happening in the employee’s immediate work environment) that are either supporting or sabotaging the impact of training back on the job? Once you know which climate factors are causing your training to succeed or fail, you can then simply design and construct a stronger, more effective bridge. This fortified bridge will then keep employees safe as they traverse the treacherous path from training to business results and help them avoid the many pitfalls that all too often cause training to fail.  

 

I call this Level 6 a ‘transfer climate’ analysis because it tells us exactly which climate factors are most important when employees attempt to transfer what they learnt in training back to their every day jobs. In fact, the amount of behaviour change, productivity improvement and return on investment that all participants are able to achieve will solely and absolutely depend on the success of this transfer. These climate elements are so powerful at predicting the overall impact of your training that they can quite literally make or break your entire initiative. It is the most critical because the results of this additional level of analysis can ultimately tell you how to improve the impact and ROI of any training programme within your organisation.  See Table 1 below with the new level added.

 

 

The first thing you’ll need to do to identify your most critical climate factors is to develop a transfer climate assessment. This is simply a list of possible suspects that you believe are either helping or hindering participants from transferring their learning back to the job. You simply attach these questions/items to your Level 3 post-training assessment and ask participants to score each factor. You then correlate these scores to your Level 3 and 4 business impact measures and the highest correlations will be the biggest and best predictors. Fortunately, you don’t have to start from scratch. While all organisations are different, after conducting years of these impact and transfer studies, the most powerful of our usual suspects have been identified.   

 

The sample assessment on p45 consists of 12 climate items that are most predictive of training impact across all training programmes, organisations and levels of employees. Whether it is a leadership programme for executives, a sales programme for service reps or technical training for finance managers, these climate factors are the strongest influencers of resultant business impact and ROI. While many others can be significant and far more specific within unique organisations, the list of factors here (or some derivation of these) typically turn out to be the most significant and conspicuous make or break factors in the group.  

 

After many years of presenting impact studies, I have yet to encounter a business group or client that does not find this Level 6 evaluation the most interesting and informative piece of the overall results presentation. Measuring Levels 1 – 5 is critical but if you stop there you’ve only told your stakeholders how their training is working and not why their training is working. Level 6 allows you to answer the most critical questions of all – why does training work for Mary and not for Joe? What can we do to change their climates and improve their training’s impact in the future?  

 

As training professionals, we put such incredible resources into trying to find the most effective training solution out there, when the first thing we need to fix is what’s in here – our internal training climate. We spend so much time and money looking for the next best thing and dreaming up the most creative ways to deliver it that the true predictors of success and impact slip right through our fingers. Instead of being so quick to look for new training or different training, our first and hardest look should be at our current training and the simple climate factors that we can fix right now. And with leaders at all levels of the organisation being able to implement and effect these positive climate changes, the importance and urgency to create a great climate should swiftly move to the forefront of any training strategy. Especially at times when you need to stretch that training budget, measuring your levels and capitalising on these factors can help you deliver your initiatives in a way that squeezes every last penny out of those precious training pounds.