TJ - The Publication for Learning and Development

Netcheck

By Garry Platt (January 2005 Issue)
0 Comments Comments
Article Rating:

Poor Best

Email to a friend | Print Version

By the time you read this the winter festivities will be all but a distant memory and the New Year will have begun. I hope you enjoyed yourself over the holidays and I also sincerely hope this year contains only good things for you. It is at this time of year that we reflect on our lives and the trials and tribulations that affect us all. As the years go by I become only too aware of the great balance and symmetry of the universe we live in. For instance, when a species becomes extinct you have to remember that at some point, somewhere else in the world, a Starbucks coffee shop opens. Now, doesn’t that make you feel better?

What is it with Starbucks? We have three perfectly good words in the English language - small, medium and large. At Starbucks, however, I have my coffee in a venti, grande or tall. Now when was the last time you went into Marks and Spencer and said: ‘Have you got these underpants in a grande, please?’ And finally, just before I finish haranguing Starbucks, do you know what latté means in French? I discovered after buying my niece one that it actually translates into ‘you’ve paid way too much for this drink’.

Driving into work this morning I was wondering what I should concentrate on for this month’s ‘Netcheck’. During the course of last year I covered return on investment in training, but I have never actually concentrated on Kirkpatrick’s model. Personally, I don’t view any of these levels except level 4 as evaluation. I prefer to use the term ‘validation’ to define what is being undertaken in the first three levels, but that’s me. Most of the training universe does recognise this as evaluation, and so be it.

If you want to learn about Kirkpatrick’s model, there are a couple of websites that I can recommend. The first is www.southalabama.edu/coe/bset/johnson/660lectures/lect5.htm This is a fairly comprehensive study explaining each of the four levels in some detail. For a somewhat shorter explanation you can go to www.nwlink.com/~donclark/hrd/sat6.html#kirkpatrick, which is a page from the famous Don Clark website. What I’m going to do now is look at each of Kirkpatrick’s levels in turn and point you towards interesting websites that may contain some content to help you develop evaluation strategies for each of these levels.

REACTION LEVEL EVALUATION
A great website at www.thirdhouseinc.com/ispiconference2002.doc provides lots of straightforward advice and guidance on designing and implementing what is sometimes referred to as a ‘reactionnaire’ or end of programme review sheet. (Never ever call them ‘happy sheets’. I think it’s a denigratory term and as documents they do serve a useful purpose. ‘Happy sheets’ implies that they are worthless and certainly they are not.) Dr Mary Lanigan has produced a splendid document that outlines a systematic approach to questionnaire design and also explains how some prediction of behaviour level transfer can be gleaned from this approach. It is, however, only a prediction and I have some doubts as to the reliability or otherwise of claiming too much for this kind of approach, but take a look all the same.

A second website is located at www.id-develops.com/showarticle.php?id=3 Here you will find a series of documents that you can print out and use during a programme. They can be downloaded and handed out individually and then collected in to see how people are thinking and feeling about different aspects of the course while it is being delivered. The areas covered include the atmosphere of the course, the pace of delivery, the relevance of the course content, the impact that the trainer is having and the level of understanding being achieved.

These documents are issued during a programme rather than at the end in order to get feedback and change the delivery of the course if it is apparent that the programme is not reaching its goals.

LEARNING LEVEL EVALUATION
The predominant but not exclusive way learning is assessed on training courses is through question and answer sessions, but more systematically and commonly by tests and quizzes. A website that looks briefly at a variety of different approaches can be found at www.unisanet.unisa.edu.au/examsuccess/Types.htm It is a single page but with links to further detailed explanations. It only introduces the topic, and from the candidate’s point of view, but it is a good starting point.

Taking the concept of test question types a stage further is another short document at http://www.sfu.ca/hccc/pdfs/Learning%20Levels%20and%20Types%20of%20Exam%20Questions.pdf While a little academic in nature, it does provide good supporting information for anyone wanting a clearer understanding of exactly what tests can, well, test!

However, for a detailed analysis of the question types available and open to the trainer to use, this is the big daddy: http://www.surveyconsole.com/tutorial/2.html While focusing on survey questions, the types of questions remain the same in test design. The added bonus is that for each area of question design there is a simple online tutorial available complete with examples.

BEHAVIOUR/SKILL LEVEL EVALUATION
At this level we seek to analyse if the individual has acquired new behaviours and is capable of applying new skills. Locating useful and practical information for undertaking this becomes more difficult to find and I suspect reflects how little is done at these higher levels of Kirkpatrick’s model. However, for a broad overview the following website provides some ideas: http://www.sabient.com/training/evaluating_level_three.htmI would also advise looking at http://www.meted.ucar.edu/resource/soo/ntcevalf.htm Although it is a company document introducing Kirkpatrick’s model, it also outlines the seven guidelines for introducing and using behaviour level evaluation. The final web page in this section is http://www.internationalmta.org/1998/9821.html,, a product of the United States Coast Guard. It purports to review a level 3 evaluation strategy for a course delivered in their organisation. As a live case study it may provide some useful insight for people following this route.

PERFORMANCE OR JOB LEVEL EVALUATION
At this level we seek to identify if anything is actually changing in the workplace and if work is being done faster, better or more accurately. For a heavy duty but fascinating look at this level of evaluation, take a look at www.edci.purdue.edu/schaffer/sps/schaffer%20evaluating%20results.htm This website reviews methodologies to evaluate any performance improvement intervention, not just training. As a detailed review it is among the best on the Internet.

However, one of the best models for analysing transfer of training to the workplace has got to be TOTEM, a model designed by the Westinghouse Electric Corporation. TOTEM is unique in my experience and has a novel way of approaching evaluation in the area of transfer into the workplace. The document located at www.edsiusa.com/media/briefs/Transfer_of_Training_Evaluation_Model.pdf is an absolute must for anyone exploring this area. It is a PDF document and you will need the requisite Adobe software to read it.

EVALUATION FRAMEWORKS
For those people wanting a ‘system on a plate’, so to speak, there are some processes and systems that might interest you. One such approach is produced by Leslie Rae, author and book reviewer for Training Journal. The documentation resides at www.businessballs.com/trainingprogramevaluation.htm, otherwise known as BusinessBalls. Rae has produced a set of PDF documents that you can download and utilise or develop to suit your needs. It is unique in that it bonds together a number of elements that both evaluate and can contribute to enhancing the learning experience. The areas that Rae’s approach touches upon are specifically: determining what the participants have learned; giving the learners time to reflect on their learning during the programme prior to completion of their post-training personal action plan; getting useful feedback in an organised manner to help with future training planning; and ensuring trainees and learners follow up their training with relevant actions to apply, improve, develop and reinforce learning attained.

Rae also has a book published called Assessing the Value of Your Training and the first chapter of the book is available online at www.gowerpub.com/TitleDetails.asp?sQueryISBN=0566085356&sPassString=Y&sKeyword It gives several overviews of evaluation and the supporting processes.

I think with all these links you should have sufficient information upon which to draw and create a coherent evaluation strategy building upon the work of Kirkpatrick. Good luck!

This month’s ‘Netcheck’ is compiled by Garry Platt, a senior consultant at the Woodland Grange Management Centre in Leamington Spa where he plays a key role in the centre’s provision of ongoing management training and development across both public and private sectors. If you think this edition of ‘Netcheck’ is good, bad, ugly or simply an affront to 10,000 years of human progress, do let Garry know at garry.platt@wgrange.com As usual, our thanks to Garry for this month’s contribution. Clare Forest will be back next month with the results of her recent Internet searching.

Ratings
http://www.thirdhouseinc.com/ispiconferene2002.doc
Content ****
Originality ****
Navigability ****
Presentation and graphics ***
Downloads and freebies *
Links *

www.unisanet.unisa.edu.au/examsuccess/Types.htm
Content ****
Originality ****
Navigability *****
Presentation and graphics ****
Downloads and freebies **
Links ***

http://www.edci.purdue.edu/schaffer/sps/schaffer%20evaluating%20results.htm
Content *****
Originality *****
Navigability ****
Presentation and graphics ***
Downloads and freebies *
Links *

Quick hits
www.magictricks.co.uk/default.asp
*****
There is a gentle increase in the number of tutors who use or employ magic to emphasise a point, help people to break a state of mind, make a learning point more memorable or simply to introduce an energiser exercise. Go to any large bookshop and you’ll find instructional books on magic and tricks. But, where can you buy ready-made, idiot-proof books that work despite you doing them, and where you get all the benefit simply because of your proximity to the outcome? Well, this website and the company that runs it has, in my experience, got customer service tied down and also has an easy-to-use and navigate website. If I were going to recommend a one-stop shop for buying magic, this would be it. http://www.emagictricks.co.uk
****
This is the alternative website I would use for material I cannot find at magictricks. This website, confusingly call emagictricks, comes a close second for ease of use and speed of service. I haven’t gone wrong to date using either.

 

Back to top | Current TJ

 

Readers Comment

Comment on this story here >

Be the first to comment on this news story