I have just returned from some good work in Moscow, Russia, and some of the most interesting conversation centered around Level 2 of the Kirkpatrick four level evaluation model. For those of you who might be unsure, Level 2 is the extent to which participants acquired the knowledge, skills, and attitudes that training programs are designed to provide. (Level 1 is reaction, Level 3 is behavior, and Level 4 is results).
There was a misunderstanding that Level 2 was synonymous with pre and post tests. While this is a common way that learning determinations are made, it is not the only one. And furthermore, it is the most difficult to measure. And if that is not enough, many professionals insist on administering pre and post knowledge tests out of tradition, not out of need.
My belief about Level 2 is this. You must consider usefulness and credibility. Usefulness means that you determine the reason that you want to collect from Level 2 data in the first place, and then you determine what method will best serve those purposes. Most people do it backwards - they decide how they want to evaluate learning, then they make up a reason. Credibility is important in asking the question, “What kind of evidence will I need in order to convince stakeholders that learning has occurred?” Some are fine with the word of the facilitator, who has ascertained the level of learning during the actual delivery of the program. Others may want to see post-test scores. Still others want to know that role plays and simulations were conducted, and that participants actually demonstrated their acquisition of the intended skills. I strongly suggest you find out what kind of L2 evaluation is indicated, and then match the various methods to those expectations.
I can think of 7 good methods just off the top of my head. They include:
 |
Tests Pre and post knowledge or post knowledge tests are common. It is difficult to build valid and reliable tests, so I usually suggest you utilize experts who know how to do it. Otherwise, the data may be inaccurate, and can cause hard feelings among participants. |
|
Knowledge Checks This can occur either during or immediately after the delivery of a course. One version of this is where the instructional designers have woven in opportunities for the facilitator to “check” to see if participants are “getting it”. Asking them to comment on scenarios or provide details or personal examples of concepts are examples of this. Of course, a good facilitator or trainer knows how to do this whether instructions are provided or not. A disadvantage of this is that one will not accumulated actual data, but they will acquire information as to whether or not participants are picking up the intended knowledge. A second version of this is to have someone back at their department ask them to share key concepts that they learned - again, a knowledge check. |
|
Role Play Observations This is a great way to evaluate skills attainment during a session. Many professionals use a peer observer to watch a role play or simulation. They use a checklist, provided by the instructor, to ‘check’ for proper demonstration of skills. |
|
Teach backs A really good way to see if someone has learned something is to provide them the opportunity to teach someone else. They may not always appreciate the offer, but it helps them to pay attention during the program, as they know they will be responsible for teaching it to others - usually work peers - after the session. It is also a good way to cross-pollinate learning. |
|
Action Plans I like to have people develop action plans after completing courses on leadership, creativity, or coaching. I then review them to see if their action steps are in line with what I have intended them to learn. |
|
Surveys Few professionals use surveys to evaluate L2, but I like them. One I particularly fond of is where you write down the learning objectives for the course, then have the participants rate their level of understanding of each objective, both before and after the program. It is a quick and inexpensive pre- post-assessment. I will admit that it is not a highly reliable measure, but is a good indicator. |
|
Refresher Course Learning is a process, not an event. Most people learn best when they are given information, then immediately allowed to use that information on the job. Make sure that the refresher courses or modules allow the participant to demonstrate their level of knowledge or skills during the sessions. |
I must admit that one of my motives for presenting easy L2 evaluation methods is to free resources for Level 3. If you only focus on Levels 1 and 2, you never get to the most powerful level - Level 3. At least consider methods other than pre and post tests as you work through this critically important level. Please also keep in mind that we at SMR - and in particular our CEO, Dr. Palan - are experts in one of the most practical and useful aspects of level 2- competency management.
SMR USA's Public Sessions, 2008
|
