The Key to Evaluating Level 2 (Learning) – Credibility
I have such mixed feelings about evaluating Kirkpatick’s Level 2 – the degree to which participants have acquired targeted knowledge, skills, and attitudes. On the positive side, it is important to conduct Level 2 evaluations for several reasons. First, it is right and fair to ensure that participants in any type of significant learning intervention actually acquire the pre-determined knowledge and skills. This is necessary for them to able to successfully apply what they learned (note, success at L2 does not necessarily lead to success at L3, which is the transfer of learning to on-the-job behavior. Second, having some evidence that learning took place is an important link in the chain of evidence that I often talk about – the way to demonstrate value of learning to the business (note again, see past issues of this newsletter which are available on www.smr-usa.com).
I often see downsides of Level 2. Way too many learning professionals make too big a deal out of it. What I mean by that is this. As I mentioned in an article I wrote on Level 1 a couple of months ago, a general weakness of our profession is that we tend to be trainer-centric. While we talk about being “all for the learner”, many of our activities demonstrate that we are all about “the trainer”. Level 2 is no exception. We often conduct pre- and post knowledge tests whether they are needed or not. Pre and post tests are designed to tell how much learning has taken place as a result of a learning initiative (or training event). Sometimes this is important, and sometimes it isn’t. My strong suggestion is that you make a conscious decision whether this effort is indicated before you undertake it. The reason I say this is that it takes a lot of effort to create tests. Validity and reliability issues are very complicated. And I have seen many excellent learning initiatives get railroaded by complaints about poorly-developed tests. In fact, I have seen examples where participants actually did worse on post test than they did on pre-tests. In one particular case, a client asked me, “Jim, our pre and post tests indicate that participants did not learn what we taught them, but our Level 3 scores indicate that they actually are successfully applying what they learned. Which should we believe?” My answer was, “well, I would believe what I see. I believe that the Level 3 indicators are accurate, and that your tests were flawed.”
Second, there is too much emphasis on evaluating knowledge for skills-based courses. Personally, I teach 2-day and 4-day programs on four level evaluation. Much of what I teach is skills-based, so I make sure that when I am evaluating how well people are learning, that I use methods that include breakout groups, presentations, and other means of observing the demonstration of knowledge rather than the reporting of knowledge.
What upsets me most about misusing Level 2 is that colleagues and clients often tell me, “we are so busy with evaluating learning that we never get to Level 3 or 4.” What do I suggest you do with all of these issues? Read on.
1. |
Use skills-based evaluation/testing for skills based programs. |
2. |
Use knowledge-based evaluation/testing for e-learning modules that precede instructor-led programs. |
3. |
Strongly consider the word credibility. This means it is important to discuss the topic of Level 2 evaluation with your business stakeholders in order to determine what evidence they would find credible. They may want statistically significant data, and they may not. You may find that you are spending a lot of time and money when no one really cares (note: I find that business partners are more interested in evidence of Level 3 drivers and Level 4 outcomes than they are Level 2). |
4. |
Strongly consider using formative evaluation methods. This means evaluation methods that occur during training programs, not afterwards. Make sure these are woven into your instructional design models so facilitators/instructors provide participants with ample opportunity to demonstrate that they have learned. |
5. |
Use pre- and post tests only when they serve a purpose. If you need to do a “short version” of this and high validity is not necessary, consider asking participants after a program to rate each learning objective as far as their understanding or competency before the program versus after the program. |
6. |
If you need to develop valid, credible tests, hire experts. |
7. |
Automate whenever possible. |
Many of you have asked me for what SMR USA is all about. Attached to this newsletter is a condensed version of some of our most popular programs and services. Note the variety of venues, yet they all have something in common – they all reinforce the notion of the business partnership model, which bridges the gap between learning and the business.
SMR USA's Public Sessions, 2008-2009
|
