How to Create E-Learning Courses
Lesson #3 – Assessment of Knowledge Transfer
By Michael D. Lawrence
January 10, 2013
It’s easy to get confused about the purpose behind E-Learning. The reason to use E-Learning is not to reduce costs, but to get business results. It’s important to identify the business goal of your E-Learning program and realize that it is a business performance improvement tool, not just a training tool.
Effective E-Learning drives us toward measurable business goals: increasing product revenue reducing turnover, reducing rework, or increasing customer satisfaction ratings. Before we measure business results of this sort, however, we need to know if the E-Learning program itself is being used and if it is having an impact on people’s knowledge and behavior.
If you’ve ever created any type of training course, you’re probably already familiar with Kirkpatrick’s 4-level model of assessment:
- Level 1: Reaction- Did they like it?
- Level 2: Learning – Did they learn it?
- Level 3: Behavior – Did they use it?.
- Level 4: Results – Did it impact the bottom line?
But many in the instructional design field might add to that another level:
- Level 5: ROI – What is the Return on Investment?
Level 1 is typically done through what is commonly called a “smile sheet”; a simple form completed at the end of the course. Kirkpatrick recommends the following guidelines to get maximum benefit from reaction sheets:
- Determine what you want to find out.
- Design a form that will quantify reactions.
- Encourage written comments and suggestions.
- Develop acceptable standards.
- Measure reactions against standards and take the appropriate action.
- Communicate reactions as appropriate.
Level 2 is often accomplished with some sort of quiz or test, and that is another subject in itself, known as “psychometrics” – the science of writing effective test questions. Perhaps I’ll tackle that topic in another article in the future. Level 2 is usually where most assessment attempts end.
In Level 2 we are looking for:
- What knowledge was learned.
- What skills were developed or improved.
- What attitudes were changed.
Some guidelines to consider:
- Use a Pre and Post Test to evaluate knowledge, skills, and/or attitudes. A written test can be used to measure knowledge and attitudes and a performance test can measure skills.
- The results of Pre and Post tests can help us determine what changes we need to make to the course.
For those that have the resources to go further, Level 3 is where we look at how the training is used on the job; the extent to which a change in behavior has occurred. Here are some guidelines for evaluating behavior:
- Allow time for a change in behavior to take place.
- Evaluate both before and after the training if possible.
- Survey or interview learners, their immediate supervisors, their subordinates and others who often observe their behavior.
Even rarer is Level 4, where we look at how the training affected the business bottom line. This is where we want to see an increase in production, a decrease in time to complete the job, or maybe an increase in quality. Metrics of this sort are difficult, but they can be quantified.
And the added step, Level 5 looks at the return on investment. Here we would look at how much the training cost (development, management, learner time, etc) and how much we got out of the training. This is tied very closely to Level 4 and in my experience it takes plenty of time and resources to measure at this level.
Transfer of Learning
Actually, since the transfer of learning happens throughout the learning cycle (development, implementation, follow-on training), it makes sense to me to evaluate the transfer of knowledge at various places in the cycle, not just after the training has ended. Perhaps we should evaluate before, during and after the training.
- Create the training with objectives built around real-life tasks.
- As much as feasible, make the training hands-on (simulations, exercises).
- · Include performance support tools that learners can take with them and use on the job (checklists, references, spreadsheets, etc.).
- Use case studies and scenarios that have learners select an appropriate response.
- Give learners the chance to practice what they have learned (simulations).
- Make sure you provide feedback and guidance throughout the training.
- · Show learners how they can apply the training to real-life.
- Check with learners after the training to find out what sort of challenges they face when applying the learning.
- Use coaching to help learners overcome these roadblocks.
- Speak with both learners and managers about how applicable the training is in the work environment.
Primarily, senior management’s main concern is the impact of the training – did what we spend on this really contribute to success of the business?
So, is all of this worth it? I certainly think it is. We spend a lot of time and money on content and development; if we don’t measure results, we can’t know if we’re getting our money’s worth. And by measuring results, we can improve our training courses, deliver a higher ROI, and save money in as well.
Remember, E-Learning is not just a training tool: it’s a business performance improvement tool. If we look at it that way, our E-Learning will be cost-effective, powerful, and in line with the business.
We’ve looked at some traditional ways to measure the transfer of knowledge and we have barely scratched the surface. In fact, when it comes to E-Learning, there are other assessment models besides the Kirkpatrick Model that look at things like Enrollment, Completion, and Test Scores. There is much to know in the area of knowledge assessment and here are some resources I recommend:
In the next lesson, we will look at Converting Instructor-Led Courses to E-Learning Format. In that lesson, we will discuss what an E-Learning course might look like and I’ll give you some tips I’ve picked up over the years about how to actually create an E-Learning course that works.