A good evaluation will cover all four dimensions of learning. This post is an overview of Donald Kirkpatrick’s four levels of evaluation, which correspond loosely to Kolb’s four learning stages.
Kirkpatrick’s evaluation model measures four elements:
Each level is based on the level before it, so in order to achieve results; participants must have a positive experience with the first three levels.
Level One: Reactions
The most basic level of evaluation is the participants’ reactions to the training.
• Did they like or dislike the training and the trainer?
• How did they feel about the training environment?
• Did they think the training was useful?
• Did they feel comfortable?
• Did they feel they had ample opportunities to participate?
This level can be measured with a few simple tools:
• Verbal feedback during and immediately after the workshop
• Subjective questionnaires during and immediately after the workshop, such as happy sheets (where participants circle a happy face or sad face for each question)
Reaction feedback is fairly easy to gather and measure. It should be gathered as close as possible to the desired time period. (For example, if you wanted to measure reactions to the first day of a workshop, you should gather reactionary feedback at the end of the first day.)
Level Two: Learning
The next level of evaluation assesses how much the participant learned. It looks at two basic areas:
• Did trainees learn what we wanted them to learn?
• Was the training experience what we wanted it to be?
This level is typically measured via tests immediately before and immediately after the training. It is important that these assessments are tied closely to the learning objectives.
Note that this level can be measured on an individual or group level. For example, you could have a verbal group-style quiz, or you could have individual assessments. When assessing group performance, however, make sure that each individual can be evaluated.
Level Three: Behavior
This level evaluates how much trainees applied the learning and changed their behavior after the training. Key questions should include:
• How quickly did trainees put their knowledge into effect back on the job?
• Were skills used correctly and relevantly?
• Was the behavior change sustained?
• Is the trainee aware of how they have changed?
• Would the trainee be able to share their knowledge with another person?
It can be challenging to evaluate changes at this level, particularly with soft topics like communication and leadership. It is important to develop a well-rounded, accurate evaluation system before training begins. Trainees will need to be evaluated on an ongoing basis in a way that is not intrusive on their daily duties. Tools like case studies, 360 degree feedback, and self-assessments can be useful as long as they are well-designed, consistent, objective, and appropriately timed.
Level Four: Results
The final level of evaluation is quantifiable results. This assesses the effect of the training on the person’s environment (their workplace, home, etc.). These are typically measurements that are in place via normal business systems, such as:
• Number of sales
• Percentage of customer complaints
• Quality ratings and failures
• Third-party inspection ratings (such as food and safety)
This is an important level of evaluation as it is often what high-level executives look for when evaluating the training. They want to know numbers and figures, with proof to back the data up.
These evaluation processes should tie in with day to day business procedures and not cause a lot of extra work. It is important, however, that the trainee knows what measurements are tied to the training before the training begins. This will help them apply context to the training and achieve better results.
As a final note, be careful of outside factors that can cloud ratings. For example, let’s say that you send your salespeople on training and you expect their sales to increase by 5% per month as a result. If the economy crashes two months after the training, your results will be clouded by outside circumstances.
Types of Measurement Tools
Previously, we talked about how evaluation should take place at four different levels. In order to effectively evaluate each level, you will need a variety of tools.
Individual goal setting is an excellent way to measure behavior and results. Trainees should set goals during the workshop and then evaluate their progress at predetermined intervals afterwards.
In order for goals to be effective, make sure they follow the SMART acronym:
• Specific: Success coach Jack Canfield states in his book The Success Principles that, “Vague goals produce vague results.” In order for you to achieve a goal, you must be very clear about what exactly you want. Often creating a list of benefits that the accomplishment of your goal will bring to your life, it will give your mind a compelling reason to pursue that goal.
• Measurable: It’s crucial for goal achievement that you are able to track your progress towards your goal. That’s why all goals need some form of objective measuring system so you can stay on track and become motivated when you enjoy the sweet taste of quantifiable progress.
• Achievable: Setting big goals is great, but setting unrealistic goals will just demotivate you. A good goal is one that challenges, but is not so unrealistic that you have virtually no chance of accomplishing it.
• Relevant: Before you even set goals, it’s a good idea to sit down and define your core values and your life purpose because it’s these tools which ultimately decide how and what goals you choose for your life. Goals, in and of themselves, do not provide any happiness. Goals that are in harmony with your life purpose do have the power to make you happy.
• Timed: Without setting deadlines for your goals, you have no real compelling reason or motivation to start working on them. By setting a deadline, your subconscious mind begins to work on that goal, night and day, to bring you closer to achievement.
Self-evaluations are effective at the first three levels of evaluation, and can be effective at the fourth level depending on the topic. Common types of self-evaluations include:
• Pre-workshop and post-workshop tests to assess learning
• Reactionary questionnaires
• Personal assessment quizzes
• Self-reporting metric systems
When measuring reactionary feedback, open-ended questions such as, “How did you feel about the training?” are fine. However, you should also include scale-based questions so that you can evaluate the group as a whole and evaluate the individual on an objective basis. When measuring learning, behavior, and results, questions that are objective and closed or scale-based are necessary for accurate assessment.
Peer reviews are an excellent tool for measuring behavioral changes. However, you must ensure that the assessment system is well designed to prevent bias.
One excellent tool is 360 degree feedback. This system is designed to gather feedback from all of the people around an employee – their co-workers, subordinates, superiors, clients, etc. There are many resources available that can help you design a good 360 degree feedback system. If the topic that you are training on has high value, it can be worthwhile to take the time to develop a peer review system to accurately measure behavioral changes.
Supervisor evaluations are an important part of evaluating behavior changes and assessing results. Like peer reviews, a behavioral evaluation system should be set up before the training. It should be ratings-based and include closed questions to help the supervisor stay objective. When asking supervisors to measure results, those results should tie in with the employee’s regular metrics whenever possible. This achieves two things: it ensures that the measurements are relevant to the employee’s day-to-day duties, and it minimizes the amount of extra work that the supervisor has to do. (Often, if measuring training causes more work for supervisors, they will often avoid completing the evaluation, or spend minimal time doing so.)
Two notes of caution about supervisor evaluations:
• The employee must know which metrics will be evaluated after the training.
• Like peer evaluation, supervisor evaluation can be biased. Develop your metrics accordingly.
Depending on the scenario, you may want to ask high-level executives in the organization to complete an evaluation. This will typically reflect behavioral changes and or measurable results. They will be particularly effective at helping you determine if your training was effective for the entire group. Make sure that these types of evaluations are necessary, focused, and short.
As well, although company executives are typically not involved in the nuts and bolts of training, they may want to see a high level evaluation report, particularly if the training was expensive, required by law, or was expected to have a high impact.
When you are planning the training, make sure to gather expectations from these key stakeholders, including timelines for results and the level of detail desired. Then, use this framework to build a results report tailored to their needs. The report will typically reflect behavioral changes and or measurable results.
Source: Measuring Results from Training workshop
River Street Consultant