How to Measure Learning Effectiveness

Those that operate in the learning space are familiar with The Kirkpatrick Model for measuring training effectiveness. This article will outline a simple method for evaluating online and on-site workshops. 

For those who aren’t familiar with The Kirkpatrick Model, I will provide a brief summary here. If you are familiar with it, you can skip ahead to the subheading of Measuring Learning: A Focus on Step 2 to continue reading.

The Kirkpatrick Model

The Kirkpatrick model has 4 levels defined as follows: [1] 

[1] www.kirkpatrickpartners.com

In a recent conversation with L&D expert, Trish Uhl, Trish conveyed that training instruction should always start with clear alignment to Level 4 and answer the following questions:

• What expected value does this training deliver to the business?

• What business outcomes are expected to be impacted as a result of this training?

For my colleagues in the HR-People Analytics field, you’ll notice a strong similarity between the questions above and the questions we ask before embarking any analytical study.

Level 3 then describes the improved human performance that people are expected to bring back to their jobs as a result of this training; people performance that is then expected to deliver the intended business outcomes. 

Although we always want to show alignment to Levels 4 and 3, time needs to pass before you can measure and evaluate whether training had the desired impact on people performance and business performance. According to Trish, time is frequently 30 days or more.  

How can you find out sooner if your training is on track to deliver the desired results? This is where measuring at The Kirkpatrick Model Levels 2 and 1 can help.  

Consider Level 2.  

With Level 2, you can use the results of standard pre- and post-training knowledge tests and basic statistics to determine if delegates have learned anything new from your training.  

Research presented by the National Research Council in the book, “How People Learn: Brain, Mind, Experience, and School,” shows that acquisition of new knowledge, i.e., learning something new, is a fundamental requirement to the design of quality instruction and ensuring that training “sticks.” So how might you collect data to “measure learning” at Level 2? 

Measuring Learning: A Focus on Step 2 

Many workshops still don’t measure learning. This is quickly being considered negligence since modern tools and data collection methods give us, and our business stakeholders, easy methods to do so.  

For those that have attempted to measure learning, i.e., Level 2, what I’ve seen mostly is a test given at the end of the program to see if students understand the material that was presented. But what if a portion of the students attending the workshop found that they learned nothing new? Measuring the knowledge of these students at the end of the program does not tell the entire story. So how do we modify this? 

Students can be given the same test at the start and finish of the workshop. By pairing up their before and after individual scores, a paired t-test can determine whether the difference in scores (before and after) are significant enough to conclude that the workshop attained its goal of conveying new knowledge to students. Note that this is not the same as looking at the before and after averages for the entire class. By significance in this measurement, I am referencing statistical significance as provided by the output of a paired t-test from a statistical program.

What to Consider the Next Time You Purchase Training

Performance objectives define the people performance improvements that are necessary to deliver the desired results to the business. Training program learning objectives must be linked to these performance objectives.  

• What are the learning objectives of the training program you are considering purchasing?  

• How will you measure the efficacy of new knowledge acquisition inherent to the stated learning objectives?

If you are a professional who makes training purchasing decisions or an individual about to purchase training for your own development, insist that the training vendor provide you with a plan on measuring the effectiveness of learning. The learning measurement plan should be aligned such that it assesses the learning objectives. The process can look something like this.

If your training provider is not conducting a proper measurement of Level 2, it is of value to understand why they aren’t. Over the past 5 years, I have been approached by several companies eager to learn how to measure the effectiveness of the programs they provide to clients. In the end, many feared that the result of the statistical measurement may not show results in their favour and they declined to proceed with establishing measurements.  

This vulnerability exposes L&D departments to risk now that business stakeholders can measure the effectiveness or lack of effectiveness of training programs (and L&D functions) for themselves. Due diligence means that students or the purchasers of training must insist on consistent measurement practice from all vendors. 

While conducting a workshop on beginner-level HR analytics last month, I conducted a knowledge pre-test. Nearing the end of the workshop, I taught the students the concept of using the paired t-test for measuring learning and let them know that was the reason why I had them take a test at the start of the workshop. I then had the students take the test again and a paired t-test was conducted. The entire class saw the results of the Level 2 measurement of the workshop in which they had just spent their day. 

What’s Different About Levels 3 and 4 

While Level 2 can and should be measured by the instructor, measurements for Levels 3 and 4 occur once the students return to their offices. For those that have the requisite technology, learning data paired with operational data allow businesses to see the impact of training on operational performance. Oftentimes in this modern age, that capability is a lot closer than many may think; especially given the tools – and unified data - currently available in common business productivity platforms. Something to keep in mind, perhaps, for future measurement adventures beyond Level 2 as in my conversation with Trish, she spoke of a growing trend of functions outside of L&D imparting performance measurements on the L&D teams. She warned that, “Those that are found deficient in performance are being made redundant.” 

For those that do not have the requisite technology, measuring students at Levels 3 and 4 may prove more challenging since it would require substantial coordination between the instructor and the students’ managers. It’s not impossible but executing such an activity will compete with a manager’s other priorities. For these companies, if the training was aligned to objectives the students need to accomplish when back in the office, corporate and individual goal setting may serve as proxy Level 3 and 4 measurements.

The Nitty-gritty for Math People

Test name: Paired Sample T-Test

What it tests: Whether the mean difference between two sets of observations is zero. In a paired sample t-test, each subject or entity is measured twice, resulting in pairs of observations.

Options: one-tail (upper and lower) or two-tail (Choose one-tail for applying to the learning measurement described in this article. Then look at the direction of the means.)

Requirements:

• The data needs to be numerical and continuous since it is based on the normal distribution.

• Independence of observations (we assume that the students are independent of each other).

• Normality of data (normally distributed or approximately so)

Conclusion

The Kirkpatrick Model is widely known and accepted. Measuring levels 3 and 4 has become much easier in recent years with advances in data access and analytics… and the survival of L&D teams is now linked to their ability to use analytics to measure their performance.  

Challenges still remain for smaller businesses who don’t have the luxury of L&D systems, but these businesses can still insist on a Level 2 measurement when purchasing third-party workshops. Level 2 for these companies does not require a large investment of time. 

Whether your company is large or small, the trends in the L&D arena are clear. The ability to measure training programs has become imperative.

---