KPI Guide: Learning Analyst
Use this document as a guide for your Learning Analyst(s) who will be consuming the data from MTM and providing guidance to others on how to manage against these areas. The same information is also available as a Word document at the bottom of this article. MTM recommends that your learning leaders update the Word document with specific guidance tailored to your organization's use of MTM and distribute it to your learning analysts.
User Guidelines
The following steps should be completed before this or any of the KPI Guides are shared with respective stakeholders.
Step 1: Review the KPI and Question columns updating as necessary to align to your current MTM evaluations.
Step 2: Review and update the “What Does this Mean” column as appropriate for your organization.
Step 3: Incorporate any relevant goals in the “Our Organization’s Goal” column. Examples include:
- At or better than All Corporate University Benchmark
- Above 6.0
- 80% Top-2 Box
Step 4: Review and revise the “How to Maximize Your Scores” section based on your specific organization’s role expectations.
Step 5: Ensure this KPI guide is provided during Learning Analyst onboarding, available in any centralized resource locations for learning analysts, and embedded in your automated report recipient emails.
| KPI Metric | MTM Post Event Questions | What does this mean? | Our Organization’s Goals |
|---|---|---|---|
| Instructor Performance | The instructor helped me understand how to apply the material. The instructor kept me engaged. The instructor helped me understand how to apply the material.The instructor kept me engaged. | Instructor effectiveness is a critical prerequisite for knowledge transfer and application. Instructors should be actively connecting the training content to the learners' jobs. | |
| Optional: Classroom Environment | The classroom environment was appropriate for learning. | Classroom environments that distract from students' ability to effectively learn the material degrade knowledge transfer, which can ultimately impact the degree of application and impact back on the job. | |
| Technology Effectiveness | The visuals and graphic design of the course were high quality.The content was user-friendly.This delivery method was effective for learning.The technology worked appropriately. | Technology can be a useful tool in accomplishing training objectives, but only when it works properly and has effective design. | |
| Content Quality | The examples / activities within the content helped me to learn.The course met its stated objectives. (Optional)The content was engaging. | Issues with content quality directly impact learners' ability to understand the content and connect it to their jobs. | |
| Knowledge Gain | I gained new knowledge / skills. | If learners are not learning anything new from the training, why are they expected to go? The less that people learn from training, the less they will apply and the less performance / business outcomes will be impacted. | |
| Job Application | I will apply what I learned to my job. | In order for training to positively impact business results, it must be applied on the job. | |
| Scrap Learning | Based on: I will use _% of this content on the job. | This is the opposite of application; it's the % of the training learners don't apply, and is considered to be wasted training time and dollars. | |
| Criticality | The content is critical to my job. | The content is critical to my job. | |
| Relevance | The content was relevant to my job. | If learners do not see the content of the training as being relevant to their job, it can be a predictor of low application. This is generally a prerequisite for criticality, and another indicator of application. | |
| Performance Improvement | My job performance will improve as a result of this experience. The content is applicable to _% of my job. Over time, my job performance in areas related to this content will improve _%. I would attribute _% of my expected improvement to this learning experience. My job performance will improve as a result of this experience.The content is applicable to _% of my job. Over time, my job performance in areas related to this content will improve _%.I would attribute _% of my expected improvement to this learning experience. | Indicates whether the training will improve the learner's job performance, and to what degree.This also helps you to understand, using the Human Capital methodology, the percent of performance improvement due to the training. This can help you to calculate Benefit to Cost Ratio (BCR), as well as help you to articulate the relative impact that training had on a business outcome. | |
| Manager Support | My manager and I discussed my goals for attendingthis learning experience. | Indicates whether the learner has partnered with theirmanager to stay accountable to applying the training. Thereare many options for helping to better engage managers inthe development process, which can be used in partnershipwith the business | |
| Optional: Support Tools | The participant materials (manual, presentation handouts, job aids, etc.) will be useful on the job. | Indicates whether the support tools provided in the training will be useful when trying to apply the training on the job. | |
| Organizational Support | Potential challenges that I expect will prevent me from applying this content include: | Indicates whether the learner perceives that there will be barriers to applying the training, so that L&D and the business can effectively partner to mitigate those barriers and maximize the training's impact back on the job. | |
| Business Outcomes Impacted | I expect this learning experience will positively impact: (check all that apply) | Indicates the business outcomes that are most likely to be impacted as a result of applying the training. Serves as a leading indicator of impact, and as confirmation that learning had or will have an impact on one or more high-level business measures. | |
| Perceived Value | This learning experience was a worthwhile investment of my time. | This is an indicator of application and impact; if this KPI is low, it indicates that learners feel that the training is a waste of time. | |
| Net Promoter | How likely are you to recommend this learning experience? | Net Promoter is a useful summary indicator of value; if a learner is willing to personally recommend the training to a colleague it is highly likely that they found it to be valuable. |
Tips for How To Maximize Scores
In order to promote continuous improvement for your organization’s learning programs it will be important, as you analyze data and share key results and insights, to embed recommended actions individuals can take to improve in key areas. See guidance below on common recommendations for maximizing scores in certain areas.
Instructor Performance
Consider opportunities to revisit the instructor selection and onboarding process. Train instructors to be master facilitators of class discussion and emphasize this as a part of instructor performance appraisal. When selecting instructors ask the candidates to facilitate a snippet of a class as part of the interview process and look for high energy and enthusiasm.
Is your organization using subject matter experts to deliver training? If so, ensure an assessment process has been established to determine if they can deliver training independently or should be paired with an instructor.
Prior to delivery of a new course, instructors should do a mock training session for SMEs or fellow instructors, or at least a small pilot with select learners. Certify instructors for key courses through a train-the-trainer program.
Establish a process for the instructors to learn about the group that they will be training. This will help with references, examples, and instructional preparation. For example, an instructor might meet with subject matter experts prior to training to build their confidence and anticipate questions learners may have.
Classroom Environment & Technology Effectiveness
We recommend starting with the fundamentals. The temperature and seats should be comfortable. There should be minimal distraction. Technology used in class should function. Learners and instructors should be able to easily hear one another. Beyond the fundamentals, aim to make your classrooms inviting with plenty of natural light, collaborative seating arrangements, and fresh design and colors.
Remember, for online delivery to be effective, it needs to be highly interactive, at the appropriate pace for the learner, and provide simulations or exercises that challenge the learner. If your vILT courses do not currently meet these criteria, work on improving the virtual experience before considering moving to a classroom experience.
If your eLearning is primarily text-based, you can dramatically increase the engagement and knowledge transfer speed by replacing text with illustrations, video, and simulations.
Provide an L&D help desk contact and promote that within eLearning or vILT courses. Leverage the bank of inquiries from learners to identify and prioritize the interface changes that are needed.
Encourage the organization to invest in technologies and physical prototypes that are best suited to provide realistic simulations of the on-the-job application of the learning.
If specific feedback is provided about technology platforms be sure to share those learner concerns with your eLearning or vILT platform vendors and ask how and when they may be addressed.
Content Quality
Review the content of low-rated courses against the objectives to ensure it covers each with adequate depth.
Has your organization embedded challenging in-class simulations or exercises that enable learners to role-play or practice the new knowledge / skills they have learned in the training? Do the course exercises provide ample opportunities for practice and feedback?
In order for examples to resonate with learners, it’s important for them to be real-life examples. Review existing exercises and consider getting learner or subject matter expert input to ensure they are realistic. Embed opportunities for learners to provide their own real life scenarios to discuss.
Make sure a standard process has been established for instructors to communicate course objectives and any expectations at the beginning of class and consider allowing learners to exit if they do not feel the course will fulfill their learning needs.
Remind instructors to tune in to participant challenges and objectives so they can deliver targeted examples, appropriately weight topics, and customize their interactions so that participants feel that their needs have been met.
Knowledge Gain
Review this data by key demographic. Is there variance in the results? Consider developing and administering pre-tests to enable potential learners to pass out of courses for which they already know much of the content. A great way to identify potential candidates for pre-tests is to start with courses in your foundational skills portfolio that have the highest volumes of learners.
Review the course listings in your LMS, learning portal or catalog to ensure learning objectives and target audience are clear for every course.
Ensure that participant and instructor materials state how the course objectives align to on the job application and business goals.
Sometimes all you need is for your instructors to ask...what is one new thing that you learned in this course? Why is it important to you? They should also remind learners it is better to make mistakes during training that on the job. Failure and corrective feedback lead to more learning than near-perfect performance in class. Challenge learners so they are likely to fail and require feedback to be successful. Really push learners out of their comfort zones in class.
Job Application, Relevance, and Addressing Scrap Learning
Review your portfolio of courses quarterly against the current priorities of the organization. Be vigilant in retiring courses that were created for a past need that is no longer a priority. To keep your course portfolio tightly aligned to the needs of the business, establish a policy where by you need to offset any new courses that are created by retiring courses that no longer provide high impact and value.
Consider if the data suggests opportunities to trim basic, redundant, or tangential content to keep the learner engaged by providing a challenging pace. More often than not with training, the pace is too slow for learners.
When you review data by key demographics is there variance in the results? If possible, allow learners to set their own pace by providing dynamic self-paced content. For exercises, consider providing beginner and expert options that enable learners to select the most appropriate challenge level.
Develop a single-page resource for critical takeaways.
Ensure your instructors are prepared to communicate how the learning aligns to the roles in the class. To be successful, make this an explicit part of both the instructional design process, as well as instructor readiness or train-the-trainer efforts. Train instructors on understanding roles and objectives of learners and equip them to modify the learning on-the-fly to make it more impactful.
Ensure your instructor-readiness processes focus heavily on connecting the new knowledge and skills with the daily professional lives of the learners.
Instructors can impact this area when they share / receive examples and during the wrap-up. They could ask participants to each share one specific way they will apply the learning to their job, and one thing they are going to share / teach someone when they get back on the job. Asking participants to do a short, fun high energy teach back to open the afternoon / the next day is a great way to get them to internalize what they learned and may impact these scores.
The use of the learning journal throughout the session in a deliberate (and interesting) way can also help participants make the connection.
After an application session or activity, encourage instructors to use a debrief to get participants to start actively thinking about applying what they learned back on the job. By the end of a course, a participant should be able to articulate at least one area of their job performance that will be improved by applying the tools and skills, in specific, measurable terms.
Performance Improvement & Business Outcomes Impacted
Are you seeing low performance improvement scores but no obvious other area exists for improvement? Perhaps training is not the best approach to change certain behaviors. When having the initial discussions with business stakeholders regarding a new training need, make sure L&D focuses on understanding the specific business goals and how employees need to change behaviors in order to accomplish those goals. Determine if training is the best approach to changing those behaviors, and if so, what forms that training should take.
This data can help you think about examples and framing for the course. For example, if you are looking at the results for a leadership course you would expect to have an impact on "Engagement", but you might notice that the biggest impact is actually in another area like "Innovation.” Review the results with an eye to spotting trends here that could help you give examples, share ideas for application and drive the overall utility of the course.
Manager and Organizational Support
Are you currently holding education sessions for managers to articulate the expectations and best practices for them to support the application of learning programs? Often, managers are unaware of the impact that they specifically can have on the application of learning and the employee performance increases that come with it. Instruct managers to have pre-training conversations with learners to set expectations and post-training discussions focused on how to apply what they've learned.
Within the course, have learners complete an action plan and direct them to share and discuss with their managers.
For strategic programs, provide job aid to managers for providing post-training support.
Hold a consultative conversation with business owners and learners' managers regarding the barriers that learners are experiencing to applying what they've learned on the job.
Support Tools
If you’ve included a student guide or workbook it should augment, rather than replicate, content that the instructor delivers. For the lowest-rated courses, if the workbook is currently redundant to the in-class instruction, recommend revising or eliminating the workbook.
Be sure your design and development is incorporating learner needs that go beyond simply learning to do something for the first time. Provide resources that address additional needs such forgetting common details, as well as when they need more depth to address a more complex or challenging scenario
Ensure that participant materials reflect real-life terms and provide guidance for how to implement action steps on the job.
Make sure that all participant materials are easily accessible via your organization’s knowledge sharing site.
Keep in mind that a one-size-fits-all approach will not always be effective when it comes to participant materials. Consider creating role-specific job aids and other sustainments resources that concisely summarize the information needed for each specific role.
Remind instructors to walk through the support materials in the training and share practical ways participants might be able to use them when back on the job.
How to Use The Qualitative Data
Your MTM evaluations also contain a number of qualitative questions such as:
- What about this learning experience was most valuable to you?
- How can we make this learning experience more valuable to you?
- Provide an example of how you plan to apply this content on the job.
Don’t forget to review this data as well! Use this to share with your audience:
- Themes from any courses that have lower performance, either overall or related to certain KPIs. Is there something specific called out that your organization could change about your delivery? Or are specific pieces of content called out as too basic, redundant, or tangential?
- Any best practice themes around what’s working well for a course?. This information can often be leveraged to improve courses with opportunity areas aligned to the feedback.
- Specific examples of how learners will apply the content. Encourage key roles to use this information to update examples as well as to celebrate the impact your organization is having! Additionally, these ‘stories’ can be very effective in executive reporting to supplement data around impact and value.
Downloadable copy of Learning Analyst KPI Guide
The attached document contains the information above and can be edited. MTM recommends that your learning leaders update the Word document with specific guidance tailored to your organization's use of MTM and distribute it to your learning analysts.