Using Data to Improve Training Programs

𝐓𝐡𝐞 𝐒𝐞𝐜𝐫𝐞𝐭 𝐭𝐨 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐓𝐡𝐚𝐭 𝐀𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐖𝐨𝐫𝐤𝐬? 𝐒𝐭𝐚𝐫𝐭 𝐚𝐭 𝐭𝐡𝐞 𝐄𝐧𝐝. 🏁 I used to think my job as an L&D professional started with a syllabus. I was wrong. Recently, I was tasked with building a learning solution for our Talent Acquisition (TA) team. The goal wasn’t just to "train recruiters"—it was to solve a business problem. Instead of looking at what they needed to know (Level 2), I started with what the business needed to achieve (Kirkpatrick Level 4). The "Reverse" Approach I didn’t start with slides. I started by analyzing Voice of the Customer (VOC) survey results, focusing on various metrics from both Hiring Managers and Candidates. Working Backwards: ✅ Level 4 (Results): I defined the business KPI. ✅ Level 3 (Behavior): Based on the VOC metrics, I identified the specific actions recruiters needed to change—specifically around "Precision Intake" and "Candidate Experience Management." ✅ Level 2 & 1 (Learning & Reaction): Only then did I design the actual training content that addressed those specific behavior gaps. The Result? The training didn't feel like a chore; it felt like a solution. Because I built it based on the actual metrics revealed in the VOC surveys, the TA team saw immediate value, and the business saw a measurable shift in hiring efficiency. The Lesson: If you want your learning solutions to be more than just "check-the-box" exercises, stop asking "What should we teach?" and start asking "What does the data say I need to solve?" How do you use VOC data to shape your enablement programs? 👇 #LearningAndDevelopment #InstructionalDesign #TalentAcquisition #KirkpatrickModel #Enablement #DataDrivenLD #BusinessImpact

As Instructional Designers, we often track training completion in spreadsheets. But rows and columns rarely show us the real shape of a learning culture. So I used Gephi to model a sample organizational training network. 🔵 Blue nodes: Training topics 🟣 Purple nodes: Employees Each connection represents actual participation, not just assignment. When the data turned into a network, the story became much clearer: 🔹 Hidden silos appeared immediately. A group of employees clustered only around Health & Safety, completely disconnected from core digital topics like Data Security. They are compliant — but isolated. 🔹 “Super Learners” stood out naturally. Employees like Emp #7 emerged as bridges between technical and soft skills. These are not just learners — they are potential mentors, knowledge carriers, and internal champions. 🔹 Core vs. Edge became visible. While Data Security sits at the heart of the learning culture, Leadership training appears at the fringe, signaling a possible disconnect between strategic development and daily learning behavior. This reminded me of something important: Instructional Design is not only about creating content. It is about revealing gaps, breaking silos, and intentionally designing connections. Spreadsheets show who completed what. Networks show who is truly connected to learning. How do you currently look at your training data: as a list — or as a living system?

Demonstrating the value of learning is easier than you think! In a recent workshop with The Institute for Transfer Effectiveness, I demonstrated how! One workshop participant was designing safety training to help employees use Microsoft 365 strategically to prevent data breaches. She was struggling to capture the value of the program for organizational leaders to understand. I used an alignment framework that incorporates Rob Brinkerhoff’s 6 L&D value propositions and mapped out how to connect her learning program with metrics that matter to organizational leaders. Here’s what that looked like! Aligning learning activities, initiatives or programs to strategic business outcomes is like looking for the through line between disparate things: learning, human performance, departmental key performance indicators, and organizational metrics. This can feel nearly impossible. The glue that holds these seemingly disparate things together are Brinkerhoff’s 6 L&D value propositions. In the safety training example we started by identifying the most relevant value proposition for the program. In this case, it was Regulatory Requirements: a learning program designed to ensure employees are complying with industry specific rules and regulations. Then we connect the L&D value proposition (Regulatory Requirements) with the most relevant outcome for the organization. In this case, it was Net Profit. If employees are complying with industry-specific rules and regulations, this consistent practice will save the organization money in fines, lawsuits, or dealing with the unpleasant consequences of safety challenges (like a data breach). Then we must do the hard work unpacking what people will be doing to support the targeted departmental KPIs. If you’re struggling to figure out the KPIs, you’ll likely find them by asking department leaders what problem they are experiencing on a regular basis that they would like solved. In this case it was too many data breaches and too many outdated files on the server causing misinformation and inconsistent practices. I discovered that what people could be doing differently to support the desired KPIs was adhering to updated protocols on how to manage data and documents within the 365 suite. If people followed the protocols with 100% fidelity, departments would experience a reduction in data breaches. Now … we have the behaviors to target in our training program and the data to use to show the value of learning: Learning metrics: Training attendance and completion rates. Capability metrics: Percentage of fidelity to data and document protocols before and after training. KPI metrics: # of documents on the server that are outdated (being at 20% of lower), # of data breaches per department being at 1 or less annually. Organizational metric: Net Profit How will you use the 6 L&D value propositions and alignment framework to tell your learning value story? #learninganddevelopment #trainingstrategy #datastrategy

Catalina S. told me something that completely reframes how we should think about skills validation. After 10+ years leading workforce transformation at Vodafone, T-Mobile, and DataCamp, she dropped this truth bomb during our latest Business AI Playbook episode: "Companies don't just want employees to know things, they want employees who can do things." Most L&D teams are still stuck measuring completion rates and quiz scores. But Catalina's seeing something different work: Evidence-based skill validation that proves real-world capability. Here's what she's implementing right now: → AI-powered surgical feedback — Johns Hopkins is using AI to analyze actual surgical videos, providing objective feedback on technique and precision, not just theoretical knowledge → Peer-led GenAI Scouts — A global engineering org turned employees into instructional designers, achieving 90% engagement and 20-40% time savings on repetitive tasks in just 6 months → Real-world retail simulations — AI roleplay environments where new hires practice customer interactions, earning badges only after demonstrating 3 successful and 3 unsuccessful scenarios with lessons learned → Skills data as strategic inventory — Finally giving companies visibility into their actual internal capabilities while supporting employee growth aspirations Catalina's challenge to every L&D leader: "We need to shift from knowledge retention to evidence-based skill validation." The companies getting this right aren't just improving training metrics. They're fundamentally changing how their workforce approaches capability development. 🎥 Watch the full conversation below 🔄 Share this if you think proving skills matters more than passing tests What's the most creative approach you've seen to validate real-world skills? #BusinessAIPlaybook #LearningInnovation #SkillsValidation #AITransformation #FutureOfWork

🚀 Exciting News! I've submitted the final edits for my chapter on "Leveraging Artificial Intelligence" in the upcoming "ATD's Handbook for Measuring and Evaluating Training (Second Edition)," edited by the esteemed Patti P. Phillips, Ph.D. In this chapter, I explore how generative AI can help solve the #1 training measurement and evaluation challenge - measuring impact. Here are three key takeaways: 1. 📊 Enhance Data Analysis for Actionable Insights Generative AI streamlines data analysis, uncovering deeper insights into training effectiveness in a fraction of the time. By offloading the heavy lifting, talent development professionals can focus on interpreting findings and translating them into meaningful recommendations to improve training programs that drive desired outcomes. 2. 🎯 Harness AI-Driven Quantitative Methods to Complement Self-Reported Feedback In today's data-driven world, it's essential to leverage both self-reported feedback and objective data to gain a comprehensive understanding of training effectiveness. By using generative AI, learning & talent development professionals can objectively determine if their training initiatives are on track to deliver the desired impact or if adjustments are needed. This data-enabled approach equips them to make informed decisions, optimize programs, and maximize the impact of their training efforts. 3. 🤖 Embrace Generative AI as an Interactive Tutor for All Things Measurement and Evaluation How might you leverage an AI-powered tutor to guide you through the entire spectrum of training measurement and evaluation, from foundational concepts to cutting-edge techniques? How might your interactive tutor provide real-time feedback, adapt to your individual learning needs, and empower you to build a rock-solid foundation in data literacy and effectively apply these skills to your training programs? The potential of generative AI in training measurement and evaluation is undeniable, but as learning & talent development professionals embrace these capabilities, it's crucial to use AI responsibly, ethically, inclusively, and transparently while preserving the critical thinking and empathy that only humans can provide. 🌟 Are organizations, business leaders, line managers, and workers getting true value from their training investments? Without measuring the impact, how can they know? It's time for learning & talent development professionals to step up and hold themselves accountable for delivering measurable outcomes. Will you rise to the challenge and leverage the power of generative AI to truly measure and evaluate the impact of your training programs? 📖 Stay tuned for the release of "ATD's Handbook for Measuring and Evaluating Training (Second Edition)" and discover how generative AI can empower learning & talent development professionals to take ownership of their training results, make data-informed decisions, and drive true measurable impact! #LearningAI #MeasurementandEvaluation

📊 L&D Isn’t Just “Looking at Training Data” — We’re Analysts Who Drive Business Decisions I’ve said before that L&D is far more than instructional design — and one of the most overlooked capabilities we bring is analysis. But here’s the trap I see many learning teams fall into: They try to build their own analytics systems… completely separate from where the business pulls its data. And when that happens? You get beautiful dashboards ❌ with zero credibility ❌ that don’t influence decisions ❌ that don’t match the business view of reality. Because here’s the truth: If L&D wants to be strategic, our data needs to come from the same place the business gets its data. That means looking beyond learning metrics and into the metrics the business actually cares about: 📈 Sales performance 📉 Attrition and retention 🎯 Behavior change in the field ⚙️ Operational efficiency 🤝 Customer experience & NPS 📚 Capability trends & talent pipeline 📞 Contact center performance (callbacks, escalations, first-call resolution) 🧭 Adoption of new tools, tech, and processes Because learning doesn’t exist in a vacuum. If you want to prove impact, you must tie learning to outcomes the business is already tracking — not create a parallel universe of data that only L&D looks at. When L&D pulls from the business data stream, something powerful happens: ✅ We speak the same language as executives ✅ We can show where capability is slipping ✅ We can predict workforce risks before they hit ✅ We can measure the real ROI of learning — not just completions ✅ We become a partner in decision-making, not a cost center This is how L&D stops “reporting activity” and starts driving strategy. Executives: 👉 When your L&D team brings you insights, are they tied to the business — or living in a separate learning dashboard that never influences decisions? If you want a strategic learning function, make sure the data they’re using is the same data you're using to run the business.

Training without measurement is like running blind—you might be moving, but are you heading in the right direction? Our Learning and Development (L&D)/ Training programs must be backed by data to drive business impact. Tracking key performance indicators ensures that training is not just happening but actually making a difference. What questions can we ask to ensure that we are getting the measurements we need to demonstrate a course's value? ✅ Alignment Always ✅ How is this course aligned with the business? How SHOULD it impact the business outcomes? (i.e., more sales, reduced risk, speed, or efficiency) Do we have access to performance metrics that show this information? ✅ Getting to Good ✅ What is the goal we are trying to achieve? Are we creating more empathetic managers? Creating better communicators? Reducing the time to competency of our front line? ✅ Needed Knowledge ✅ Do we know what they know right now? Should we conduct a pre and post-assessment of knowledge, skills, or abilities? ✅ Data Discovery ✅ Where is the performance data stored? Who has access to it? Can automated reports be sent to the team monthly to determine the impact of the training? We all know the standard metrics - participation, completion, satisfaction - but let's go beyond the basics. Measuring learning isn’t about checking a box—it’s about ensuring training works. What questions do you ask - to get the data you need - to prove your work has an awesome impact?? Let’s discuss! 👇 #LearningMetrics #TrainingEffectiveness #TalentDevelopment #ContinuousLearning #WorkplaceAnalytics #LeadershipDevelopment #BusinessGrowth #LeadershipTraining #TalentDevelopment #LearningAndDevelopment #TalentManagement #Training #OrganizationalDevelopment

Don't ask your trainees to rank how confident they feel: — "After the training, I feel confident to perform my job." 1) Strongly Disagree 2) Disagree 3) Neither Agree or Disagree 4) Agree 5) Strongly Agree — You'll end up with an average of 3.9 (or something like that). But what are you supposed to do with a 3.9? What decisions should you make? What specific actions should be taken? It’s impossible to know. Instead: Ask questions that reveal insights related to the effectiveness of the training. — “How confident are you when applying this training to real work situations? (Select all that apply)” A) I AM CONFIDENT I can successfully perform because I PERFORMED REAL WORK during the training and received HANDS ON COACHING B) I AM CONFIDENT because the training challenged me WITH AMPLE PRACTICE on WORK-RELATED TASKS C) I’M NOT FULLY CONFIDENT because the training DID NOT PROVIDE ENOUGH practice on WORK-RELATED TASKS D) I AM NOT CONFIDENT because the training DID NOT challenge me with practice on WORK-RELATED TASKS E) I HAVE ZERO CONFIDENCE that I can successfully perform because the training DID NOT REVIEW WORK-RELATED TASKS — One look at survey results that gauge the effectiveness of training will leave you with immediate decisions and actions to make. #salesenablement #salestraining PS - “confidence to apply” is only one important factor to assess. Read Will Thalheimer’s “Performance-Focused Learner Surveys” for the other pillars of training effectiveness.

We spent $200K on training last year. AI replaced 80% of it for $20K. And our employees learned more. Not because AI is magic. Because we finally stopped treating training like a checkbox. Here's 9 ways we use AI to train employees (that actually work): 1/ Personalized Learning Paths That Adapt → AI analyzes skill gaps in real-time → Creates custom curricula for each employee 💡 Reality: Our junior marketer mastered analytics 3x faster with AI-tailored lessons. 2/ Role-Play Scenarios Without the Awkwardness → AI simulates difficult conversations → Practice firing someone, negotiating, giving feedback 💡 Reality: New managers improved conflict resolution skills 67% using AI role-play vs traditional workshops. 3/ Just-In-Time Micro-Learning → AI serves bite-sized lessons when needed → Learning happens in the flow of work 💡 Reality: Retention rates jumped from 20% to 74% when we switched to AI micro-learning. 4/ Real-Time Performance Coaching → AI analyzes actual work output → Provides immediate, specific feedback 💡 Reality: Our sales team's close rate improved 31% with AI analyzing their calls and suggesting improvements. 5/ Peer Learning Networks at Scale → AI matches employees with complementary skills → Facilitates knowledge sharing across departments 💡 Reality: Cross-department collaboration increased 5x when AI started suggesting learning partners. 6/ Language and Communication Training → AI analyzes emails, presentations, reports → Suggests improvements for clarity and impact 💡 Reality: Customer sat scores rose 22% after AI helped our support team improve their written communication. 7/ Simulation-Based Technical Training → AI creates safe environments to practice → Mistakes become learning, not disasters 💡 Reality: Developers ship production-ready code 40% faster after AI simulation training. 8/ Continuous Skill Assessment → AI tracks skill development over time → Identifies when someone's ready for new challenges 💡 Reality: Internal promotions increased 60% when we could actually see skill progression data. 9/ Cultural and Soft Skills Development → AI analyzes team interactions → Identifies gaps in emotional intelligence 💡 Reality: Team engagement scores improved 43% after AI-guided soft skills development. Here's our AI training framework: Start Small: ✓ Pick one department ✓ Choose one skill gap ✓ Run 30-day pilot ✓ Measure actual behavior change Scale Smart: ✓ Use pilot data to refine approach ✓ Expand to adjacent teams ✓ Let success stories drive adoption ✓ Keep human connection central But here's what AI can't do: Inspire. Motivate. Empathize. Build culture. The magic happens when we use AI to handle the what and when of training. So humans can focus on the why and how it matters. How are you using AI to develop your team? Share below 👇 ♻️ Repost if your network needs this training revolution. DM me if you want to discuss how to develop your own AI training plan.

Before any internal training, I run a survey to identify common issues and refine the agenda based on real needs. It allows me to summarize and interpret, and this is what I just got for a planned workshop: "Bottom Line: The group is not yet Monte Carlo-ready across the board, but they are close enough that the workshop can introduce the technique and address the blockers to using it well. The most powerful thing the workshop can do is help participants see what needs to be true before the simulation is trustworthy — because fixing those conditions will improve delivery even before a single simulation is run." I'm NOT sure I could run training now without this upfront guidance, as it keeps me grounded on what might stick. I can go deep on the math. But my workshop survey helps me NOT teach things that won't stick. It's essentially free. Again, I built it because the current survey tools NEVER gave this type of guidance. Please give it a try and help me improve it: https://askpilot.io And for those interested, here is what it recommends in agenda: Recommended Workshop Sequencing (based on readiness gaps) Rather than jumping straight to Monte Carlo, the survey data suggests a natural sequencing: Step 1 — Establish Foundations Fix backlog hygiene, estimation consistency, and the definition of done BEFORE running simulations. Step 2 — Make the Invisible Visible Tag unplanned work. Map dependencies upfront. Start tracking blocker duration and external lead times. Step 3 — Stabilize the Input Address work readiness at intake. Stabilize priorities. Aim for a "clean" 8–12 week throughput baseline. Step 4 — Run Monte Carlo with Caveats Start simple. Use throughput-based simulation. Be explicit about what the model assumes and where the data is still noisy. Step 5 — Refine and Extend Add item type segmentation. Model external dependency lead times. Improve flow efficiency. Tighten the forecast.