Phillips Associates 1 Presented by: Ken Phillips Phillips Associates September 11, 2012 DEMONSTRATE VALUE AND GET YOUR “SEAT AT THE TABLE” 51 TIPS & BEST KEPT SECRETS FOR DESIGNING LEVEL 1 - 4 EVALUATIONS
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Phillips Associates 1
Presented by: Ken Phillips Phillips Associates September 11, 2012
DEMONSTRATE VALUE AND GET
YOUR “SEAT AT THE TABLE”
51 TIPS & BEST KEPT SECRETS FOR
DESIGNING LEVEL 1 - 4 EVALUATIONS
Phillips Associates 2
AGENDA
➤ Measurement and evaluation essentials ➤ Create Level 1 evaluations that obtain
powerful data ➤ Crack the code of test question design
and take your Level 2 evaluations to 2.5 ➤ Capture elusive Level 3 data: the secrets
of survey design ➤ Level 4 evaluations ➤ Making sense out of evaluation data
Phillips Associates 3
M & E OVERARCHING GOALS
Decide whether to continue or discontinue a learning program
Improve a learning program
Demonstrate value of a learning program to executives/stakeholders
Improve management of learning function – run learning like a business
Phillips Associates 4
Aggregating data collected about a learning
program, interpreting and drawing conclusions
from the results and making recommendations
based on the conclusions
MEASUREMENT vs. EVALUATION
Collecting data about the outcome of a learning program
Measurement =
Evaluation =
Phillips Associates 5
Levels of Evaluation Measurement Focus Time Frame
Level 1: Reaction Participant reaction to a learning program
Conclusion of learning program
Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes
Conclusion of learning program or within 6 to 8 weeks after
Level 3: Behavior Degree to which participants applied back-on-the-job what was learned
2 to 12 months
Level 4: Results Degree to which targeted business outcomes were achieved
9 to 18 months
Level 5: ROI Degree to which monetary program benefits exceed program costs
9 to 18 months
KIRKPATRICK/PHILLIPS EVALUATION MODEL
Phillips Associates 6
LEVEL OF EVALUATION BEING MEASURED?
➤ Situation 1: Participants will be able to
distinguish good examples of
paraphrases from poor examples.
Level 2: Knowledge
Phillips Associates 7
LEVEL OF EVALUATION BEING MEASURED?
➤ Situation 2: Participants provide
feedback on the relevancy of a learning
program to them and their job.
Level 1: Reaction
Phillips Associates 8
LEVEL OF EVALUATION BEING MEASURED?
➤ Situation 3: Six months after a learning
program employee engagement scores
increase by 10% over the previous year.
Level 4: Results
Phillips Associates 9
LEVEL OF EVALUATION BEING MEASURED?
➤ Situation 4: The estimated benefit of the
increase in employee engagement is
$250K and the program costs were
$150K.
Level 5: ROI
Phillips Associates 10
LEVEL OF EVALUATION BEING MEASURED?
➤ Situation 5: One week following a
learning program customer service rep
follow-up phone calls increase by an
average of 6 per rep.
Level 3: Behavior
Phillips Associates 11
“CHAIN OF EVIDENCE” DETERMINENTS
Strategic importance of learning program
Cost of learning program
Number of participants attending program
Stakeholder/business executive expectations
Phillips Associates 12
11 SURPRISING TECHNIQUES FOR
OBTAINING POWERFUL DATA FROM
LEVEL 1 EVALUATIONS
Phillips Associates 13
LEVEL 1 FACTS*
92% of organizations evaluate at least some learning programs at Level 1
Organizations that use Level 1s on average evaluate 78% of all learning programs
36% of organizations view data collected as having high or very high value
*2009 ASTD Research Study: The Value of Evaluation: Making Training Evaluations More Effective
Phillips Associates 14
WHY THE DISCONNECT?
➤ Level 1 Evaluations focus on learning department not business operation data
➤ Examples • Facility • Course design • Facilitator • Food
➤ The data collected is not used
Phillips Associates 15
WHAT’S THE SOLUTION?
➤Create Level 1 Evaluations that produce data with high perceived value for both you & your business stakeholders
➤Aggregate data and conduct trend lines or comparisons at group, project, department, regional level
Phillips Associates 16
*Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006.
DATA COLLECTION METHODS*
We hope you have found this excerpt interesting. To get access to the full version, please consider becoming a member of the Northeast Wisconsin Chapter of ASTD.
Go to
http://www.new.astd.org/Default.aspx?pageId=955638
to join, today!
Related Documents
