Phone: 888‐579‐9814 Email: info@work‐learning.com Website: Work‐Learning.com Audits: LearningAudit.com Smiles: SmileSheets.com Blog: WillAtWorkLearning.com Twitter: @WillWorkLearn Will Thalheimer, PhD President Work‐Learning Research, Inc. Somerville, Massachusetts, US Measuring eLearning to Create Virtuous Cycles of Continuous Improvement DevLearn – September/October 2015 Bridging Gap between Research and Practice
36
Embed
September/October 2015 Measuring eLearning to Create … · Measuring eLearning to Create Virtuous Cycles of Continuous Improvement ... EXERCISES 4 With REALISTIC SCENARIO ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
Very Weak Relationship between Levels
Correlation between levels?
Level 1to
Level 2
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied
Psychology, 93, 280-295.
Correlation between levels?
No
Practical Significance
Weak Relationship is below .30 and .09 is VERY WEAK
So…SMILE SHEETS tell us VERY LITTLE about Learning
Three Biases in the WayWe Measure Level 2 Learning
WHEN did you Measure Learning?
90 %
WHERE did you Measure Learning?
91 %
Aggleton, J. P., & Waskett, L. (1999). The ability of odours to serve as state-dependent cues for real-world memories: Can Viking smells aid the recall of Viking experiences? British Journal of Psychology, 90, 1-7.
Balsam, P. D. (1985). The functions of context in learning and performance. In P. D. Balsam & A. Tomie (Eds.) Context and Learning. Hillsdale, NJ: Erlbaum.Bjork, R. A., & Richardson-Klavehn, A. (1989). On the puzzling relationship between environmental context and human memory. In C. Izawa (Ed.) Current Issues in Cognitive Processes: The Tulane
Floweree Symposium on Cognition (pp. 313-344). Hillsdale, NJ: Erlbaum.Bouton, M. E. (1993). Context, time, and memory retrieval in the interference paradigms of Pavlovian learning. Psychological Bulletin, 114, 80-99.
Bower, G. H., Monteiro, K. P., and Gilligan, S. G. (1978). Emotional mood as context for learning and recall. Journal of Verbal Learning and Verbal Behavior, 17, 573-585.Bransford, J. D., Franks, J. J., Morris, C. D., & Stein, B. S. (1979). Some general constraints on learning and memory research. In L. S. Cermak & F. I. M. Craik (Eds.), Levels of processing in human
memory (pp. 331-354). Hillsdale, NJ: Erlbaum.Cassaday, H. J., Bloomfield, R. E., Hayward, N. (2002). Relaxed conditions can provide memory cues in both undergraduates and primary school children. British Journal of Educational Psychology,
72(4), 531-547.Cousins, R., & Hanley, J. R. (1996). The effect of environmental context on recall and category clustering scores following relational and individual item processing: A test of the outshining
hypothesis. Memory, 4, 79-90.Dalton, P. (1993). The role of stimulus familiarity in context-dependent recognition. Memory & Cognition, 21, 223-234.
Davies, G. (1986). Context effects in episodic memory: A review. Cahiers de Psychologie Cognitive, 6, 157-174.Dijkstra, K., Kaschak, M. P., & Zwaan, R. A. (2007). Body posture facilitates retrieval of autobiographical memories. Cognition, 102, 139-149.
Dulsky, S. G. (1935). The effect of a change of background on recall and relearning. Journal of Experimental Psychology, 18, 725-740.Eich, E. (1985). Context, memory, and integrated item/context imagery. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 764-770.
Eich, E. (1995). Mood as a mediator of place dependent memory. Journal of Experimental Psychology: General, 124(3), 293-308.Eich, J. E. (1980). The cue dependent nature of state dependent retrieval. Memory and Cognition, 8, 157-173.
Fernandez, A., & Glenberg, A. M. (1985). Changing environmental context does not reliably affect memory. Memory & Cognition, 13, 333-345.Gartman, L. M., & Johnson, N. F. (1972). Massed versus distributed repetition of homographs: A test of the differential encoding hypothesis. Journal of Verbal Learning and Verbal Behavior, 11, 801-
808.Godden, D. R., and Baddeley, A. D. (1975). Context dependency in two natural environments: on land and underwater. British Journal of Psychology, 91, 99-104.
Godden, D., & Baddeley, A. (1980). When does context influence recognition memory? British Journal of Psychology, 71, 99-104.Grant, H. M., Bredahl, L. C., Clay, J., Ferrie, J., Groves, J. E., McDorman, T. A., & Dark, V. J. (1998). Context-dependent memory for meaningful material: Information for students. Applied Cognitive
Psychology, 12, 617-623. Herz, R. S. (1997). The effects of cue distinctiveness on odor-based context-dependent memory. Memory & Cognition, 25(3), 375-380.
Jacoby, L. L. (1983). Remembering the data: Analyzing interactive processes in reading. Journal of Verbal Learning and Verbal Behavior, 22, 485-508.Johnson, A. J., Miles, C. (2008). Chewing gum and context-dependent memory: The independent roles of chewing gum and mint flavour. British Journal of Psychology, 99(2), 293-306.
Marian, V., & Kaushanskaya, M. (2007). Language context guides memory content. Psychonomic Bulletin & Review, 14(5), 925-933.Marian, V., & Neisser, E. (2000). Language-dependent recall of autobiographical memories. Journal of Experimental Psychology: General, 129, 361-368.
Mead, K. M. L., & Ball, L. J. (2007). Music tonality and context-dependent recall: The influence of key change and mood mediation. European Journal of Cognitive Psychology, 19(1), 59-79.Pan, S. (1926). The influence of context upon learning and recall. Journal of Experimental Psychology, 9, 468-491.
Parker, A., & Gellatly, A. (1997). Moveable cues: A practical method for reducing context-dependent forgetting. Applied Cognitive Psychology, 11, 163-173.Prestera, G. E., Clariana, R., & Peck, A. (2005) Memory-Context Effects of Screen Color in Multiple-Choice and Fill-in Tests. Journal of Educational Multimedia and Hypermedia, 14(4), 2005, 415-436.
Riccio, D. C., Richardson, R., & Ebner, D. L. (1984). Memory retrieval deficits based upon altered contextual cues: A paradox. Psychological Bulletin, 96, 152-165.Roediger, H. L., III, & Guynn, M. J. (1996). Retrieval processes. In E. L. Bjork & R. A. Bjork (eds.), Memory (pp. 197-236). San Diego, CA: Academic Press.
Russo, R., Ward, G., Geurts, H., & Scheres, A. (1999). When unfamiliarity matters: Changing environmental context between study and test affects recognition memory for unfamiliar stimuli. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 488-499.
Schab, F. R. (1990). Odors and remembrances of things past. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 648-655.Schroers, M., Prigot, J., & Fagen, J. (2007) The effect of a salient odor context on memory retrieval in young infants. Infant Behavior & Development. 30(4), 685-689.
Smith, S. M. (1979). Remembering in and out of context. Journal of Experimental Psychology: Human Learning and Memory, 5, 460-471.Smith, S. M. (1982). Enhancement of recall using multiple environmental contexts during learning. Memory & Cognition, 10, 405-412.Smith, S. M. (1984). A comparison of two techniques for reducing context-dependent forgetting. Memory & Cognition, 12, 477-482.
Smith, S. M. (1985). Background music and context-dependent memory. American Journal of Psychology, 98, 591-603.Smith, S. M. (1988). Environmental context-dependent memory. In G. M. Davies & D. M. Thomson (eds.) Memory in Context: Context in Memory (pp. 13-34), Chichester, UK: Wiley.
Smith, S. M. (1995). Mood is a component of mental context: Comment on Eich (1995). Journal of Experimental Psychology: General, 124(3), 309-310.Smith, S. M., & Rothkopf, E. Z. (1984). Contextual enrichment and distribution of practice in the classroom. Cognition and Instruction, 1, 341-358.
Smith, S. M., & Vela, E. (2001). Environmental context-dependent memory: A review and meta-analysis. Psychonomic Bulletin & Review, 8, 203-220.Smith, S. M., Glenberg, A., and Bjork, R. A. (1978). Environmental context and human memory. Memory and Cognition, 6, 342-353.
Spear, N. E. (1978). The processing of memories: Forgetting and retention. Hillsdale, NJ: Erlbaum.Thompson, L. A., Williams, K. L., L'Esperance, P. R., Cornelius, J. (2001) Context-dependent memory under stressful conditions: The case of skydiving. Human Factors, 43(4), 611-619.
Tulving, E., & Thompson, D. M. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352-373.Vela, E. (1984). Memory as a function of environmental context. Paper presented at the 30th annual meeting of the Southwest Psychological Association, New Orleans, LA.
Weiss, W., & Margolis, G. (1954). The effect of context stimuli on learning and retention. Journal of Experimental Psychology, 48, 318-322.Wright, D. L., & Shea, C. H. (1991). Context dependencies in motor skills. Memory & Cognition, 19, 361-370.
Tested in the learning room or in a different room. Does this matter?
Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.
0
5
10
15
20
25
Tested ina
DifferentRoom
Tested inthe
LearningRoom
MeasurementLevels
Level AReal World
Level C Scenario-Based
Decisions
Level DMemorization
Questions
Level B Simulation
Level EAttendance
Level FAffiliation
Certification
Quasi-Certification
“Level C represents the last level of certification that can be considered to assess an ability to perform on the job. Level D represents the first quantum jump away from fidelity in assessment and should be
used with caution.”
Scenario‐Based Question Example
Alena wants to start a firm that helps farmers grow food organically. She has a degree in sustainable agriculture and has worked for a non‐profit organization for seven years doing similar work. She’s developed a marketing plan, a financial plan, and has found several farmers who would pay her if she went out on her own. What should Alena do first—before she tells her boss that she’s quitting to start her own firm?
A. Create a cash flow statement to determine whether her predicted income will support the business through the first year.
B. Form a group of advisors with experience in both small‐business management and agriculture.
C. Determine whether she has enough seed money and start‐up capital to get started.
D. Analyze her values and goals to ensure that the proposed business will support them.
Smile Sheets Can be Improved!
Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree
54321
4.1
Compared to:
• Previous• Standard• Others
Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:
On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):
“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)
We’d like to trust our learners…
But the research shows that they don’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
Transmogrify
Strongly AgreeAgree
Neither Agree Nor DisagreeDisagree
Strongly Disagree
We Start with Fuzzy
Adjectives
54321
Magically We Turn Adjectives
Into Numbers
We Average
Responses, Losing
More Info
3.9
4.2
3.7
4.0
4.1
4.0
4.2
3.4
We Choose One
Question and Report Results
4.1
My Journey in tryingTo create a better
Smile Sheet
My “New” Smile Sheet
Specific Concepts
How much Value?
ConceptNew?
HowWell
Taught?
Simple Overall Ratings
Helping Learners Calibrate
Best Feedback Comes from Comments
http://is.gd/TrainingResearch2012
http://is.gd/ddResearch
Performance-FocusedSmile Sheet
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNINGEFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or
removed? Are we avoiding numeric averages that discourage a standards-based decision on
success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at least try
to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
Three Key Goals:
1. Are the Questions Well-Designed?
2. Are Learners Making Good Smile Sheet Decisions?
3. Are The Data we’re Getting Clear and Actionable?
Ensuring that we are focused on the science-
of-learning factors that matter!
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNINGEFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or
removed? Are we avoiding numeric averages that discourage a standards-based decision on
success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at least try
to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
What Most Smile Sheets
Provide
QUATERNARYGOALS
Traditional Smile Sheets
Learners rate instructors as credible and engaging?
Learners say that classroom environment was conducive
In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?
A. I’m NOT AT ALL ABLE to put the concepts into practice.
B. I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.
C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.
D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY-COMPETENT LEVEL in using the concepts taught.
E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.
• “Takes raw data as input and specifies the computations required to turn those data into measurements and assessments.”
• Provides a list of constructs, which can represent experiences, tasks, conditions, etc.
• Based as an XML Schema
• Study underway at Simulation Interoperability Standards Organization (SISO) for standard
References:• Stacy, W., Ayers, J., Freeman, J., & Haimson, C. (2006). Representing Human Performance
with Human Performance Measurement Language. Washington, DC: Aptima, Inc. • Walker, A., Tolland, M., & Webb, S. (2015) Using a Human Performance Markup Language
for Simulator‐Based Training. Woburn, MA: Aptima, Inc.
Problems with the Kirkpatrick/Phillips 4- or 5-Level Models of Learning Evaluation
• Pushes us to focus on weighing outcomes. Is largely silent on learning support and learning-design improvement.
• Training centric. Ignores prompting mechanisms & on-the-job learning.
• Ignores the role that management and the business side must play.
• Implies that higher levels are more important than lower levels.
• Ignores the causal chain from learning to remembering to performance to results.
• Ignores the fact that learners forget and that learning interventions can be good at creating understanding but poor at minimizing forgetting.
• Pushes us to value learner ratings as predictive of learning and on-the-job performance.
“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…
[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings.
[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… (p. 91)
http://is.gd/TrainingResearch2012
“The goal of training evaluation is not to prove the value of training; the goal of evaluation is to
improve the value of training.” (p. 94‐95)
Tim Mooney and Rob BrinkerhoffCourageous Training:
Bold Actions for Business Results.
Final Thoughts:
We have a responsibility to build learning programs that are effective.
Valid feedback enables improvement.
We must work to get good feedback.
We should aim to create virtuous cycles of continuous improvement.