Kirkpatrick's model

1

Assessments and the Kirkpatrick Assessments and the Kirkpatrick ModelModel

Orlando V. Griego, PhDOrlando V. Griego, PhD

2

Then he looked at wisdom and Then he looked at wisdom and appraised it; he confirmed it and appraised it; he confirmed it and

tested it. tested it. Job 28:27Job 28:27

3

Class Objective• Write questions by

the end of class the represent Kirkpatrick’s four levels of training.

4

Activity Objective• Using an activity, be

able to measure my effectiveness from individual to team by increasing my score

5

Behind the 8 ball

6

Donald Kirkpatrick

• Kirkpatrick developed a model of training

evaluation in 1959

• Arguably the most widely used approach

• Simple, Flexible and Complete

• 4-level model

7

The Four Levels

• Level I: Evaluate Reaction

• Level II: Evaluate Learning

• Level III: Evaluate Behavior

• Level IV: Evaluate Results• Fifth level was recently “added” for

return on investment (“ROI”) but this was not in Kirkpatrick’s original model

8

Relationship Between Levels

• Each subsequent level is predicated upon doing evaluation at lower level

• A Level 3 will be of marginal use, if a Level 2 evaluation is not conducted

Level 1 - Reaction

Was the environment suitable for learning?

Level 2 - KnowledgeDid they learn anything

Level 3 - BehaviorKSA being used on the job?

Level 4 - ResultsWas it worth it?

9

Improve Learning Environment

Improve Knowledge/Skill transfer

Check Performance Environment

Check Requirements, Systems and Processes

Only by assessing each level can we yield actionable results

Level 1 - Reaction

Was the environment suitable for learning?

Level 2 - KnowledgeDid they learn anything

Level 3 - BehaviorKSA being used on the job?

Level 4 - ResultsWas it worth it?

10

Types of Assessments Used at Each Level

Level 1 - Reaction

Was the environment suitable for learning?

Level 2 - KnowledgeDid they learn anything

Level 3 - BehaviorKSA being used on the job?

Level 4 - ResultsWas it worth it?

Type Form

SummativeCorrelation of business results with other assessment results

Summative Observation of Performance

360° Survey

Diagnostic

Summative

Self-assessment

Test

Reaction

Formative

Survey

Real-time Polling

Quizzing

11

Reaction - What Is It?

• How favorably participants react to the training (“Customer satisfaction”)– Collects reactions to

instructor, course, and learning environment

– Communicates to trainees that their feedback is valued

– Can provide quantitative information

12

Reaction - What It Looks Like

• Questionnaire - Most common collection tool– Content: I enjoyed the content.– Methods: The seminar approach

helped me learn.– Media: The AVs were helpful to me.– Trainer style: I liked the instructor.– Facilities: The room was useful for

my learning.– Course materials: The materials

provided make my learning better.

SD: 1 2 3 4 5 6 :SA

13

Reaction - How to Perform

• Determine what you want to find out

• Design a form to collect/quantify reactions

• Do Immediately• Develop acceptable

scoring standards• Follow-up as appropriate

14

Learning - What Is It?

• Knowledge

• Skills

• Attitudes

15

Learning - What It Looks Like

• Media used to measure learning:– Text: I felt the textbook help me

learn.– Voice: The audio materials

increased my knowledge.– Demonstration: I learned well from

the demonstration.

• Methods used to measure learning:– Interviews– Surveys– Tests (pre-/post-)– Observations– Combinations

16

Learning - How to Perform

• Use a control group, if feasible

• Evaluate knowledge, skills, and/or attitudes before and after

• Get 100% participation or use statistical sample

• Follow-up as appropriate

17

Behavior - What Is It?

• Transfer of knowledge, skills, and/or attitude to the real world– Measure achievement of

performance objectives

18

Behavior - What It Looks Like

• Observe performer, first-hand

• Survey key people who observe performer

• Use checklists, questionnaires, interviews, or combinations– I believe coming in early is

helpful.– I work better on the new

production system.

19

Behavior - How to Perform

• Evaluate before and after training

• Allow ample time before observing

• Survey key people• Consider cost vs.

benefits– 100% participation or a

sampling– Repeated evaluations

at appropriate intervals– Use of a control group

20

Results - What Is It?

• Assesses “bottom line,” final results

• Definition of “results” dependent upon the goal of the training program

21

Results - What It Looks Like• Depends upon objectives of

training program– Quantify: Bottom line,

productivity, improvement• Proof vs. Evidence

– Proof is concrete– Evidence is soft

• Sample Questions:– I believe the new system has

improved productivity.– My training has allowed me to

be more productive.– My boss is a better leader

after my training.

22

Results - How to Perform

• Use a control group• Allow time for results

to be realized• Measure before and

after the program• Consider cost versus

benefits• Be satisfied with

evidence when proof is not possible

23

Summary & Assignment

• Level I: Evaluate Reaction

• Level II: Evaluate Learning

• Level III: Evaluate Behavior

• Level IV: Evaluate Results

• Write five questions for each level using 4 different training scenarios. – Use “I” questions.

– Insure they apply to the content

– Tricks: Thesaursus, book TOC, ask each other

24

Readings to Consider

• Kirkpatrick, Donald L. (1998). Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers.

• Worthen, Blaine R, James R. Sanders, Jody L. Fitzpatrick (1997). Program Evaluation: Alternative Approaches and Practical Guidelines (Second Edition).Addison, Wesley, Longman, Inc.

• Kirkpatrick, Donald L. (1998). Another Look at Evaluating Training Programs. American Society for Training & Development.

• Sieloff, Debra A. (1999). The Bridge Evaluation Model. International Society for Performance Improvement.