Evaluation Procedures Dr. Steve Training & Development INP6325.

Evaluation Procedures

Dr. SteveTraining & Development

INP6325

Perspectives on Training

1. Trainee – What if trainee does well in training, but gets fired or laid off for being ineffective on the job?

2. Trainer – What if trainee develops what s/he thinks is effective training, but supervisors won’t let workers use newly learned techniques/skills?

3. Organization – What if organization spent big money on training program, but employees are still not effective?

Applied Questions

Training specialists interested in:1. Whether criteria (performance, $ savings)

indicate improvement following training2. Whether improvement was a result of training3. Whether training will be equally effective with

different group of trainees4. Whether training will be beneficial to other

organizations.

Research Questions

1. Which of two or more training programs is the best and why?

2. Which type of training works best with which type of worker?

3. What type of organization can benefit most by this type of training?

Types of Training Researcher (Randall)

Negativists – Evaluating training is either impossible or unnecessary Often decisions made based on anecdotal evidence

Positivists – Only rigorous scientific evaluation is worthwhile If there’s no experimental data, it’s a waste of time

Often don’t have resources for full-blown experiment

Frustrates (Activists) – All training should be evaluated, but must recognize that evaluation quality varies depending on constraints.

Training Evaluations

Training Evaluation only as good as the: Question that is asked Appropriateness of the criteria Evaluation design

Training Evaluations: Ethical Issues

Ethical issues of empirical study Use of control group means some may not get

training Eliminates promotion opportunities

Control group may feel slighted Trainees act differently if know in experiment

Hawthorne Effect

Formative vs. Summative Evaluation

Formative Evaluation – evaluate whether training program is operating as planned BEFORE it is implemented Focus on process criteria – how training should be

implemented Summative Evaluation – Does training

produce the expected outcome? Trained vs untrained group comparisons’ Comparative summative evaluation – which of

2 or more training approaches produces greatest benefit

Threats to Validity of Training

Internal Validity – did training make a difference in this situation? Training Validity – Does trainee’s training

performance meet criteria for training program? Transfer Validity – Does trainee’s job

performance meet criteria? External Validity – Can training results be

generalized to other trainees or settings? Intraorganizational Validity – Will training be

effective on new group of trainees? Interorganizational Validity – Can training

program be used successfully at other organizations?

Threats to Internal Validity

1. History – Events that happen between pre- & post- tests have nothing to do with training

Contamination of results (ex: layoffs, accidents, holidays, etc.)

2. Maturation – Biological or psychological effects of time on training

Effects of fatigue or disinterest

3. Testing – pre-test sensitizes trainee to upcoming post-test

Trainee prepares for it, or learn from pretest Ex: GRE score GRE course GRE score improves

4. Instrumentation – questionable reliability of test measure

If fluctuation in scores is normal, can’t attribute change to training

Threats to Internal Validity

5. Statistical Regression – Extreme (hi or lo) pre-test scores will often regress to the mean on post-test

6. Differential Selection of Participants – if allow participants to volunteer, experimental group may be more highly motivated

Random selection, random assign, or matched groups

Threats to Internal Validity

Pretest Posttest

• Experimental• Control

7. Experimental Mortality – differential loss of participants between groups

Trainees who did poorly on pre-test may get discouraged, fired, quit, etc.

Training may coincide with other project deadlines

8. Interactions – differential effects that one threat might have on another

Ex: testing effects different in each group due to selection

Threats to Internal Validity

9. Diffusion or Limitation of Treatments – members of one group share information with the other

10. Compensatory Equalization of Treatments – to avoid perceived differential treatment, some alternative might be given to control group

No longer true control, but 2nd experimental group

Threats to Internal Validity

11. John Henry Effect – competition between experimental and control group increases motivation of control group to work harder

12. Demoralization of Control Groups – passive-aggressive response to not being selected for training

Threats to Internal Validity

1. Reactive Effect of Pre-testing (sensitization)- effects of pre-test lead to increase sensitivity of instruction

Increased attention to material that was seen in pre-test

2. Interaction of Selection & Experimental Treatment (representative sample) – characteristics of the group chosen for evaluation may be different than in future groups

Threats to External Validity

3. Reactive Effects of Experimental Settings – Experimental group knows they’re being observed causing them to react differently from future groups

Hawthorne, Guinea pig, or Pygmalion Effects

4. Multiple-Treatment Interference (carryover)– problem with within subjects experiments

participant is exposed to more than one treatment - previous treatments will affect later ones

Threats to External Validity

Training Evaluation Designs

Pre-Experimental Designs One Shot Case Study:

No control group

X T2

One Group Pre-Test/Post-Test: Before and after comparison, no control group

T1 X T2

Static Group Comparison: X T2

-- T2

True Experimental Designs Pre-test/Post-test Control Group:

Controls for most internal validity threats (except diff treatment)

R T1 X T2

R T1 -- T2

Solomon 4-Group Design: Controls for both internal and external threats

R T1 X T2

R T1 --- T2

R -- X T2

R -- --- T2

Post-Test Only Control: Like static group, but with randomization

R X T2

R T2

Training Evaluation Designs

Quasi-Experimental Designs Time Series

Longitudinal method rules out maturation, testing, stat regression

T1 T2 T3 T4 X T5 T6 T7 T8

Non-Equivalent Control Group Same as pre-test/post-test control, but not random

selectionT1 X T2

T1 --- T2

Training Evaluation Designs

Other Evaluation Methods

ANCOVA – use pre-test as covariate in comparing post-test means Example covary measure of ability prior to training Useful when small r between pre & post testR T1 X T2

R T1 --- T2 Correlations – correlate training performance to on-

the-job performance Doesn’t necessarily mean training had effect

Content Validity – SMEs opinion that KSAs covered in training were those identified in needs assessment