Training Evaluations

Training EvaluationsTraining effectiveness Training outcomes

measures that the trainer & company use to evaluate training programs

Training evaluation is the process of collecting outcomes needed to determine if training is effective

Evaluation design to whom, what, when, and how information needed for determining the effectiveness of the training program

Reasons For Evaluating Training

Training grew from 1.5 percent of payroll in 1996 to 2 percents in 1998 – a 40 percent increase in 2 yearsTraining investment is increasing because learning creates knowledge which differentiates between those companies and employees

Training EvaluationFormat evaluation refers to evaluation conducted to improve the training process

Helps to ensure1. The training program is well organized and runs

smoothly 2. Trainees learn and are satisfied with the program

Pilot testing is the process of previewing the training program with potential trainees & managers

Summative evaluation refers to evaluation conducted to determine the extent to which trainees have changes as a result of the training program

Why a Training Program Should Be Evaluated

1. To identify the program’s strengths and weaknesses

2. To assess the content, organization, and administration

3. To identify which trainees benefited

4. To determine the financial benefits and cost of the programs

5. To compare the costs and benefits of training versus non-training

6. To compare the costs and benefits of different training programs to choose the best one

Evaluation ProcessConduct a Needs Analysis

Develop Measurable Learning Outcomes

Develop Outcome Measures

Choose an Evaluation Strategy

Plan and Execute the Evaluation

Outcomes Used In Evaluating Training

ProgramsOutcome Example How Measured What is Measured

Cognitive *Safety rules *Pen & paper tests *Acquisition of *Electrical principles *Work sample knowledge *Steps in appraisal interview

Skill-based *Use a jigsaw *Observation *Behavior *Listening skills *Work sample *Skills *Coaching skills *Ratings *Land an airplane

Affective *Satisfaction with *Interviews *Motivation training *Focus groups *Reaction to *Beliefs regarding *Attitude surveys program other cultures *Attitudes

Outcomes Used In Evaluating Training

Programs Cont. Outcome Example How Measured What is

Measured

Results *Absenteeism *Observation *Company *Accidents *Track data from payoff *Patterns information system or performance record

Return on *Dollars *Identify and compare *EconomicInvestment costs and benefits of value of the program training

Evaluating Training Programs

Instructor evaluation measures a trainer’s or instructor’s successResults are used to determine the training program’s payoff for the company

How Do You Know If Your Outcomes Are Good?

RelevanceCriteria relevance Criterion contamination Criterion deficiency

How Do You Know If Your Outcomes Are Good?

ReliabilityDiscrimination Practicality

Evaluation DesignsThreats to validity: Alternative explanations for evaluation results

Threats to validityThe believability of the study resultsThe extent to which the evaluation results are generalizable to other groups of trainees

Internal validity believability of the study results

Characteristics of the company The outcome measuresPersons evaluation studyInitial group differences

Methods To Control For Threats To Validity

Pre and posttestsUse of comparison groupsRandom assignments

Comparison Of Evaluation Designs

Design Groups Pretraining Posttraining Cost Time Strength

Posttest-only Trainees No Yes Low Low LowPretest/Posttest Trainees Yes Yes Low Low

Med. Posttest-only Trainees No Yes Med. Med. Med. comparison comparison Pretest/posttest Trainees Yes Yes Med. Med.

High comparison comparison Time series Trainees Yes Yes, several Med. Med.

Med.Time series Trainees Yes Yes, several High Med.

High comparison and reversal

Determining Return On Investment

Cost-benefit analysis is the process of determining the economic benefits of a training program using accounting methodsTraining cost information is important for several reasons:

1. To understand total expenditures2. To compare the costs of alternatives3. To evaluate the proportion of money 4. To control costs

Determining CostsSeven categories related to the cost sources:

1. Costs related to program development2. Instructional materials for trainers/trainees3. Equipment 4. Facilities5. Travel/lodging6. Salary of trainer7. Cost of lost productivity while employees

attend the program

Determining BenefitsMethod used to identify the potential benefits of training

1. Technical, academic, and practitioner 2. Pilot training programs3. Observance of successful job

performance4. Asking trainees and their manager to

provide estimates of training benefits

Cost-Benefit EstimateFor example, one group of 20 trainees estimated the total overall monthly cost benefit of $336,000 related to business improvements, and showed an average 70 percent confidence level with that estimate. Seventy percent multiplied by $336,000 gave a cost-benefit of $235,200. This number was divided by 20 ($235,20020 trainees) to give an average estimated cost benefit for the 20 trainees ($11,760).

Return On Investment To calculate return on investment use the following steps:

1. Identify outcomes2. Place value on the outcomes3. Determine the change 4. Obtain an annual amount of benefits5. Determine the training costs (direct costs +

indirect costs + development costs +overhead costs + compensation for trainees)

6. Calculate the total savings7. Calculate the return

Determining Costs For a Cost-Benefit Analysis

Direct CostsInstructor $0In-house instructor (12 days @ $125 per day) 1,500Fringe benefits (25% of salary) 375Travel expenses 0 Materials ($60 X 56 trainees) 3,360Classroom space and audiovisual equipment 600 (12 days @ $50 per day) Refreshments ($4 per day X 3 days X 56 trainees 672Total direct costs 6,507

Indirect CostsTraining management 0Clerical and administrative salaries 750 Fringe benefits (25% of salary) 187Postage, shipping, and telephone 0Pre- and posttraining learning materials ($4 X 56 Trainees) 224Total indirect costs 1,161

Determining Costs For a Cost-Benefit Analysis

Development Costs Fee for program purchase 3,600Instructor training Registration fee 1,400 Travel and lodging 975 Salary 625 Benefits (25% of salary) 156Total development costs 6,756

Overhead CostsGeneral organizational support, top management time 1,443 (10% of direct, indirect, and development costs) Total overhead costs 1,443

Compensation for traineesTrainees’ salaries and benefits (based on time away) 16,969Total training costs 32,836Cost per trainee 587

Determining Benefits For a Cost-Benefit Analysis

Operational How Results Results Differences ExpressedResults Area Measured Before After dollars Training Training Quality Percentage 2% rejected 1.5 % rejected .5% $720 per day, of panels rejected 1,220 panels 1,080 panels 360 panels $172,800 per-day per day per yr. Housekeeping Visual 10 defects 2 defects 8 defects not inspection (average) (average) measurable using 20 item in $ checklist

Preventable Number of 24 per year 16 per year 8 per yr. $48,000 accidents accidents per yr. Direct Cost of $144,000 per yr. $96,000 per yr. $48,000 per yr. accidentsROI= Return = Operational results = $220,800 =6.7 Investment Training costs $32,835