1 16 training best practices
Post on 01-Nov-2014
681 Views
Preview:
DESCRIPTION
Transcript
Training Best Practices:Needs Assessment and Evaluation
PresentedJanuary 25, 2008
By Kristi L. Thompson, M Ed
Sierra Associateswww.KristiThompson.com
Sierra Associates 2008 2
Objectives
• Participants will be provided the information and tools needed to conduct a basic Needs Assessment for course development.
• Participants will be provided the information and tools needed to conduct a Course Evaluation.
• Participants will be provided some tips on structuring budget requests.
Sierra Associates 2008 3
Learning is provided in order
to improve performance
on the present job.
(Nadler, 1984)
Sierra Associates 2008 4
What have you done today to enhance
(or at least insure against the decline of)
the relative overall useful-skill level
of your work force
vis-a-vis competitors?
- Tom Peters
Thriving on Chaos
Sierra Associates 2008 5
To Train or Not High
Job Knowledge
Low
Problem: Low Motivation
Method
Assess personal consequences/ rewards
system
Problem: Systemic
Method
Consider system issues, problem is out of control of the employee
Problem: Bad Fit
MethodConsider improper
placement of employee in the position
Problem: Lack of Knowledge or Tools
MethodTraining
Low High
Employee attitude/desire to perform the job
Sierra Associates 2008 6
My Favorite Two
Sierra Associates 2008 7
Training Approaches
Intervention Approach
Training is an intervention for solving problems involving employees. Focus is on performance and/or organizational results as corrections to problems. Steps taken when performance issue is identified.
System Approach
Training is a part of a continuous improvement process, fully integrated in the regular process of organizational improvement. Some steps done only once.
Sierra Associates 2008 8
Instructional Design – Our Focus
Analyze – Analyze system (department, job, etc.) to gain a complete
understanding of it– Compile an inventory of all tasks associated with each job (if
needed)– Select tasks that need to be trained– Build performance measures for the tasks to be trained– Choose instructional setting for the tasks to be trained– Estimate what is going to cost to train the tasks
• Design• Develop• Implement• Evaluate
– Review and evaluate during each phase (analyze, design, develop, implement)
– Perform evaluations– Revise training system based on evaluations Full ISD model in Appendix
Sierra Associates 2008 9
What is an Needs Assessment?
“Needs assessment is the systematic effort that we make to gather opinions and ideas from a variety of sources on
performance problems or new systems and technologies.”
Allison Rossett (1987)
Sierra Associates 2008 10
Why do a Needs Assessment?
• To make sure we are applying the right solution to the problem
• To identify what learning will be accomplished
• To identify what changes in behavior and performance are expected
• To determine the expected economic costs and benefits
Sierra Associates 2008 11
Steps of a Needs Assessment
• Conduct Task Analysis and compile a Task Inventory
• Perform a Gap Analysis
• Select which tasks will be addressed
• Determine performance measures for the trained task
• Select Training Method
• Estimate training costs
Sierra Associates 2008 12
Task Analysis/Inventory
• What• The breakdown of performance into detailed levels of specificity• Statements of
what will be donehow it will be done for what end result
Ex: Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays.
• Why• To determine the operational components of a job, skill, goal, or
objective• To describe what and how they are performed• To describe the sequence and scope
• When• Whenever there are new processes or equipment, when job
performance is below standards, or when requests for changes to current training or for new training are received
Task Inventory
Sierra Associates 2008 13
Task Inventory – Example
Department: IT Help Desk Date: July 7, 2007Analyst: Barty Crouch Department Supervisor: Susan MalfoyJob Title: Administrative Assistant Job Code: 0742
Task Number and Task: 0742-1 Types orders received by mail, telephone, or in person at a minimum rate of
45 WPM into a computer database under general supervision of the HelpDesk Manager to document calls.
0742-2 Receives and answers customer questions under close supervision of theHelp Desk Manager in order to provide good customer relations.
0742-3 Delivers triage and delivery information received by mail, telephone, or inperson under general supervision of the Help Desk Manager to fulfillcustomer requests.
0742-4 Acts as liaison between customer and various departments under closesupervision of the Help Desk Manager in order to provide good customersatisfaction.
0742-5 Post and maintains records in a computer database without supervision to provide the department with historical records for statistical needs.
….
Sierra Associates 2008 14
Gap Analysis Performance Behaviors
• What • Comparison of actual performance against new or existing
standards
• Why • To identify the performance gap between what is actually done and
what is required or expected
• When• An intervention is required• New processes and/or procedures• New equipment/hardware• New Applications • New technologies• Change in staffing• Reduction in productivity• Governmental mandates• Security breaches• Routinely, as part of continuous improvement process
Sierra Associates 2008 15
Gap Analysis Questions
• For each identified task:• Determine current performance level on task• Determine desired performance level
Task Current Performance Level
Desired Performance Level
Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays.
Emp. often makes mistakes, missing info approx 50% of the time
Emp. repeatedly asks for assistance on producing report, particularly on recalculating Pivot tables
Reports are fully complete and accurate every week
Emp. is able to produce report with no assistance
Sierra Associates 2008 16
Tasks Selection
• What• A determination of which tasks will be addressed in the training• Directs course objectives
• Why• To determine the scope and content of the training
• When• An intervention is required• New processes and/or procedures• New equipment/hardware• New Applications • New technologies• Change in staffing• Reduction in productivity• Governmental mandates• Security breaches• Routinely, as part of continuous improvement process
Sierra Associates 2008 17
Tasks Selection
• Required– The task/topic is required by law or for safety
• Risk– There is a high risk if task is not done correctly– The task is critical
• Complexity– Task is difficult or complex– Task is done frequently– Task is time-consuming– Task is critical to the performance of the role/project
• Team Considerations– Task requires coordination with other staff or with other tasks– Task is part of a collective set of tasks
• Performance– Task is required for acceptable role performance– Task distinguishes star performers
Task Selection
Sierra Associates 2008 18
Task Selection - example
Task Current Performance
Level
Desired Performance
Level
Select - Criteria
Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays.
Emp. often makes mistakes, missing info approx 50% of the time
Emp. repeatedly asks for assistance on producing report, particularly on recalculating Pivot tables
Reports are fully complete and accurate every week
Emp. is able to produce report with no assistance
Include – Performance
Sierra Associates 2008 19
Select Training Format
• What• A determination of training delivery mode
• Why• Identify the best tool for the job• Determine training materials and format• Begin to understand possible costs
• When• For every training intervention or project
Can be done at the task level or just generally
Sierra Associates 2008 20
Training Format Options
• One-on-one/small group coaching• Boot Camp – small, very specialized audience• Job Aid• Classroom• Vendor course• OJT• Mentoring• CBT• e-Learning• Text-based• Video• Active Learning
Training Format flowchart
Sierra Associates 2008 21
Budget
• What do They want?– The greatest needs addressed with the least cost
(ROI)
• Structure your requests by– Projects supported (the bigger the better)– Productivity increase expected (SWAG!)– Number of employees to affected by a particular
course/product (one product, many people)– Problems “solved” with metrics
Training Costs Worksheet
Sierra Associates 2008 22
No money, no time, no resources
Step Method
1. Identify performance problems
Questionnaire/Survey
Focus group (see Lunchtime Assessment)
2. Select which tasks Use the most reported from the survey
3. Develop performance measures
You decide based on time and resource limits for course
4. Select method of instruction/setting
Lecture/classroom
5. Estimate costs/budget On location with internal SME/trainer
Sierra Associates 2008 23
Lunchtime Assessment1. Gather a possible audience (Pizza works well.) Ask each person to write
down their ten most important training needs in a particular area. Emphasize “specific needs”. A good format is “I need to know how to…”
2. Capture the training needs on the white board or flip chart. Avoid duplicate answers by questioning.
3. Use a weighted voting process to prioritize the training needs across the group (sticky dots) to prioritize the list. Give half as many dots as there are items. Tell needs assessment participants to place their dots on the chart to vote for their priorities.
4. List the training needs in order of importance, with the number of points assigned as votes determining priority, as determined by the sticky dot voting process.
A Step Beyond
• Have each person mark their top two needs on their lists of ten most important and then collect the lists.
• Share these Top Twos with the manager for personal development work.
Sierra Associates 2008 24
Basic Data Gathering
TYPE OF INFO
needed
SOURCES OF INFORMATION
OBSERVATION INTERVIEW
DOCUMENTATION
AUDIENCEObserve in work environment
Interview/survey audience or supervisors
Review employee files or personnel records
TASKS
Observe audience or expert performing
Interview expert or other performers
Review job descriptions, policy statements, and trouble reports
CONTENT
Observe expert or creators of product/process
Interview SMEs, policymakers, marketers, or managers
Review product plans, specifications, and marketing guidelines
Sierra Associates 2008 25
IT Training Needs Assessment?
IT Trait Process Modification
Obsolescence •Use vendor training for very specialized skills/tasks•Define supported tools/apps/skills and limit training to these
Enormous amt of possible subject matter:
New products daily
New services daily
CHANGES daily!
•Define supported tools/apps/skills and limit training to these•For static info – develop documentation •For active info – consider the fastest Training method
Entire campus as possible audience with different needs/levels/application for same tasks
•Identify audience groupings related to skills/knowledge/performance needed.•Tweak content/materials for each audience.•Train the Trainer – use department IT stars
Sierra Associates 2008 26
From Pamphlet 350-70-6http://www-tradoc.monroe.army.mil/tpubs/pams/p350-70-6.pdf
The linkage of Assessment and Evaluation
Sierra Associates 2008 27
Evaluation Along the Way
• Phase 1 - Analysis phase• Determining if it really is a training problem • Make sure there isn’t already a solution available
• Phase 2 - Design phase• Ensure the course content link directly to the objectives
• Phase 3 - Development phase • Perform rapid-prototyping (implement-evaluate-implement-
evaluate-etc) or • Have another training specialist check the solutions
• Phase 4 – Implementation phase• Use Level 1 evaluations (next slides)
Sierra Associates 2008 28
Four Levels of Evaluation
• Reaction • Learning• Behavior or performance • Outcomes or results
Sierra Associates 2008 29
Level One - Reaction
• Measures and evaluates the learner’s reactions and opinions about training program itself
• Does not measure what new skills the learners have acquired or what they have learned that will transfer back to the working environment.
• Method - Attitude questionnaires after classes
COWC Level One Level One Questionaire
Sierra Associates 2008 30
Level One - Questions to Ask
• Rate the instructor • Rate the course• Estimate how much you have learned in the course• Rate the effectiveness of the instructor in stimulating
your interest in the subject• Rate the effectiveness of this course in challenging you
intellectually
Minimally– What major conclusion did you draw from today's
session?– What major questions remain in your mind?
Sierra Associates 2008 31
Level Two - Learning
• Measures and evaluates the changes in the participants’ skills, knowledge, or attitudes as a result of the training: did they learn anything?
• Methods – Pre- and post-testing • Questions should be directly linked to the selected
tasks and the course content
Sierra Associates 2008 32
Level Three - Performance (behavior)
• Measures and evaluates the transfer of the learning to the job or organization: can and do people use their newly acquired learning on the job?
• Methods – • Formal (testing or Manager interview)• Informal (observation)• Must take place after the learners have returned to
their jobs
Manager Level 3 Evaluation
Employee Level 3 Evaluation
Sierra Associates 2008 33
Level Four - Results
• Measures and evaluates the impact of the training on the productivity and profitability of the organization: what "return" has the organization received from the training?
• Methods –– Develop a "balanced scorecard“ with four perspectives
• Financial • Customer Satisfaction • Internal Perspective• Innovation and Learning
– Linked to Vision/Mission statement on these aspects– Allow time for the behavior change to take place– Evaluate both before and after the program
Sierra Associates 2008 34
Balanced Score Card
• Financial • A measurement, such as an ROI, that shows a monetary
return, or the impact itself, such as how the output is affected• (Can be either soft or hard results.)
• Customer• Improving an area in which the organization differentiates
itself from competitors to attract, retain, and deepen relationships with its targeted customers
• Internal• Achieve excellence by improving such processes as supply-
chain management, production process, or support process• Innovation and Learning
• Ensuring the learning package supports a climate for organizational change, innovation, and the growth of individuals
VinFin Balanced Scorecard example
Sierra Associates 2008 35
Big Picture
•
Sierra Associates 2008 36
References
• Rossett, Allison, (1987). Training Needs Assessment (Techniques in Training and Performance Development Series. Educational Technology Pubns (August 1987)
• Kirkpatrick, Donald, (1994). Evaluating Training Programs – The Four Levels. San Francisco, CA: Berrett-Koehler Publishers, Inc.
• Multimedia Courseware Development Guidehttp://www.tradoc.army.mil/tpubs/pams/p350-70-2.doc
• Writing a learning objective standard that measures Learning Objective (LO) performance.http://www.asat.army.mil/briefings/Learning%20objective%20standard%20writing.doc
• Big Dog's Performance Coaching Page http://nwlink.com/~donclark/perform/coach.html
• Design Training Needs Assessment Surveys - http://adulted.about.com/od/trngneedsasst/a/TNAsurveys.htm
Sierra Associates 2008 37
Resources
• First Things Fast: A Handbook for Performance Analysis by Allison Rossett (Nov 6, 1998)
• Job Aids and Performance Support: Moving From Knowledge in the Classroom to Knowledge Everywhere (Essential Knowledge Resource) by Allison Rossett and Lisa Schafer (Nov 3, 2006)
• A Handbook of Job Aids by Allison Rossett and Jeannette Gautier-Downes (Jun 15, 1991)
• Beyond the Podium: Delivering Training and Performance to a Digital World by
Allison Rossett and Kendra Sheldon (May 23, 2001) • Article – What is a Balanced Scorecard
http://www.balancedscorecard.org/BSCResources/AbouttheBalancedScorecard/tabid/55/Default.aspx
Digital Tools Survey
Sierra Associates 2008 38
Training Requirements
Sierra Associates 2008 39
• Kristi Thompson is the Principal of Sierra Associates. She has over 20 years of experience in the development of stronger work environments. Kristi works with groups on workplace and relationship processes to redirect the focus and perspective, resulting in higher morale, focus and productivity. She has applied her expertise in the high-performance manufacturing, service industry, hi-tech companies, academic institutions and the medical field. She has presented on numerous topics such as Change, Conflict, Team Development and Quality Improvement.
• Kristi received her Masters of Education from the University of Massachusetts Amherst and her Bachelors of Science in Chemistry from Worcester Polytechnic Institute.
• Sierra Associates• 8 Munn Road• Monson, MA 01057• 413 267 3585• Cell: 508 789 4112• www.KristiThompson.com• sierra@samnet.net
Sierra Associates 2008 40
Data gathering methods 1
Method Pros Cons
Questionnaire/ Survey population
–Can customize per audience–Can be anonymous–Can and should ask different style questions–Efficient in collecting lots of data–The easiest way of collecting needs assessment data.
–% return is often less than 20%–Takes time and skill to develop well–Directed/limited answers–Should be limited to 10 minute completion time–What people tell you and what they really do may differ–Can’t probe deeper
Documentation review (i.e., job descriptions, performance appraisals)
-Provides management/HR perspective
–May be difficult or impossible to obtain desired documents–May require interpretation/ explanation
Interview managers/ supervisors
–Good for complex or undefined problems/areas–Can be modified “on the fly”–Personal touch
Time and resource intensive
Sierra Associates 2008 41
Data gathering methods 2
Method Pros Cons
Observations –The most direct method of collecting needs assessment data
–Very resource consuming (time and people resources)
Review performance appraisals of lesser performers
–Time efficient –Dependent upon the quality of documentation and honesty in documents
–Best used in conjunction with conversations with Mgr
Focus groups –Good for complex or undefined problems/areas
–Can be modified “on the fly”
–Personal touch
–Ideas can build on each other
–Relatively effective and efficient
–Scheduling issues
–Wide range of participants in attendance
–what people tell you and what they really do may differ
–Sometimes important to verify the results with observations and document analysis
Sierra Associates 2008 42
Steps for a full Analysis
Step Method Toolbox Template
1. Understand the client’s business (Analyze system)
• Interview manager/department head
System Overview Instrument
2. Conduct a Task Inventory*
• See next slides Task Inventory
3. Select which tasks will be addressed
• Discuss with manager/department head/client
Task Selection
4. Build performance measures for the trained tasks
• Review performance appraisals
• Interview manager/department head
• Interview manager/department head
•Ask HR/EAP
Task Performance Measure
Task Inventory
Task Selection
Task Performance Template example
System Overview example
Sierra Associates 2008 43
Steps for a Needs Assessment
5. Select method of instruction/ instruction setting
Lockstep
Classroom (In-house or Vendor)
Boot Camp
Video
Self-Paced
Personalized System of Instruction (PSI)
Computer Based Training (CBT)
Text Instruction
e-learning or Internet Distance Learning (IDL)
Job
Job Performance Aid (JPA)
On-The-Job (OJT)
Training Method Decision Flow
6. Estimate costs
Training Cost Worksheet
top related