Improving the Effectiveness of Interviewer Administered Surveys though Refusal Avoidance Training Grace E. O’Neill Presented by Anne Russell U.S. Census Bureau ICES-III - June 19, 2007
Mar 27, 2015
Improving the Effectiveness of Interviewer Administered Surveys
though Refusal Avoidance Training
Grace E. O’Neill
Presented by Anne Russell
U.S. Census Bureau
ICES-III - June 19, 2007
2
Outline of Presentation
• Background
• Respondent focus groups
• Refusal avoidance training
• Training content
• Discussion
• Future developments
3
Background
4
Respondent contact staff
• Interviewers: outgoing calls to gain cooperation and to gather data
• Clerks: incoming calls to provide basic information and resend forms
• Analysts: professional staff who place outgoing calls concerning data errors or nonresponse follow-up
5
Respondent contact training
• Interviewers: Centralized training on telephone skills and refusal avoidance and conversion
• Clerks: Shadow experienced clerks with informal discussion
• Analysts: Basic training on respondent contact techniques
6
Why clerks?
• Self-administered paper/ electronic forms
• Establishment respondents need less encouragement to participate
• Nature of telephone calls
7
However…
Over time, the role of the clerk has evolved into the role of an interviewer
• Less cooperative respondents
• More data collected over the telephone
• More nonresponse follow-up telephone calls
Yet, their training has not evolved
8
Respondent Focus Groups
9
Focus Groups
• Monthly Trade Surveys
• Wholesale and retail firms
• Conducted by outside firm
• Six focus groups, 44 participants
• Respondent’s views and impressions
• Ways to improve survey
10
Focus Groups
Finding
• Inexperienced clerks needed skills to increase participation by respondents, especially on voluntary surveys
Solution
• Refusal avoidance training
11
Refusal Avoidance Training
12
Refusal Avoidance Training
• Groves and McGonagle (2001)
– Assemble respondent concerns
– Develop responses
– Train interviewers to classify concerns
– Train interviewers to provide quick and appropriate responses
13
Refusal Avoidance Training
• Interactive
• Cooperative learning
• Flexible
• Survey specific
• Provides telephone skills, refusal avoidance techniques, and improves communication
14
Training specifics
• Monthly Trade Surveys (MTS) and Quarterly Services Survey (QSS)
• 43 MTS clerks, 14 QSS clerks
• Mix of tenure and survey experience
• Supervisor and survey manager involvement
• Eight hours over two days
15
Training Preparation
• Adapt household- based training to establishment-based survey
• Adapt training to the MTS or QSS
• Identify respondent concerns and develop solutions
• FAQ Job Aid
16
Training Content
17
Refusal avoidance training modules
Module 1: IntroductionModule 2: Survey SpecificModule 3: Shared ExperienceModule 4: Preparing a Telephone CallModule 5: Telephone SkillsModule 6: Identify, Analyzing, and Dealing with
Reluctance and RefusalModule 7: Recovering from Negative CallsModule 8: Wrap-up and Evaluation
18
Module 1
Introduction
• Introduces training to clerks
• Introduces the trainer and clerks to each other
• Provides training schedule
19
Module 2
Survey Specific
• Introduces survey specific content
• Conducted by survey manager or survey staff
20
Module 3
Shared Experience
• Identify clerks’ biggest concerns and difficulties
• Develops solutions
• Concerns are used in a later module
21
Module 3 example
• Trainer: “What are some of the things respondents say when you talk to them?”
• Clerks: “Why should I do this,” “I’m not a wholesaler” etc.
• Group: Identify appropriate responses • Trainer: Make sure identified concerns
are addressed and adds new concerns to list
22
Module 4
Preparing for the Telephone Call
• Asks clerks how they prepare for telephone calls
• Assess what tools clerks use to find information about company and what tools they might need
23
Module 5
Telephone Skills
• Assess clerks active listening skills
• Discuss tone
• Discuss mechanics of placing a telephone call
24
Module 6
Identifying, Analyzing and Dealing with Reluctance and Refusal
• Classify concerns identified in Module 3 as reluctance and/ or refusal
• Further discuss solutions
• Paired practice during class
• Reviewed updated FAQ Job Aid
25
Module 7
Recovering from Negative Calls
• Discusses recovering from refusals and other negative calls
• Helps clerks to evaluate negative experiences
26
Module 8
Wrap-up and Evaluation
• Review main training points
• Clarify any remaining concerns
• Evaluations by clerks
27
Results
28
Results
Clerk evaluation
• Usefulness of workshop
• Usefulness of skills learned
• Increased confidence, preparation, and communication
29
Results
Response rates
Caveats:
• Not experimentally tested
• Confounded by other survey conditions
• Don’t know how many potential refusals clerks prevented
30
Results: MTS
Initially lower from previous response period
• Loss of clerks
• Misclassification of refusals
31
Results: MTS
Over time
• Good response rates, decrease in wholesale refusals
• Continued improvement
• Clerks have more responsibilities
32
Results: QSS
Decline in response rates
Imputation remained stable
• More companies refused
• Larger companies continued to report
33
Results: QSS
• Communication has improved
• Clerks are open about sharing concerns with survey managers
• Bi-weekly telephone calls with call center
• Annual refresher training
34
Discussion
35
Benefits
• Centralized dissemination of skills and information
• Practice occurs in a test environment
• Provides training at regular intervals
• Proactive training instead of reactive training
36
Benefits
• Communication between clerks, supervisors, and survey managers
• Clerks feel invested in data collection process
• Survey managers gain direct insight into data collection process
37
Costs
• Staff time writing and delivering training
• Telephone coverage
• Monetary cost of training
• Difficult to provide conclusive evidence
38
Future Developments
39
Future Developments
• Formalized process for clerk training
• Data capture of call concerns
• Follow-up evaluation by clerks
• Analyst training
40
Thank You
Please feel free to contact the author at:
Grace E. O’Neill
Email: [email protected]
Phone: 301-763-3537