1 Quality Control for Field Operations
1
Quality Control for Field Operations
2
Overview
Goal• To ensure the quality of survey field
work
Purpose• To detect and deter interviewer errors
and falsification
3
Overview (cont.)
What do we check in QC for field work?• Interviewer procedures• Forms• Interviewer productivity• Interviews• Address listing
4
Overview (cont.)
What are the tools for performing QC on field work?• Interviewer observations• Formal / Informal reviews• Monitoring reports• Reinterview• Address listing QC
5
Interviewer Observations
What are the goals of interviewer observations?• To assess effectiveness of interviewer
training
• To ensure interviewers understand and follow prescribed interviewing procedures
• To provide interviewers with feedback to encourage continuous improvement
6
Interviewer Observations (cont.)
What are interviewer observations?• Supervisor “shadows” interviewer during
initial assignments• Observe interviewer’s work to ensure
appropriate procedures are followed• Evaluate interviewer’s work and provide
positive and negative feedback to interviewer
• See Handout on Field Control Form Information
7
Interviewer Observations (cont.)
What are the results of observations?• Supervisor decides if interviewer’s
performance is acceptable
• If so, allow interviewer to continue work independently
• If not, re-train interviewer on procedures or release interviewer from the operation
8
Formal / Informal Reviews
What are the goals of reviews?
• To ensure the quality of the data collected on forms and questionnaires
• To ensure interviewers and clerks understand and follow procedures for completing forms and questionnaires
9
Formal / Informal Reviews (cont.)
How are informal reviews performed?• Supervisor meets with employees to
review forms and/or questionnaires
• Review for legibility, completeness, and accuracy
• Provide positive and negative feedback to employee for continuous improvement
• Repair errors where necessary
10
Formal / Informal Reviews (cont.)
How are formal reviews performed?• Usually performed in a field office by
office staff
• Review questionnaires or listings for legibility, completeness, and accuracy
• Repair errors or return materials to the field when necessary
11
Formal / Informal Reviews (cont.)
What work products are subject to reviews?
• Any form or questionnaire that requires a quality check (e.g., payroll forms, questionnaires, address listing forms, etc.)
12
Formal / Informal Reviews (cont.)
What are the results of informal reviews?• Supervisor determines if the form or
questionnaire meets quality standards
• If so, the form/questionnaire is accepted and submitted
• If not, repair the form/questionnaire prior to submission
13
Monitoring Reports
What are the goals of monitoring reports?
• To track status of operations
• To inform management of progress
• To identify problems early for timely rectification
14
Monitoring Reports (cont.)
How do we know what reports to monitor for QC purposes?• Define quality goals for the operation (e.g.,
response rates, productivity rates, cost, etc.)
• Identify variables that provide information on quality goals
• Design report(s) that present the important variables for quick review and assessment
15
Monitoring Reports (cont.)
How do we monitor the reports for QC purposes?
• Determine necessary frequency for monitoring (usually daily)
• Look for outlier values (e.g., low response rates, high costs, etc.)
• Determine reason(s) for outlier values
• Take appropriate action
16
Reinterview
What is reinterview?
• Quality check on interviewing activities
• Primary tool for ensuring quality of interviewers’ field work
• Independent, objective review of interviewing work
17
Reinterview (cont.)
What types of reinterview can be used?
• Random reinterview
• Administrative reinterview
• Supplemental reinterview
18
Reinterview (cont.)
What decisions are made in reinterview?• Falsification (release interviewer)
• Interviewer error (retraining/feedback)
• Respondent error (no action)
• Rework may be necessary for falsification or other serious errors
19
Random Reinterview
What is the purpose of the random reinterview component?
• Provides broad protection over entire workload and all interviewers
• Serves as an effective deterrent for interviewer falsification
20
Random Reinterview (cont.)
What is the sampling design for the random reinterview component?• Select random sample of cases within
every enumerator’s workload, or
• Two-stage design: take a sample of interviewers, then a sample of cases within those interviewers’ workloads
21
Administrative Reinterview
What is the purpose of the administrative reinterview component?
• Provides “targeted” approach to identifying suspect work
• Provides valuable performance data to field supervisors
22
Administrative Reinterview (cont.)
How does the administrative model work?• Accumulates data on work
characteristics for all interviewers• Vacant housing units• Partial interviews• Single-person households• Deleted housing units• Average population per housing unit
23
Administrative Reinterview (cont.)
How does the administrative model work? (cont.)
• Compare accumulated data for each interviewer to the data for all other interviewers in their assignment area
• Identify and flag interviewers who are significantly different than their area average
24
Administrative Reinterview (cont.)
How does the administrative model work? (cont.)
• Provide field supervisors with a list of “outlier” interviewers and what variables caused the flag
• Field supervisors determine if the interviewers’ “outlier” status is explicable
• More of suspect interviewer’s cases put into reinterview
25
Supplemental Reinterview
What is the purpose of the supplemental reinterview?
• Provides field supervisors with the capability of checking work for selected interviewers
• Tool for use during investigation of suspected falsification
26
Supplemental Reinterview (cont.)
What is the sampling design for the supplemental reinterview?
• Select a random sample of cases from the interviewer’s workload
• Field supervisor can specify how many cases to check
27
Address Listing QC
What is the goal of address listing QC? To ensure the quality of address
listing work
• To protect against undercoverage and overcoverage
28
Address Listing QC (cont.)
What are the tools for performing address listing QC?
Dependent QC
• Advance listing
• Suppression
29
Address Listing QC (cont.)
What decisions are made in address listing QC?
• If errors exceed acceptable limit, recanvass assignment area
• Repair minor mistakes
• Listers may be released or retrained depending on severity of errors
30
Dependent QC
What is dependent QC?
• Random, independent check of addresses listed by survey listers
• Field supervisors verify existence of housing units or validity of housing unit deletes
31
Dependent QC (cont.)
What is the sampling design for dependent QC?
• Select a random sample of housing units to verify from the listers’ workload
• Sample should include all types of listings (e.g., verifications, adds, deletes)
32
Advance Listing
What is advance listing?
• Field supervisor does initial listing of assignment areas prior to making assignments to listers
• Gives supervisor knowledge of the area so they are able to detect faulty or inadequate work
33
Suppression
What is suppression?
• Selected addresses are purposely left off listings
• Listers’ ability to find and add the suppressed addresses is assessed
• Provides confidence in the listers’ ability to perform adequate canvassing