NCRM is funded by the Economic and Social Research Council NCRM is funded by the Economic and Social Research Council 1 Interviewers, nonresponse bias and measurement error Patrick Sturgis University of Southampton Research Methods Festival, Oxford, 2-5 July 2012
25
Embed
Interviewers, nonresponse bias and measurement error
Interviewers, nonresponse bias and measurement error. Patrick Sturgis University of Southampton. Research Methods Festival, Oxford, 2-5 July 2012. Co-authors. Ian Brunton -Smith (University of Surrey) Joel Williams (TNS-BMRB ). Background and Motivation. Common Causes of Survey Error. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council1
Interviewers, nonresponse bias and measurement
errorPatrick Sturgis
University of Southampton
Research Methods Festival, Oxford, 2-5 July 2012
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Co-authors
• Ian Brunton-Smith (University of Surrey)
• Joel Williams (TNS-BMRB)
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Background and Motivation
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Common Causes of Survey Error
• Recent attention has focused on common causes of nonresponse and measurement error (cf. 2010 special issue of POQ on Total Survey Error)
• Agencies often target field resources at persuading reluctant respondents to meet response rate targets
• These respondents are less motivated and potentially less able to complete questionnaire accurately
4
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Error trade-offs• So potential reduction in nonresponse bias my
be offset by increase in measurement error
• Growing evidence that this does happen in practice (Kreuter et al 2010; Sakshaug et al 2010)
• This work has focused on respondents so far
• What about interviewers?
5
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Interviewers as common cause?• Interviewers can cause nonresponse bias and
measurement error• Success in obtaining contact and cooperation related to
interviewer characteristics:– Tailoring– Maintaining an interaction– Personality– Attitudes and beliefs
• Some interviewers don’t get interviews, where ‘better’ interviewers would
• If these ‘lost’ respondents are different on survey variables, result is biased population estimates
6
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Interviewer variance
• Interviewers cause measurement error through the way they administer the questionnaire
• At the individual level, these can be considered biases (response different to true value)
• But across respondents and interviewers the result is larger variances
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Interviewer Variance • E.g. interviewer always reads same question
incorrectly
• Across interviews, this creates within-interviewer correlation – same as geographical clustering
• Interviewer contribution to variance of estimator denoted ρInt
8
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
How is ρInt related to nonresponse?
• Anecdotal evidence that more successful interviewers on the doorstep are less diligent at sticking to questionnaire wording and instructions
• Alternatively, some interviewers are good at what they do, others are not so good
• Either way, we should anticipate a correlation between response rate and ρInt?
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Conceptual model 1
• Implications for total survey error• MSEint = bias2 + varianceint
Agreeable-ness
Response rate
Deviation from questionnaire wording and instructions
+
ρInterviewer
Tailoring
++
Nonresponse bias
VarianceInt+
-+
-
10
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Conceptual model 2
• Implications for total survey error• MSEint = bias2 + varianceint
Conscientiou-sness
Response rate
Deviation from questionnaire wording and instructions
+
ρInterviewer
Tailoring
++
Nonresponse bias
VarianceInt
+
-+
+
11
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Analytical approach
• Fit cross-classified multilevel models to face to face interview data
• Partition ρ into area and interviewer components
• Examine variation in ρInt across distribution of measures of interviewer success in obtaining contact and cooperation seperately
12
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Measuring interviewer success
• Average response rate problematic as indicator of interviewer success on the doorstep
• Our measure of interviewer success– Calculate ‘expected’ response propensity for all original issue cases
based on geodemographic characteristics and paradata– Take mean of ratio of expected to observed rate across all cases for
each interviewer– Do this separately for contact and cooperation– Group into ‘success quantiles’
13
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Modelling strategy
• Adjusts estimates for clustering of interviews within sample points• Models also include individual, interviewer and area controls to account for
non-random allocation of respondents to interviewers
Cross-classified multilevel models with a complex interviewer error structure
Allows simultaneous estimation of separate ρInt for each interviewer success quintile
NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council
Data and analysis
• British Crime Survey (2005/06)– 43,465 respondents , 472 interviewers, 3,782 areas
• 36 items asked of all respondents which were non-factual and included probes and/or show-cards
• Cross-classified multilevel model with complex error term at interviewer level