A Latent Class Call-back Model for Survey Nonresponse Paul P. Biemer RTI International and UNC-CH Michael W. Link Centers for Disease Control and Prevention
Jan 05, 2016
A Latent Class Call-back Model for Survey Nonresponse
Paul P. Biemer
RTI International and UNC-CH
Michael W. Link
Centers for Disease Control and Prevention
Outline
• Motivation for the study
• Early cooperator effects (ECE) in the Behavior Risk Factor Surveillance System (BRFSS)
• Call-back models – Manifest and latent
• Model extensions
• Application to the BRFSS
• Results
• Summary and conclusions
Terms and definitions • Cooperators = units that will eventually respond
at some request or call-back
• Non-cooperators (also called hardcore nonrespondents) = units that will not respond to any call-back
Terms and definitions (cont’d) • Early cooperator = Cooperators that respond at
early calls (say, 5 or less)
• Later cooperators = Cooperators that respond at later calls (say, 6 or more)
• Early cooperator effect (ECE) = expected difference in estimates based on early vs. early + later cooperators (say, )
5E( )y y
Response rates as a function of number of call attempts
0
10
20
30
40
50
60
1-5 6-7 8-9 10-11 12-14 15+
Int
Ref
NC
Number of call attempts
Illustration 1- Have you ever been told by a doctor, nurse or other health professional that you had asthma?
Number of call attempts
1-5 1-15+
Percent “yes” 13.8 13.4
Small ECE maximum of 5 calls is adequate
Illustration 2- During the past 12 months, have you had a flu shot?
Number of call attempts
1-5 1-15+
Percent “yes” 38.3 35.8
Larger ECE max of 5 call attempts may be biasing
Could consider other definitions of “early cooperator.”
Why study ECE?
• Effort (and costs) could be saved if ECE is small
• If ECE is not small, adjustments may be applied to reduce it
• May need to adjust for HCNRs, not only later cooperators
What adjustments can be applied to reduce the ECE?
• Nonresponse adjustments– Requires characteristics of nonrespondents– Lack of information a limitation for some surveys
• Post-stratification adjustments– Requires known target population totals within adjustment
cells– Variables limited to those available externally
• Call-back model adjustments– Assumes response propensity is function of level of effort
required to obtain a response and grouping variables– Related work of Drew and Fuller (1980), Politz and
Simmons (1949), others
ECE in the BRFSS
Survey details
• One of the largest RDD surveys in the world
• Estimates the prevalence of risk behaviors and preventive health practices
• Monthly, state-based, cross-sectional survey
• Target population is adults in telephone hh’s
• Data source: 2004 survey with ~300,000 interviews
ECE in the BRFSS (cont’d)
• Early cooperator defined as responding with 5 fewer call attempts
• Examined differences in – demographic characteristics– 10 selected health characteristics overall and by
demographic domain
• ECE estimated by
• Data weighted by base weights only
5y y
Typical Values of ECE
General Health - Exc Asthma
Drink Alcohol Flu Shot
Prevalence 21% 13% 53% 36%
Total 1.2 0.3 -2.2 2.6
Male 1.3 0.1 -1.7 2.9
Female 1.1 0.4 -2.0 2.2
White, non-Hispanic
1.6 0.1 -2.7 2.4
Black, non-Hispanic
2.5 0.6 -2.2 1.1
Hispanic -0.7 1.5 -0.9 1.4
Typical Values of ECE (cont’d)
EducationGeneral Health Asthma
Drink Alcohol
Influenza Shot
< High school 1.1 1.4 -2.6 2.9
High school 1.4 0.1 -2.5 2.7
> High school 1.0 0.3 -1.7 2.4
Number of adults
One 2.1 0.1 -2.9 3.1
Two 1.1 0.2 -2.0 2.5
Three or more 0.5 0.9 -1.8 1.4
Summary of the Results
• Early cooperators are different from later cooperators on many dimensions
• For most characteristics ECE is relatively small– Less than 3 percentage points at aggregate level– Rarely more than 3 points for domains
• For some characteristics, ECE may be important
• Other definitions of ECE also considered
Hardcore Nonresponse Bias
• Hardcore Nonrespondents = Units that will not respond under the current survey protocol no matter the number of call-backs
• ECE does not include the bias due to hardcore nonrespondents
• Total nonresponse bias = Bias due to cooperators who did not respond + bias due to hardcore nonrespondents
• Adjusting for ECE may not remove bias due to HCNR
Call-back Models for Adjusting for ECE and HCNR Bias
• General idea– Estimate the response propensity for subgroups of the
population– Response propensity is modeled as a function level of
effort (LOE) to obtain a response
• Two models are considered– Manifest model (MM) – Ignores HCNR– Latent class model (LCM) –Includes HCNR
• Includes a latent indicator variable to represent the HCNR’s in the population
• Why latent?
Illustration for 5 Call-backs
Group A Group B Group B
11111 33111 33332
31111 33333 33333
33111 33311 33331
11111 33332 33332
33331 31111 33333
. . . . . . . . .
1 = interview; 2 = noninterview; 3 = noncontact
Illustration for 5 Call-backs
Group A
High response propensity
Group B
Medium response propensity
Group B
Low response propensity
11111 33111 33332
31111 33333 33333
33111 33311 33331
11111 33332 33332
33331 31111 33333
. . . . . . . . .
1 = interview; 2 = noninterview; 3 = noncontact
Potential Advantages over Post-Stratification
• Post-stratification adjustments (PSA’s) depend upon the availability of external benchmarks or auxiliary data– Selection of control variables is quite limited– Target populations also quite limited– Adjust for “ignorable” nonresponse only
Potential Advantages over Post-Stratification
• Call-back model can rely only on internal variables– Weighting classes can be defined for any variables
collected in the survey– Can be applied for any target population– Greater ability to selected variables that are highly
correlated with response propensity– Adjust for “ignorable” and “nonignorable” nonresponse
Modeling Framework
• Simple random sampling
• Survey eligibility is known for all sample members
• No right censoring– (i.e., all noncontacts received maximum LOE)
Extensions to relax these assumptions are described in the paper
Incorporating the Model-based Weights
Unadjusted estimator of the mean
1
1 1 1
ˆrgnK K
r gi g rgg i g
y n y y
Adjusted estimator of the mean
1
K
g rgg
y y
Based on the sample distribution
Estimated from the model
Two Models for Estimating
MM (Manifest Model)
Assumes all nonrespondents would eventually respond at some LOE (i.e., all nonrespondents have a positive probability of response)
LCM (Latent class model)
Incorporates 0 probability of response for the hardcore nonrespondents (HNCR’s)
g
Technical Details
Notation
1,...,l L Levels of effort (LOE)
1,2,3lo Outcome of LOE l where
1=interview, 2 = noninterview, 3=noncontact
*l LOE associated with state S=1 or 2
1,...,g K Grouping variable (weighting class variable)
Notation
*( ,2)n l
( ,3)n L
*,2|l g
*( ,1, )n l g Number of sample persons in group g interviewed at LOE l*
Number of sample persons noninterviewed at LOE l*
Number of sample persons never contacted after L (max LOE) attempts
*,1|l g Probability person in group g is interviewed at LOE l*
Probability person in group g is noninterviewed at LOE l*
,3|L gProbability person in group g is never contacted
General Idea –Outcome Patterns for 5 Call-backs
Cooperator HCNR
11111 0
31111 0
33111 0
33311 0
33331 0
22222
32222
33222
33322
33332
33333
3,1| , 2g x
2,1| , 2g x
4,1| , 2g x
5,1| , 2g x
1,2| , 2g x
2,2| , 2g x
3,2| , 2g x
4,2| , 2g x
5,2| , 2g x
5,3| , 2g x
1,1| , 2g x
1,2| , 1g x
2,2| , 1g x
3,2| , 1g x
4,2| , 1g x
5,2| , 1g x
5,3| , 1g x
Likelihood for the Manifest Model
*
*
*
**,1|
,
*
,2|,
,3|
log ( ) ( ,1, )log
( ,2)log( )
( ,3)log( )
g l gg l
g l ggl
g L gg
n l g
n l
n L
‹
This model is appropriate when
(a) Every sample member has a positive probability of responding at some LOE, or
(b) Adjustment for ECE only is desired
Likelihood for the Latent Class Model
*
*
*
*
*2 ,1| , 2
,
*1 2 ,2| , 2
1 2 ,3| , 2
log ( ) ( ,1, )log
( ,2)log( )
( ,3)log( )
g x l g xg l
x x g l g xgl
x x g L g xg
n l g
n l
n L
‹
Introduces a latent variable X where X = 1, if HCNR and X = 2, if otherwise
Appropriate when some sample members have a 0 probability of responding and adjustment for total nonresponse (Later Cooperators + HCNR’s) is desired
Results
Four Estimators were Considered
• Unadjusted estimator
• Estimator using MM estimates of
• Estimator using LCM estimates of
• Estimator using CPS estimates of– i.e., usual PSA estimator
– treated as the “gold standard”
g
gg
Comparison of the ECE for a Maximum Five Callbacks Strategy Before and After MM Adjustment
GENHLTH
Estimate
%
Unadjusted
ECE
Manifest Model
ECE
Excellent 20.7 -0.9 -0.6
Very good 33.1 -0.4 -0.4
Good 29.6 0.1 0.1
ALCOHOL 52.8 -2.2 -1.8
ASTHMA 13.4 0.3 0.5
DIABETES 8.8 0.7 0.3
FLUSHOT 35.8 2.5 -0.8
HLTHCOV 86.0 0.8 -1.3
PHYMO 18.7 2.2 0.9
Differences between PSA and Unadjusted and Adjusted Estimates for a Maximum Five Callbacks
GENHLTHPSA
Estimate
Diff
Unadj
Diff
MM
Diff
LCM
Excellent 20.7 -0.6 -0.3 -1.2
Very good 33.1 0.3 -0.1 -1.6
Good 29.6 -0.1 -0.1 1.0
ALCOHOL 52.8 -0.6 -0.3 -0.8
ASTHMA 13.4 -0.3 -0.1 0.1
DIABETES 8.8 0.9 0.4 0.6
FLUSHOT 35.8 4.2 0.9 -0.1
HLTHCOV 86.0 2.7 0.6 -2.5
PHYMO 18.7 1.9 0.7 -0.2
Estimating the Potential Bias Reduction
• BRFSS data do not exhibit very large nonresponse biases
• Therefore, consider a variable, Y, that has maximum nonresponse bias given the BRFSS nonresponse rates
• To do this, we form
• Yg BRFSS response rate for group g
• Compute the relative difference between unadjusted and adjusted estimates and the PSA estimate of the mean of Y
Absolute Relative Differences (|RDL|) for Unadjusted and Adjusted Estimators as a Function of Number of Call-backs
No. ofCall-Backs |RDU,L| (%) |RDMM,L| (%) |RDLCM,L| (%)
5 8.8 5.2 1.4
7 6.9 4.0 2.5
9 5.8 3.4 2.9
11 8.8 3.4 2.9
14 4.5 2.6 2.8
15 4.0 2.3 2.4
Conclusions
• ECE for 5 call-backs is generally small, but can be moderately high for some characteristics
• The Manifest Model can be employed to reduce ECE
• The Latent Class Model can be employed to reduce total nonresponse bias (Later Cooperators + HCNR bias)
• Future research should focus on– Variable selection– Comparisons of MSEs of the estimators– Small/medium size sample properties– Integration with other post-survey weight adjustments