Nonresponse Rates and Nonresponse Bias In Surveys

Nonresponse Rates and Nonresponse Bias In Surveys

Robert M. GrovesUniversity of Michigan and Joint Program in Survey Methodology, USA

Emilia PeytchevaUniversity of Michigan, USA

Funding from the Methodology, Measurement, and Statistics Program of the US National Science Foundation, Grant 0297435

Four Mutually-Problematic Observations

1. With 100% response rates probability sampling offers an inferential paradigm with measurable uncertainties for unbiased estimates

2. Response rates are declining3. Keeter et al. (2000), Curtin et al. (2000),

Merkle and Edelman (2002) show no nonresponse bias associated with varying nonresponse rates

4. Practitioners are urged to achieve high response rates

Result: Confusion among practitioners

Assembly of Prior Studies of Nonresponse Bias

• Search of peer-reviewed and other publications• 47 articles reporting 59 studies • About 959 separate estimates (566

percentages)– mean nonresponse rate is 36%– mean bias is 8% of the full sample estimate

• We treat this as 959 observations, weighted by sample sizes, multiply-imputed for item missing data, standard errors reflecting clustering into 59 studies and imputation variance

Percentage Absolute Relative Bias

mean sample full unadjusted the is y

mean respondent unadjusted the is y where

n

r

n

nr

y

yy )(*100

Percentage Absolute Relative Nonresponse Bias by Nonresponse

Rate for 959 Estimates from 59 Studies

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80Nonresponse Rate

Per

cen

tag

e A

bso

lute

Rel

ativ

e B

ias

Conclusions from 959 Estimates

• Examples of large nonresponse bias exist

• Variation in nonresponse bias lies mostly among estimates within the same survey

• The nonresponse rate by itself is not a good predictor of nonresponse bias

• [Note: We cannot infer from the scatterplot about what would happen within a study if response rates were increased]

Thinking Causally About Nonresponse Rates and Nonresponse Error

• Key scientific question concerns mechanisms of response propensity that create covariance with survey variable

where is the covariance between the survey

variable, y, and the response propensity, p

• What mechanisms produce the covariance?

pyyE yp

nr

)(

yp

Alternative Causal Models for Studies of Nonresponse Rates and Nonresponse Bias

2. Common Cause Model

Z

P Y

4. Nonresponse-MeasurementError Model

P

Z

P Y

X

1. Separate Causes Model

Y

Y P

3. Survey VariableCause Model

5. Nonresponse Error Attenuation Model

Y P

Y* Y*

Types of Hypotheses about Influences on Nonresponse Error

• Influences on response rates– urbanicity, gender, age– topic of survey, population’s interest in topic– mode of data collection– prenotification, incentives

• Sponsorship– prior involvement of population with sponsor– government vs. other sponsor

• Types of measures– attitudinal vs. behavioral– questions related to topic of survey vs. others

• Type of statistic– means on counts/continuous variables vs. percentages– differences of subclass means

Types of Hypotheses about Influences on Nonresponse Error

• Influences on response rates– urbanicity, gender, age– topic of survey, population’s interest in topic– mode of data collection– prenotification, incentives

• Sponsorship– prior involvement of population with sponsor– government vs. other sponsor

• Types of measures– attitudinal vs. behavioral– questions related to topic of survey vs. others

• Type of statistic– mean vs. percentages– differences of subclass means

Attributes ofSurveys

Attributes ofEstimates

Exploratory Analysis in Two Steps

Examine only estimates that are percentages, using standardized values of the percentages

• Step 1: pooling all 566 estimates, examine

, difference of respondent and nonrespondent means

• Step 2: separating the estimates by their survey’s response rate, examine

, nonresponse bias

mr yy

)( nr yy

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by

Type of Population

0

0.05

0.1

0.15

0.2

0.25

Specific General

Pro

po

rtio

n o

f S

tan

dar

d D

evia

tio

n

)( mr yy

diff=.075ste=.033

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by

Type of Population

0

0.05

0.1

0.15

0.2

0.25

Specific General

Pro

po

rtio

n o

f S

tan

dar

d D

evia

tio

n

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

<23% 23-38% 39%+

Specific General

)( mr yy )( nr yy

diff=.075 diff= .013 .0013 .045ste=.033 ste= .0092 .0058 .020

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by

Mode

0

0.05

0.1

0.15

0.2

0.25

Self-Administered Interviewer-Administered

Pro

po

rtio

n o

f S

tan

dar

d D

evia

tio

n

00.010.020.030.04

0.050.060.070.080.09

<23% 23-38% 39%+

Self-Administered

Interviewer-Administered

)( mr yy )( nr yy

diff=.040 diff= .00091 .0012 .025ste=.024 ste= .0011 .0055 .011

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by

Involvement with Sponsor

0

0.05

0.1

0.15

0.2

0.25

Involvement withSponsor

No Involvement withSponsor

Pro

po

rtio

n o

f S

tan

dar

d D

evia

tio

n

00.010.020.030.040.050.060.070.080.09

<23% 23-38% 39%+

Involvement with Sponsor

No involvement with Sponsor

)( mr yy )( nr yy

diff=.052 diff= .017 .0014 .029ste=.022 ste= .0079 .0063 .010

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by

Type of Measure

0

0.05

0.1

0.15

0.2

0.25

Behavioral Attitudinal

Pro

po

rtio

n o

f S

tan

dar

d D

evia

tio

n

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

<23% 23-38% 39%+

Behavioral Attitudinal

)( mr yy )( nr yy

diff=.14 diff= .084 ---- .070ste=.022 ste= .0074 ---- .016

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by

Statistic’s Relevance to Topic

0

0.05

0.1

0.15

0.2

0.25

Not Relevant Relevant

Pro

po

rtio

n o

f S

tan

dar

d D

evia

tio

n

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

<23% 23-38% 39%+

Not Relevant Relevant

)( mr yy )( nr yy

diff=.006 diff= .000 -.0014 .014ste=.023 ste= .0074 .0091 .025

Respondent-Nonrespondent Functions on All Estimates by Type of Estimator

0

1

2

3

4

5

6

7

Mean Percentage

No

nre

spo

nse

bia

s

0

0.5

1

1.5

2

2.5

3

3.5

4

<23% 23-38% 39%+

Mean Percentage

)( mr yy )( nr yy

diff=4.37 diff= 1.07 1.38 2.14ste=0.72 ste= 0.38 0.39 0.51

Do Differences of Subclass Means have Lower Nonresponse Bias?

• When estimating subclass differences, we hope that nonresponse biases of the two estimates cancel

• 120 reported estimates of subclass means and their differences

• Only 45 of them have bias of the differences of subclass means lower than average bias of the two subclass means– this comports with only 45 having two subclass

means with biases of the same sign

0

2

4

6

8

10

12

14

16

0 2 4 6 8 10 12 14 16

Absolute Value of Bias of Subclass Mean

Ab

solu

te V

alu

e o

f B

ias

of

Dif

fere

nce

of

Tw

o

Su

bcl

ass

Mea

ns

Absolute Value of Bias of Difference of Subclass Mean by Absolute Value of Subclass Mean

Types of Hypotheses about Influences on Nonresponse Error

• Influences on response rates– urbanicity, gender, age– topic of survey, population’s interest in topic– mode of data collection– prenotification, incentives

• Sponsorship– prior involvement of population with sponsor– government vs. other sponsor

• Types of measures– attitudinal vs. behavioral– questions related to topic of survey vs. others

• Type of statistic– mean vs. percentages– differences of subclass means

Five Summary Statements1. Large nonresponse biases exist2. Most variation in nonresponse biases lie among

estimates in the same survey3. Some types of surveys are more susceptible to biases

(e.g., interviewer-administered, studies of population without prior involvement with the sponsor, general population surveys)

4. Some types of estimates seem more susceptible to biases (e.g., measures of attitudes, percentages, (maybe) estimates related to survey topic)

5. Differences of subclass means do not tend to have lower biases that the individual subclass means

Note: preliminary and exploratory analysis; there is much more to do