Survey Experiments
Dec 23, 2015
Defined
• Uses a survey question as its measurement device
• Manipulates the content, order, format, or other characteristics of the survey as a treatment
Methodological Issues
• Missing Data
• Matching
• Both can be an issue in experiments other than surveys
Missing Data
• Some observations missing data on the DV or IVs
• If missing at random, not a problem to drop from the analysis
• But usually not missing at random• Deleting non-random missing causes bias
Missing Data II
• Data can also be missing intentionally:• Some cases not “treated”• Possible to “guess” what would have
happened to a subject had they been in another treatment group– Allows within-subject comparison of two
treatments, the one they received and the one they could have received
Solution: Imputation
• Suppose Yi = a + b1Xi1 + b2Xi2 +ei
• But Yi missing for some observations
• Xi1 and Xi2 not missing
• Regress Y on Xi1 and Xi2 for all non-missing observations
• Use b1 and b2 to calculate predicted Ypi
Better Yet: Multiple Imputation
• Ypi is a predicted value with uncertainty
• Multiple imputation predicts multiple values for Yp
i drawn from a distribution of predicted values
• 5 or so predicted Ypi sufficient for inference,
no need for many• Gary King’s Amelia program available free on-
line
Matching
• Experiments can be pre-matched to avoid large random sample
• Match subjects on important characteristics such as– Sex– Race– Age– Education levels– Other traits?
Matching
• Often necessary in field experiment when randomization more difficult to control
• propensity score is the probability of an observation being assigned to a particular treatment in a study given a set of known variables.
• Propensity scores reduce selection bias by equating groups based on these variables
Question
• Why do people change their answers to survey questions if the order of questions changes?
• Does changing survey responses indicate that people do not have well-formed opinions
Theory
• Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue
• Separable Preferences: What a person wants on every issue is independent of what they get on other issues
Method
• Randomize the order of pairs of survey questions– For some issues, aggregate responses different
across question order
• Each subject answers questions in order– Issue 1 then Issue 2– Issue 2 then Issue 1
Method
• Impute what subject would have answered had they heard questions in different order
• For each question we then haveYi (if first) – Yi (if second)
• One of these will be imputed for each person since they cannot answer a question both first and second in the order
• First study to analyze individual differences in question orders, not simply aggregate differences
Conclusions
• Nonseparable preferences explain question order effects
• Political information level does not • Response instability not due to uninformed
respondents
Are Survey Experiments Externally Valid?
JASON BARABAS and JENNIFER JERITAmerican Political Science Review
2010
Question
• Many survey experiments expose subjects to different information to show effect of on responses
• In a survey experiment, subjects are a “captive audience” that must pay attention
• Do the same information effects appear in the real world
• Compare survey experiments with natural experiments
Method
• Survey experiments give people to political information about immigration and medical care
• Pre-post survey also in field during change in medical insurance and immigration– Ask respondents which media sources they use
• Is the effect of information in the survey experiment as large as in the natural experiment?