Comparing different weighting procedures for volunteer online panels Stephanie Steinmetz and Kea Tijdens AIAS Lunch Seminar, 1. October 2009 erasmus studio
Sep 08, 2014
Comparing different weighting procedures for
volunteer online panels
Stephanie Steinmetz and Kea Tijdens
AIAS Lunch Seminar, 1. October 2009
erasmus studio
Outline
BackgroundSources of errors in ((volunteer) web) surveysWeighing - a solution?Example for the German and Dutch WageIndicator dataResultsConclusion and Outlook
Background
Increasing importance of web surveysGermany: between 2000 und 2007 from 3% to 27% (ADM, 2007)
Advantagestime and cost reduction, interactivity, flexibility, ‘worldwide’ coverage, no interviewer influence
DisadvantagesRepresentativeness? To what degree are(volunteer) web survey results representative of the general public?
Types of web surveys (see Couper, 2000)
Sample selection isprobability based = representative- intercept surveys,- online access panels,- mixed-mode surveys
not probability based = representative? - entertainment surveys - self-selected web surveys - volunteer online panels
Sources of error
Combination of causes1 (Non)Coverage: number of people having internet access +
differences between the persons with and without internet access.
2 Sampling/Self-selection: no comprehensive list of Internet users to draw probability-based sample + people with specific characteristics participate in a volunteer online panel.
3 Non-response: Not all persons finish the questionnaire, people with specific characteristics might have a higher non-response.
+ measurement, processing and adjustment errors
Weighting - a possible solution?
Weighting is a mean to correct subsequently for systematic survey errors and to adjust the sample to the target population.
Expectation disappearance of significant differences between web survey & random reference survey.☺ = web survey data can be adjusted to be representative
of general public.= persistence of differences due to other error sources,
like measurement and processing errors
Solution: Post-stratification weighting
Aim: Adjustment for demographic under- and over-representations between sample and target population
Method: %population (reference data) / %sample (web) = weighting coefficient
Findings: Necessary but has a rather limited impact (Vehovar et al. 1999, Loosvelt and Sonck 2008)
corrects for proportionality but not necessarily for representativeness of substantive answers
...but
Previous research comparing web and traditional methods
Significant differences can be observed for web respondents. They...– are more intensive users of the Internet, more
technically-oriented (Bandilla et al. 2003; Vehovar et al. 1999)
– have a larger social trust & a greater subjective control over their lives (Lenhart et al. 2003)
– are more politically and socially active (Duffy et al. 2005)
Solution: Propensity Score Adjustment (PSA)
Origin: experimental studies (Rosenbaum & Rubin, 1983)
Aim: to correct for differences due to the varying inclination to participate in web surveys (Harrison Interactive).
Findings: Mixed (Taylor 2005; Bethlehem & Stoop 2007)
- some differences disappeared by demographicweighting,
- some only after additional PSA, and
- others continued to exist or become even larger
PSA - method (see Schonlau et al. 2009)
Web and probability-based reference survey are combined in one data fileLogistic regression of people’s probability to participate in the web survey given demographic and/or attitudinal variables estimation of PSMake distribution of these propensity scores similar for web survey and random sample = calculation of weight wpsi (1/ psi if W = 1 (in the web survey), and 1/(1-psi) if W = 0 (in the reference survey))
web survey and random sample do not differ significantly for selected variables included in the PS
Example - the WageIndicator data
Web surveys: German and Dutch WageIndicatordata, year 2006, employees, age 16-75, cross monthly income 400€-10000€ (Dutch net hourly income)
NGerman= 21914NDutch = 8015
Reference surveys: Same restrictions Germany (GSOEP, 2006) N= 7993Netherlands (OSA, 2006) N= 2019
Selection bias - socio-demographics
02040
6080
100
men
wom
en
low
med
ium
high
16-3
4
35-4
4
45-7
5
sex education cohort
LS SOEP
02040
6080
100
men
wom
en
low
med
ium
high
16-3
4
35-4
4
45-7
5
sex education cohort
LW OSA
Germany Netherlands
Selection bias - Labour markert
0
20
40
60
80
100
manual non manual full part below above
occupation working time unemployment
LS SOEP
020406080
100
manual nonmanual
full part below above
occupation working time unemployment
LW OSA
Germany Netherlands
Selection bias - satisfaction
0
20
40
60
80
100
not satisfied satisfied not satisfied satisfied
health satisfaction job satisfaction
LS SOEP
0
20
40
60
80
100
not satisfied satisfied not satisfied satisfied
health satisfaction job satisfaction
LW OSA
Germany Netherlands
Summary
Similarities: underrepresentation of women, people between 45 und 75, part-timers,persons from regions with high unemployment,unsatisfied people
Differences: underrepresentation ofDE: highly educated, manual workersNL: low and medium educated, non-manual workers
Two possible solutionsa) Post-stratification weightingb) PSA
Weights
A) 6 post-stratification weights: W1= gender (2), education (2) and cohort (2) W2= gender (2), education (2), cohort (2) and part time (2)W3= gender (2), education (2), cohort (2) and nonmanual (2)W4= gender (2), education (2), cohort (2), part time (2) and jobsatW5= gender (2), education (2), cohort (2), nonmanual (2) and jobsatW6= part(2) and jobsat(2)
B) 4 PSA weightsPS1 = treat women edu2 coh2 nonman part perm nojob logwagemoPS2 = treat women edu2 coh2 nonman part perm nojob logwagemo +
healthsatPS3 = treat women edu2 coh2 nonman part perm nojob logwagemo +
jobsatPS4 = treat women edu2 coh2 nonman part perm nojob logwagemo +
healthsat jobsat
Results: Germany – Mean income
0 € 50 € 100 € 150 € 200 € 250 € 300 € 350 €
Diff
Diff1
Diff2
Diff3
Diff4
Diff5
Diff6
PS1
PS2
PS3
PS4
Diff Diff1 Diff2 Diff3 Diff4 Diff5 Diff6 PS1 PS2 PS3 PS4
Results: Germany – distributions
-30
-20
-10
0
10
20
30
men women 16-34 35-65+ lowmed medhigh manual nonman full part dissat sat
Gender Cohort Education Nonmanual Part-time Jobsat
Diff DiffW2 DiffW6 DiffPS1 DiffPS2
Results: Germany - Income Regression
Results: NL – mean income
-0,6 € -0,5 € -0,4 € -0,3 € -0,2 € -0,1 € 0,0 € 0,1 € 0,2 €
Diff
Diff1
Diff2
Diff3
Diff4
Diff5
Diff6
PS1
PS2
PS3
PS4
Diff Diff1 Diff2 Diff3 Diff4 Diff5 Diff6 PS1 PS2 PS3 PS4
Results: NL - distributions
-40
-30
-20
-10
0
10
20
30
40
men women 16-34 35-65+ lowmed medhigh manual nonman full part dissat sat
Gender Cohort Education Nonmanual Part-time Jobsat
Diff DiffW1 DiffW2 DiffW3 DiffPS1 DiffPS3
Results: Netherlands - Income Regression
Conclusion
Impact weightingBoth weighting methods show no substantial impact
Moreover- no consistency within weights- for some weights differences become larger (?!)- effect of weights differ between countries
Weighting cannot improve representativeness of (volunteer) web surveys
Problems- Reference surveys (also biased?), mode effects,
unobservables (not measured)
Discussion
Possible solutions for representativeness:Improving weights through inclusion of morevariables or advanced/mixed weighting proceduresOnly mixed-mode surveys (time and cost-reduction disappears)Non-representative use of web survey data
(only for experiments or exploratory analysis) OR
questioning the definition of representativeness (content vs. methodological)survey quality ≠ absolute
Representativness of surveys
0%
2%
4%
6%
8%
10%
12%
14%
Parttim
e Male
15-24
yr hi
gher
Fullti
me Male
15-24
yr hi
gher
Parttim
e Fem
ale 15
-24 yr
high
er
Fullti
me Fem
ale 15
-24 yr
high
er
Fulltim
e Fem
ale 15
-24 yr
lower
Parttim
e Male
25-44
yr lo
wer
Parttim
e Male
45-64
yr lo
wer
Fulltim
e Fem
ale 45
-64 yr
lower
Parttim
e Male
15-24
yr m
iddle
Parttim
e Male
15-24
yr lo
wer
Fulltim
e Fem
ale 25
-44 yr
lower
Parttim
e Fem
ale 15
-24 yr
lower
Fullti
me Fem
ale 15
-24 yr
midd
le
Parttim
e Male
45-64
yr m
iddle
Parttim
e Male
25-44
yr hi
gher
Fullti
me Fem
ale 45
-64 yr
high
er
Fulltim
e Fem
ale 45
-64 yr
midd
le
Parttim
e Male
25-44
yr m
iddle
Parttim
e Male
45-64
yr hi
gher
Fullti
me Male
15-24
yr lo
wer
Parttim
e Fem
ale 15
-24 yr
midd
le
Fulltim
e Male
15-24
yr m
iddle
Parttim
e Fem
ale 25
-44 yr
lower
Parttim
e Fem
ale 45
-64 yr
high
er
Parttim
e Fem
ale 45
-64 yr
lower
Fullti
me Fem
ale 25
-44 yr
midd
le
Fullti
me Fem
ale 25
-44 yr
high
er
Parttim
e Fem
ale 45
-64 yr
midd
le
Fullti
me Male
45-64
yr lo
wer
Parttim
e Fem
ale 25
-44 yr
high
er
Fullti
me Male
25-44
yr lo
wer
Fullti
me Male
45-64
yr hi
gher
Fulltim
e Male
45-64
yr m
iddle
Parttim
e Fem
ale 25
-44 yr
midd
le
Fullti
me Male
25-44
yr hi
gher
Fulltim
e Male
25-44
yr m
iddle
World Value Survey_NL_% _1999
Labour Force Survey_NL_% _2005
WageIndicator_NL_% _2005
Telepanel_NL_% _2002
Outlook
Comparison: more countries
Methods: - Combination of different weighting
techniques (see Lee & Valliant, 2009)
- Weighting with ‚better‘ reference survey and more webograhic variables (LISS panel= parallel survey, identical questionnaire + same mode)
The end
Thanks' for listening......questions ?...comments and suggestions?
contact: [email protected]
erasmus studio