ESOMAR Panel Research 2006 Barcelona The Effects of Panel Recruitment and Management on Research Results Dutch Online Panel Study NOPVO 2006 An initiative of Pieter Willems - Millward Brown Robert van Ossenbruggen - ProCression Ted Vonk - Onderzoekpaleis with the support of the MOA
43
Embed
ESOMAR Panel Research 2006 Barcelona The Effects of Panel Recruitment and Management on Research Results Dutch Online Panel Study NOPVO 2006 A n initiative.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ESOMAR Panel Research 2006 Barcelona
The Effects of Panel Recruitment and Management on Research
Results
Dutch Online Panel StudyNOPVO 2006
An initiative ofPieter Willems - Millward Brown
Robert van Ossenbruggen - ProCression Ted Vonk - Onderzoekpaleis
with the support of the MOA
ESOMAR Panel Research 2006 – Barcelona - 2
COPYRIGHT
This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006).
It is allowed to publish the outcomes of this study if you refer to:
• The name of the study (NOPVO 2006)
• The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown)
• Does the type of respondent explain the differences between panels?
• Conclusions and recommendations
ESOMAR Panel Research 2006 Barcelona
Introduction
ESOMAR Panel Research 2006 – Barcelona - 5
Some facts and figures about The Netherlands
Demographics
• Population: 16,3 million
• 18-65 years: 10,5 million
• Internet penetration: 82%
• Broadband penetration: 63% (of all households, not internet users)
Market Research
• Total market in 2005: € 278 million ($ 346 – 12th largest market)
• Online data collection: 25% of quantitative revenue
• Over 25 commercial online access panels
• Majority of panels exist for longer than 5 years
ESOMAR Panel Research 2006 – Barcelona - 6
The Dutch Online Panel Study (NOPVO)
Why?1. To provide an answer to quality concerns
2. To provide input for guidelines for online panel innovation
3. A unique situation: large number of panels available
Research questions - NOPVO1. How do panels work? Panel overview
2. Who is the online panel member?
3. How do panellists respond?
4. To what extent is opinion and marketing data influenced by the choice of panel?
No objective: to qualify good or bad panels
ESOMAR Panel Research 2006 – Barcelona - 7
Two parts of NOPVO
1. Inventory of all online panels in the Netherlands1. Answers to the 25 questions of ESOMAR
2. Overview available at www.moaweb.nl
2. Empirical study• One survey
• Every panel draws sample of 1000 respondents
• Same fieldwork period of 7 days (no reminders)
• Omnibus questionnaire to avoid selective response
19 panels participated:
Representing over 1.600.000 panellists
ESOMAR Panel Research 2006 – Barcelona - 8
Extensive data available
Per panel:-Type of incentive-Age of Panel-Invitation policy -etc. Per panel member:
-Duration of panel membership-Method of recruitment-Number of invitations-Individual response rate-Socio demos, etc.
Per respondent:-Response (time)-Membership of other panels- Data from the questionnaire
ESOMAR Panel Research 2006 – Barcelona - 9
Some facts about participating panels
• Almost all panels give incentives for cooperation• Usage of different types of incentives• Average € 1,20 for 10 minutes
• Most panels have a restriction policy for inviting panel members to complete a survey
• 1.650.000 million panel members• Smallest panel:15.000• Largest panel: 230.000
• Several panels do not have full details available about interview history of panel members
ESOMAR Panel Research 2006 – Barcelona - 10
Observations about the samples• Composition of samples to duration of panel membership differs a
lot per panel• Panels with mainly respondents recruited during last 6 months• Panels with mainly panel members recruited during set up of panel
• Recruitment methods vary substantially across panels • Most panels use a mixture of recruitment methods• Some panels use one single method (online, by telephone)
Recruitment method of panel members in gross sample
16%
15%
22%
6%
18%
1%
1%
5%
10%
5%
Buying addresses
Self registration
Via links/banners
Snowballing
By telephone
From existing panels
Advertising in print media
Via website surveys
Via traditional research
Other
ESOMAR Panel Research 2006 Barcelona
Different approaches, different results?
ESOMAR Panel Research 2006 – Barcelona - 12
Differences in results between panels limited
• There will always be some fluctuations because of sampling (sampling error)
• Most of the differences between panels are about what one could expect because of confidence intervals
• There are huge differences between extreme scores• Extreme scores lead to different conclusions • Change of panel could lead to change in trends
Substantial differences between minimum and maximum scores
53%
12%
60%
21%
62%
19%
69%
35%
82%
31%
77%
50%
Paid job for over 15hours per week
Travels with publictransport toworkplace
Awareness Corona Surfing on theInternet: 10+ hours
per week
min total max
ESOMAR Panel Research 2006 – Barcelona - 13
Substantial differences caused by a few panels with extreme scores
How to read the graph
•Largest differences (N=19)
•Minimum and maximum excluded (N=17)
•2 minima and maxima excluded (N=15)
Reduced differences by excluding extreme panels
0%
20%
40%
60%
80%
100%
n=19 n=17 n=15 n=19 n=17 n=15 n=19 n=17 n=15
Paid job for over 15 hoursper week
Travels with publictransport to workplace
Awareness Corona
min total max
28% 15% 9% 19% 10% 9% 17% 9% 7%
Extreme score are spread over all panels•Lowest number of extreme scores: 1%
•Highest number of extreme scores 12%
ESOMAR Panel Research 2006 Barcelona
Does the response rate affect the
results?
ESOMAR Panel Research 2006 – Barcelona - 15
NOPVO response overall: 50%
• Level of response is a result of several factors
• Panel age, method of recruitment, incentives, panel care, etc
• It is hard to determine the effects of a single factor
• NOPVO response overall: 50%• 19.000 panel members invited….9.462 completed the questionnaire…
• Substantial differences in response per panel
Response per panel
18 19
36 39 39 39 40 4149 54 55 56 60 60 62 66 68 70
77
50%
ESOMAR Panel Research 2006 – Barcelona - 16
High/low response explained(1)• Panels with a low response rate are panels that do not remove non-
responding panellists from their panel database• Panels invited panel members who did not complete a single questionnaire in
over a year despite many invitations• Number of panel members is an important communication topic to the market
• New panel members respond better than old panel members
• Effect of incentives• Certainty about incentive is important• Positive effect of incentive stops at 1 euro (10 minute questionnaire)
Response per type of incentive
70%
62%
58%
49%
39%
29%
Cash
No incentive
Value points
Charity and lotteries
Value points and lotteries
Lotteries
response %Response and value of incentive
59%
57%
37%
42%
> 1 euro
1 euro
< 1 euro
No (fixed) incentive
response %
ESOMAR Panel Research 2006 – Barcelona - 17
High/low response explained(2)
• Method of recruitment: • Recruitment by invitation (by telephone, traditional research) gives
higher response than self selection (self registration, links/banners)
Response per recruitment method
58%
45%
By invitation
Self selection
respons %
ESOMAR Panel Research 2006 – Barcelona - 18
Outcomes do not depend on response rate
Number of …Panels with
response <40%
Panels with response 40-60%
Panels with response>60%
Words in open ended question 24,7 25,7 25,9
Awareness beer brands (spont.) 6,1 6,1 6,3
Awareness beer brands (aided) 20,1 19,8 20,0
Glasses beer per week 11,1 10,9 10,4
Advertisements seen 5,8 5,8 5,8
Awareness TV brands (aided) 13,8 13,5 13,8
No different means on survey questions between low/high responding panels
ESOMAR Panel Research 2006 – Barcelona - 19
No response effects explained
• Non response stages
• During recruitment
• Removal of bad/non responding panel members• For a specific survey
• Non-response effects only occur if there are differences between respondents and non-respondents• There is no distortion because of missing categories (they are missing
in all panels)
• Members of online panels are a homogeneous group• They are willing to participate in surveys• They are heavy internet users
• Non-response effects take place while recruiting panels• double opt-in• explicit consent to participate in research
ESOMAR Panel Research 2006 – Barcelona - 20
Representative power of panels is more limited than assumed
• Missing groups in panels• Ethnic minorities/immigrants are missing almost completely
(10% of population, 35% in major cities)
• They are missing in all data collection methods
• Structural deviations in NOPVO study • Too many heavy Internet users
• Too many voters (90% instead of 79%)
• Voting behaviour compared with last elections
> Number of Christian Democrat voters (16% instead of 29%)
> Too many Socialist Party voters (14% instead of 6%)
• People who go to church are underrepresented
• Large part of the population is not represented in the panels
ESOMAR Panel Research 2006 – Barcelona - 21
Response rate: panel strategy
• A high response rate does NOT indicate high sample quality
• Response rate is a panel strategy component, driven by rules of economics• Panel setup and maintenance determine response rate
• Value of response rate is evaluated against recruitment costs
• Response rate can be influenced by• Excluding hard to reach groups
• Selection of good i.e responding panellists
• The response problems of traditional research are not solved. Online research is dealing with it more efficient.
ESOMAR Panel Research 2006 – Barcelona - 22
Panel overlap diminishes panel management effects • 62% of respondents are a member of multiple panels
• Average number of memberships of all respondents: 2,73 panels
• Panel overlap varies between 29% and 88% per panel
• Highest single overlap between two panels: 42%
With this level of overlap a restriction policy for inviting panellists does not make much sense
Number of panel memberships1 panel
38%
2 panels15%
3 panels14%
4 panels10%
6-10 panels13%
Over 10 panels
3%
5 panels7%
ESOMAR Panel Research 2006 – Barcelona - 23
Highest overlap with addresses from a broker
• 81% of panellists recruited with bought addresses are members of multiple panels
• Recruitment by invitation (telephone, traditional research) smallest overlap
• Average number of panels varies between 2,0 recruitment by telephone and 4,3 by bought addresses
Number of panels per recruitment method
19 26 33 4053
63
38
4346
46 35
3429
39
2820 14 18
9 616
Buyingaddresses
Via links/banners
Via websitesurveys
Selfregistration
Viatraditionalresearch
By telephone Total
%
1 panel 2-5 panels over 6 panels
ESOMAR Panel Research 2006 – Barcelona - 24
20%-80% rule is valid for online panel research
• Multiple panel members respond better and faster
• Limited group of panellists complete substantial part of all questionnaires
Respondents, completed questionnaires and panel memberships
78%
20%
6%
12%
5%
13%
4%
12%
2%4%
25%
1%8%
9%
900.000 Panellists25.000.000 completed
questionnaires
11 panels of more
6-10 panels
5 panels
4 panels
3 panels
2 panels
1 panel
ESOMAR Panel Research 2006 Barcelona
Does the type of respondent explain differences between
panels?
ESOMAR Panel Research 2006 – Barcelona - 26
“It's the perfect job for women who want to stay at home with their children, or anyone who needs extra money they would like to earn from home.”
“By joining this site, you are joining a community of professional survey-takers with information on the best, paid survey sites in the UK, links to UK survey news, bulletin boards and advice on the business of online paid surveys.”
• The ‘loyal’ respondent: takes time to complete questionnaire and puts effort in open ended questions
• Operationalization NOPVO: ‘index loyalty’=
1. duration of interview (Z) +
2. # words open ended questions (Z)
• Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 29
Nice to meet you: the Inattentive Respondent
• The inattentive respondent: opposite of loyal respondent
• Thus: speeds through questionnaire and minimum input open ended questions
• Operationalization NOPVO:‘Index inattentiveness’= index loyalty * -1
• Selection for analysis: top 10%
ESOMAR Panel Research 2006 – Barcelona - 30
Present typology analogous to earlier studies
1 Smith, R. & Hofma Brown, H., (2005). Assessing the Quality of Data from Online Panels: Moving Forward with Confidence. White paper, Harris Interactive.
2 Downes-Le guin, T. (2005). Satisficing Behavior in Online Panelists. White paper, Doxus.
also relevant in this discussion:
Knapton, K. (2006). Catch Me If You Can! (How to Keep Cheaters Out of Your Next Online Survey). Presentation at The Market Research Event. Los Angeles, October 23, 2006.
NOPVO Similar to… Harris1 Doxus2
Professional Hyperactives
Loyal Optimizers
Inattentive Inattentives Satisficers
Fraudulents Gamers
ESOMAR Panel Research 2006 – Barcelona - 31
Respondent type & characteristics
• Professional respondent• internet frequency • ‘fresh’ panels • education • unfit for work • satisfaction income • democracy • perceived health
• Loyal respondent• education
• Inattentive respondent• older respondents
ESOMAR Panel Research 2006 – Barcelona - 32
35%56% 47%
17%
23%
14%
17%
19%
33%
9%
30%
31%
14%
45%
17%
23%
Total professional loyal inattentive
Traditional datacollection
Adresses purchased
Through links/banners
Other
Recruitment: loyals from traditional data collection methods
inattentive:Self select
loyal: By invitation
professional: Self selection
recruitment panellist (data available of 12 panels)
ESOMAR Panel Research 2006 – Barcelona - 33
4% 1% 3%
54%
39%
69%
34%
17%
22%
9%
30%
26%
39%
17%
33%
5%
Total professional loyal inattentive
Nice to do
Incentive
Other intrinsic drivers*
Other
Motivation: incentive or intrinsic
inattentive:incentive
driven
loyal: intrinsic
motivations
professional: nice to do
Reasons for participating in internet surveys. Which one is the most important to you?
*Such as:
• Curiosity
• I like to give my opinion
• I like to see results
• I like to be part of an online
community
ESOMAR Panel Research 2006 – Barcelona - 34
7% 8%15%
16% 18%
9%
22%
22% 21%
18%
21%
43% 42%
50%
34%
12% 11%21%
8%
2%Totaal professioneel loyaal gemakzuchtig
Strongly
Normal
Some
A bit
not interested
Political interest: large differences
inattentive: weak
politicalinterest
loyal: strong
politicalinterest
professional: no
effect
To what extent are you interested in politics?
ESOMAR Panel Research 2006 – Barcelona - 35
Add familiarity: differences between types up to 15%...• Largest effect for inattentive respondents
• Professional respondent just as atypical as loyal respondents
(zero =
total)
-12%
-8%
-4%
0%
4%
8%
12%
Ach
mea
Cit
roën
Do
ve
Fo
rtis
Maz
da
Niv
ea
OH
RA
ON
VZ
T-M
ob
ile
Vo
daf
on
e
Vo
lksw
agen
professional loyal inattentive
ESOMAR Panel Research 2006 – Barcelona - 36
Brand familiarity: differences between types up to 20%...
-15%
-10%
-5%
0%
5%
10%
15%
Aiw
a
Aka
i
Ban
g &
Olu
fsen
Bek
o
Fin
lux
Gru
nd
ig
Hit
atch
i
JVC
Len
co LG
Lo
ewe
Met
z
NE
C
Ori
on
Pan
aso
nic
Ph
ilip
s
Pio
nee
r
San
yo
Sam
sun
g
Sh
arp
So
ny
Th
om
son
To
shib
a
Ves
tel
Vie
wso
nic
professional loyal inattentive
ESOMAR Panel Research 2006 – Barcelona - 37
Inattentive respondents: the easy way out
1%2%
14%
6%
1% 1%
12%
7%
0%1%
12%
4%
7%6%
21%
12%
clicking through('straightlining')
'don't want to say' (1+) 'don't know' (1+) 6 or more (of 10) neutralscores on 5-p scale
total professional loyal inattentive
ESOMAR Panel Research 2006 – Barcelona - 38
Main concern: the inattentive respondent
• Q: Is the concern for the professional respondent justified?
• A: Partly, because this type of respondent is atypical, but the effects are not large
• Q: Is the loyal respondent also the ideal respondent?
• A: This is material for further investigation• Serious effort (by definition)• High data quality• But: atypical… now who really is atypical? The loyal or the not so loyal?
• Q: Is the inattentive respondent a threat to online panel research?
• A: Absolutely: atypical group with large effects responsible for noise in data
ESOMAR Panel Research 2006 – Barcelona - 39
Respondent behaviour varies between panels
relatively fewinattentiverespondents
relatively manyinattentiverespondents
0% 20% 40% 60% 80% 100%
S
R
Q
P
O
N
M
L
K
J
I
H
G
F
E
D
C
B
A
total
inattentive
professional
loyal
other
ESOMAR Panel Research 2006 Barcelona
Conclusions
Conclusions
• In general: the differences in outcomes between panels are limited despite differences in panel management and panel maintenance
• However, there are differences• The differences are not a result of differences in response or sample
composition. • Respondent types, defined by questionnaire experience or effort,
display more or less atypical and suspect response behaviour• Differences between panels can substantially be explained by
differences in distribution of these respondent types
• Panels comprise a specific group of respondents that differ on relevant criteria from the national population. The representative power of online panels is more limited than assumed so far
• Response percentage does not indicate sample or panel quality. The response rate is an indication of the level of efficiency of the panel provider.
• Online panels have a substantial overlap in terms of panellists. This reduces the impact of a panel management strategy.
ESOMAR Panel Research 2006 – Barcelona - 42
Recommendations
• Monitor suspect respondents and remove the really bad ones
• Deploy other, new recruitment methods to get the missing part of the population in the panels
• Use online panels for the right kind of research goalsuse panels for