AAPOR 2015 Mini-Conference Reassessing Today's Survey ...– Our own panel (iSay) – Other panels 2. ‘Ad hoc’ panelists REALLOCATED from other surveys 3. Non-panel aka River –

© 2015 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary informationand may not be disclosed or reproduced without the prior written consent of Ipsos.

May 17th, 2015

AAPOR 2015 Mini-ConferenceReassessing Today's Survey Methods:Voters, Election and Non-Probability Samples

Meta-Analysis of Online Paneland Non-Panel Sampling:Electoral and Non-ElectoralBehavior Metrics

2

The survey research industry is in an era of transition and flux: ‘traditional’ methodologies (now defined aslive interviewer landline phone and cellular phone) are waning in prevalence – if not importance – whilenewer methodologies (online, IVR, blended) are gaining prominence. The impact of this on measurementaccuracy is still being explored, with new information emerging all the time.

To further this body of knowledge specifically within online work, we analyze behavior measurementaccuracy of the two main sampling approaches for internet-based research: panel only and non-panel(River and other approaches) sampling. Our paper will comprise a meta-analysis of nonprobablity onlinesampling, looking across over 400 data points, at election and non-election behaviors as validating points.Behavior measurement accuracy will be assessed against external benchmarks from market-accepted datasources such as the US Census, Current Population Survey (CPS), and the General Social Survey (GSS).

The analysis will focus specifically on differences in accuracy between sample sources or blendedcompositions, as well as which socio-demographic subgroups are most accurate across a range of differentvalidation measures.

Abstract

3

Dr Clifford YoungPresident

US Public Affairs

Dr Robert PetrinVice President

US Public Affairs

Julia ClarkSenior Vice President

US Public Affairs

4

5

Our world is changing…has changed

66

7

Our client wanted their data…

Interactive (visuals & video)Faster (same-day & daily)Granular (huuuuge samples)

We gave them a RIVER of data...

11,000 interviews/month (c. 350/day), 365 days/yearQuestionnaire changes dailyAssessing same-day events (parsing)

Responding to Client Needs

8

So – we innovated

IPSOS BLENDED SAMPLING(aka: panel + non-panel)

9

Types of Nonprobability Sample

1. ‘Known’ panels from which we draw targeted sample– Our own panel (iSay)– Other panels

2. ‘Ad hoc’ panelists REALLOCATED from other surveys

3. Non-panel aka River– ALL non-panel sample, including social media,

communities, gaming, reward & loyalty programs, and adnetworks

All goes into the ROUTER

10

Think of the router as a very clever funnel

Your Survey

Respondent

Respondent

Respondent

RespondentRespondentRespondent

OtherSurveys

OtherSurveys

QualifyDoes NotQualifyR

EALLOCATION

11

This approach requires a HIGH volume of survey work

o c. 50 river suppliers from over 200 sites

o Screen c. 130,000 respondents /day

o Yields c. 4 million respondents each month

65% -75% of these are unique respondents

o Over 48,000,000 respondents a year

12

We know it works(and, for many of our clients, ‘knowing it works’ is good enough…)

13

• Measuring incidence of people who received flu shot in January 2013• 40% of Americans stated that they already got a flu shot this season;

CDC figure 41.5%

Flu Shot Study (also: 2012 and 2014 elections)

14

Research Question:What are the differences from anaccuracy standpoint between thesenonprobability sample types?

15

Aggregate data from May 1st 2014 through May 1st

2015

Focus on 37 questions with 102 variables 612cases for which there is a robust market comparator(Census, CPS, other government statistics, largepublished probability-based survey, etc.)Looked at Average Absolute Difference

Methodology

16

133,623 interviews total…

Ipsos Panel(iSay), 25%

ReallocatedPanel, 25%

River, 41%

OtherPanels,

9%

Ipsos Panel (iSay)Reallocated PanelRiverOther Panels

17

133,623 interviews total…

Ipsos Panel(iSay), 25%

ReallocatedPanel, 25%

River, 41%

OtherPanels,

9%

Ipsos Panel (iSay)Reallocated PanelRiverOther Panels

18

FINDINGS

19

First, some important dynamics re. online generally, and ‘market comparisons’

QUESTION RESPONSES WEIGHTED UNWEIGHTED MARKET TOTAL DIFF ISAY DIFF AMPARIO DIFF

TotalWeighted

WGT iSayallocated

WGT Ampario(re-allocated)

TotalUnweighted

UNWGTiSay

allocated

UNWGTAmpario (re-

allocated) MARKET WGT UNWGT WGT UNWGT WGT UNWGTWhich of the options below bestdescribes how your think of yourself?

Heterosexual/straight

91% 92% 91% 92% 93% 92% 96.6% 6.1% 4.2% 5.0% 3.7% 6.1% 4.3%Gay/lesbian 3% 4% 3% 3% 3% 2% 1.6% 1.7% 1.1% 1.9% 1.4% 1.2% 0.8%Bisexual 4% 3% 4% 3% 3% 3% 0.7% 3.0% 2.3% 2.4% 2.0% 3.1% 2.3%Other 3% 2% 3% 2% 1% 2%Total 133623 33320 54935 133623 33320 54935

How often do you attend religiousservices?

Seldom/never 41% 41% 41% 39% 39% 39% 29.0% 12.3% 10.4% 11.6% 9.9% 12.2% 10.4%Monthly or yearly 34% 34% 33% 35% 35% 34% 33.0% 0.9% 1.7% 1.5% 1.9% 0.0% 0.9%Weekly or more 19% 20% 18% 20% 22% 20% 37.0% 18.3% 16.5% 16.8% 15.1% 18.6% 17.0%Don’t know/noanswer

6% 5% 8% 5% 4% 7% 1.0% 5.2% 4.4% 3.8% 3.2% 6.5% 5.6%Total 133623 33320 54935 133623 33320 54935

Would you describe yourself as aborn-again or evangelical Christian?

Yes 31% 30% 31% 29% 28% 30% 35.0% 4.3% 5.6% 4.7% 6.7% 4.1% 4.8%No 69% 70% 69% 71% 72% 70% 65.0% 4.3% 5.6% 4.7% 6.7% 4.1% 4.8%Total 78051 16933 32384 78051 16933 32384

TM491Y15_TOT - Do you currentlyhave health insurance from a stategovernment health coverageexchange or throughHealthcare.gov? (BASED TO TOTALRESPONDENTS, ASKED OF OUTOF POCKET OR MEDICAID ONLY)

Yes 10% 9% 11% 9% 9% 9% 8.8% 1.4% 0.1% 0.7% 0.2% 1.9% 0.3%No+Not Sure+notapplicable

90% 91% 89% 91% 91% 91% 91.2% 1.4% 0.1% 0.7% 0.2% 1.9% 0.3%Total 23267 4951 9924 23267 4951 9924

20

FINDING 1:Panelists are more ‘engaged’ (RVs; Twitter users)

QUESTION RESPONSES WEIGHTED

MARKET

TOTALDIFF ISAY DIFF RIVER DIFF

Twitter - Howfrequently, if at all,do you use oraccess the followingsites?

TOTAL PANEL RIVER TOTAL PANEL RIVERUse TOTAL 40% 44% 38% 23.0% 16.9% 20.6% 14.6%

Do not use 60% 56% 62% 77.0% 16.9% 20.6% 14.6%

Are you currentlyregistered to vote,or not?

Yes 79% 82% 78% 71.2% 7.5% 10.6% 6.5%

No 18% 16% 18% 28.8% 10.7% 12.7% 10.4%

21

FINDING 2:River sample more likely to say ‘Don’t know’

QUESTION RESPONSES WEIGHTED

PANEL RIVERAPPROVAL2 -Summary

TOTAL APPROVE 40% 38%TOTALDISAPPROVE

54% 54%

Not sure 6% 8%Total 33320 54935

Are you currentlyregistered to vote, ornot?

Yes 82% 78%No 16% 18%Don’t know /Refused

2% 4%

Total 33320 54935How often do youattend religiousservices?

Seldom/never 41% 41%Monthly or yearly 34% 33%Weekly or more 20% 18%Don’t know/noanswer

5% 8%

Total 33320 54935

QUESTION RESPONSES WEIGHTED

PANEL RIVERFor whom did youvote for President in2012?

Barack Obama(Democrat)

51% 52%

Mitt Romney(Republican)

40% 38%

Other 6% 6%Unsure / Refused 3% 5%Total 26857 41117

Generally speaking,would you say thingsin this country areheading in the rightdirection, or are theyoff on the wrongtrack?

Right direction 25% 23%Wrong track 61% 60%Don’t know 14% 16%Total 33320 54935

22

FINDING 3:Panel and Non-Panel also showed strong similarities for

UNWEIGHTED demographics

DEMO UNWEIGHTED

TOTAL PANEL RIVER

DEMOG AVERAGE 6.8% 7.4% 6.8%

TOTAL AVERAGE 5.7% 6.2% 5.4%

23

FINDING 4:Panel and Non-Panel were surprisingly similar, with the biggest

gap on attitudinal variables(e.g. on marijuana legalization; ACA)

NON DEMO WEIGHTED

TOTAL PANEL RIVER

SOCIAL AVERAGE 5.2% 5.4% 4.7%

POLITICAL AVERAGE 4.6% 5.3% 4.5%

ATTITUD AVERAGE 4.3% 5.2% 3.8%

TOTAL AVERAGE 4.2% 5.1% 4.1%

24

Thank You

[email protected]

[email protected]

[email protected]