Page 1
www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
Redirected Inbound Call Sampling (RICS)–An Emerging Fit for Purpose Non-probability Telephone Sample Design
Burton Levine
Insights Webinar
April 9, 2019
Page 2
Need and Problem
Need:
Organizations must be in a position to provide objective, accurate, and timely
information that informs decision making.
Problem:
“Continued reliance on (traditional*) sample surveys as the principal means of
collecting statistical data is threatened by the increasing difficulty and cost in
conducting the surveys, with consequent threats to data quality, and by the
increasing demand for more and faster information.”
*added by the presenter
-Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy (2017)
Page 3
RICS is a Potential Solution
Redirected Inbound Call Sampling (RICS)
Low cost
– About one-tenth the cost compared to outbound telephone sampling
Quick
– Can collect data on thousands of respondents a day
Low burden
– The respondent is already on the phone
Page 4
Introducing Redirected Inbound Call Sampling (RICS) Surveys
RICS survey participants come from:
Calls to nonworking toll-free numbers (e.g., Area Code: 800, 888)
Direct inward dialing (toll) calls that fail to connect to their intendent target
Page 5
RICS Data Collection Methods
VoiceKeypad
RICS Data Collection
Methods
Interactive
Voice
Response
WebLive
Interviewer
Page 6
Redirecting Inbound Toll-free Calls is Commonplace
Redirecting inbound calls is
commonplace
9% of toll-free number are redirected
5.4 million toll-free telephone numbers
are redirected.
Area
code
n Redirected
(%)
800 126 29.4
844 126 0.8
855 126 0.0
866 126 7.9
877 126 7.1
888 126 8.7
Total 756 9.0 (7.0, 11.0)
Innovation is applying RICS to population surveillance.
Page 7
RICS Surveys Fielded (and in the field) by Presenter
Date SurveyFunding
organizationRespondents
September 2015 BRFSS Evaluation Study RTI 6,799
October 2016 National Adult Tobacco Survey NYS-DOH 4,302
March 2017 NHIS Evaluation Study RTI 9,478
April 2017 New York City Sleep Study NYC-DHMH 1,532
August 2017 National Adult Tobacco Survey NYS-DOH 4,599
February 2018 National Adult Tobacco Study NYS-DOH 4,091
August 2018 RICS-to-Web pilot FL-DOH 500
Date SurveyFunding
organizationRespondents
March 2019 Informed consent user experience
survey
RTI 6,000
March 2019 National Adult Tobacco Survey NYS-DOH 4,000
Completed studies
Studies in progress
Studies in red are evaluation studies.
Studies in black are commissioned by government clients.
Page 8
Future RICS Surveys by Presenter
Future studies
Date SurveyFunding
organizationRespondents
2019 NY State Tribal Cigarette sales NYS-DOH 14,000
2019 May---2024
(quarterly)
NY- Adult Tobacco Survey RICS-to-CATI Pilot NYS-DOH 1,500 per quarter
(for 5 years)
2019--2024 National Adult Tobacco Survey RICS-to-CATI* NYS-DOH 4,000 per year
(for 5 years)
2019--2024 California Tobacco Control Program—Media
Evaluation Study
CA-DOH 700 per month
(for 5 years)
Page 9
NHIS Evaluation Study
The results of the NHIS evaluation study have been accepted for publication
in the AAPOR journal Public Opinion Quarterly
We created a 27-question instrument that mimics questions from the
National Health Interview Survey (NHIS) - Adult sample
We recruited inbound callers to complete the interview via interactive voice
response (IVR) system.
Issue: we fielded continuous questions as categorical because of high item
nonresponse.
Page 10
NHIS Evaluation Study—Data collection metrics
Data collection for two separate one-week periods in 2017
Inbound calls Respondents
AAPOR4
response rate
137,840 9,478 7.7%
Median interview length:
Page 11
NHIS Evaluation Study— Comparison of demographic distributions among: population, RICS respondents and BRFSS respondents
RICS UWE : 1.25 /1.0 = 1.25
BRFSS UWE : 4.1 / 3.2 = 1.28
Page 12
NHIS Evaluation Study—Night owls
We define night owls as individuals who respond
to the survey between 9pm and 9am.
31% of respondents are night owls
Page 13
NHIS Evaluation Study—Night owls (continued)
On average, compared to day-timers, night owls
are:
• More male
• Younger
• Less White-NH
• Lower educated
These are the groups that are underrepresented
in outbound telephone surveys.
Page 14
NHIS Evaluation Study— Calculating Sampling Weights
Demographic distributions for calibration: Sex (2-levels)
Age category (6-levels)
Race/ethnicity (5-levels)
Educational attainment (4-levels)
Census division (9-levels)
We created delete-one jackknife weights (30 groups)
Base
Weight
or
Pseudo-
Inclusion
Probabilities
Calibrate to
demographic
distributions
Page 15
NHIS Evaluation Study— Comparing categorical outcomes
Page 16
NHIS Evaluation Study— Comparing continuous outcomes
Page 17
NHIS Evaluation Study—Primacy Effect
If you get sick or have an accident, how worried are you that you
will be able to pay your medical bills?
Order RICS Order 1 RICS Order 2
1 Very worried Not at all worried
2 Somewhat worried Somewhat worried
3 Not at all worried Very worried
Page 18
IVR Design Recommendations
Recommendations based on the IVR evaluation study:
Allow continuous responses
Use global “prefer not to answer” prompt
Add a softball question before the first study question
For the most important questions do not allow the respondent to
“barge-in”
Page 19
Comparison of demographic distributions—Investigate Representativeness
Page 20
Standardized calibration adjustment index (SCAI)
Design weights used in the SCAI in general
Are based on the inverse of the selection probability.
Have not been adjusted for nonresponse nor trimmed.
Sum to the population total.
In RICS the design weights are
Equal for all respondents (i.e., the sampling mechanism mimics a SRS), and
Sum to the population total
Standardized calibration adjustment index (SCAI)
SCAI = 0.5 ∗ 100 ∗σ⋁𝑖 𝐷𝑒𝑠𝑖𝑔𝑛 𝑊𝑇𝑖 − 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠 𝑊𝑇𝑖
σ⋁𝑖 𝐴𝑛𝑎𝑙𝑦𝑠𝑖𝑠 𝑊𝑇𝑖
Page 21
SCAI—Interpretation
If there was only one calibration category with two levels (i.e., sex) the SCAI is the absolute
value of the difference in the percentage of one category, using the design weight, and the
population.
Percent male
SCAIPopulation
(%)Design
weights (%)
50 30 20
50 40 10
50 50 0
50 60 10
50 70 20
Page 22
Investigate Representativeness with SCAI
Year Study Frame Mode N SCAI
2014 NY-Adult Tobacco Study ABS PAPI 3,899 24.6
2015 NY-Adult Tobacco Study ABS Web/PAPI 2,802 29.8
2016 NY-Adult Tobacco Study ABS Web/PAPI 1,872 25.3
2017 NY-Adult Tobacco Study Telephone CATI 3,286 16.9
2016 National Adult Tobacco Study Telephone CATI 1,582 19.5
2017 National Adult Tobacco Study Telephone CATI 1,690 19.6
2016 National Adult Tobacco Study RICS IVR 4,302 19.0
2017 National Adult Tobacco Study RICS IVR 4,599 20.2
2018 National Adult Tobacco Study RICS IVR 4,060 22.2
Average
SCAI
26.6
18.7
20.5
Take away—The RICS IVR data matched the population better than ABS and
almost as well as outbound telephone.
ABS-address based sample PAPI-paper and pencil interview
CATI-computer assisted telephone interview IVR-interactive voice response
Page 23
Next Frontier of RICS Methodology at RTI—RICS IVR-to-CATI
We piloted collecting New York Adult Tobacco Survey data using
RICS IVR to screen calls
Eligible sample members are redirected to CATI data collection
We obtained 500 CATI respondents using 500 interviewer hours
In comparison outbound telephone sampling takes about 4 interviewer hours
per complete.
Page 24
Good Fits for RICS IVR Methodology
RICS methodology with IVR data collection require:
Short survey instrument (up to 35 questions)
– Break-offs
Simple questions
Target geographies comprised of an area code or groups of area codes
Minimal Screening Criteria
Extensive screening increases costs
Examples of good fits for RICS IVR
Surveillance of population fleeing a natural disaster (Hurricane Harvey)
National or state surveillance of gastroenteritis (stomach flu)
Page 25
Next Steps
Work in progress
Informed consent / user experience study
Evaluate RICS IVR-to-CATI data for representativeness and bias
Develop methodology to optimally design and combine data
collections that use both RICS and outbound CATI
Page 26
Contact Information
Burton Levine
[email protected]
(919) 541-1252
26