Transcript
icf.com
we areCan We Do This Another Way? Potential Nonprobability Sample Sources for Social and Health Survey
Matt Jans, Davia Moyse, Yang Yang Deng, Ronaldo Iachan, Lee Harding, Kristie Healey, James Dayton - ICFScott Worthge – Quest MindshareLaura O'Campo, Sarah Chung - MFour FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Motivation for Nonprobability MethodsEase of data collection Lower cost than probability surveysQuicker to collectUsed in many scientific fields
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Do They Work? Yes, for simple data collection Fit-for-use surveys and studies
Yes, as supplemental sampling frames Though with impact on effective sample size
Maybe, as replacements for probability surveys Different panels can obtain different results (Kennedy, Mercer, Keeter, et al, 2016) Weighting helps
Two possible methods Geofenced sampling Amazon MTurk
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
About GeofencingSampling Geographical “fence” drawn around a point Respondents with geo-tracking app invited to a
survey when crossing the fence
Measurement Asking questions in the moment of an activity or
just after Could reduce recall bias and make out-of-reach
topics and experiences measurable
FedCASIC 2021 - Virtual
Panel Member Dashboard A Single Question
ICF proprietary and confidential. Do not copy, distribute, or disclose.
About the Amazon Mturk Method
FedCASIC 2021 - Virtual
Sampling US and international samples
– “Workers”
Measurement Questionnaires (“tasks”) programmed in MTurk or third-party software All the usual benefits of electronic/web questionnaires Workers may be more amenable to unconventional measurement tasks than conventional
respondents
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Research Questions1. Can a geofenced sample of grocery, convenience, and home improvement
stores produce demographic and health estimates similar to a gold-standard probability sample health survey?
2. Which estimates?
3. When estimates differ, do they tend to be over- or underestimates, and what is their magnitude?
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
BRFSS Benchmark Pilot Methodology
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
BRFSS Benchmark Pilot Geofenced SamplesSampling MFour’s Survey on the Go® mobile opt-in panel Geofenced grocery, convenience, and home improvement stores
– 50m geofence from store entry Reminders at 1hr, 24hrs, and 36hrs
Questionnaire Demographics Health survey topics (wording taken from BRFSS) Image capture (tobacco, alcohol, or sugar-sweetened beverage display or product; or
anything convenient!)
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Is MFour Surveys on the Go® Representative?
FedCASIC 2021 - Virtual
MFour US
60%
50%
40%
30%
20%
10%
0%Sex
(Female)50+
YearsHispanic/
LatinoEducation
(% College +)Black 35–49
Years25–34Years
18–24 Years
≤17 Years
Race/Ethnicity Age
53%51%
21%18%
14% 13%
35%
48%
41%
27%30%
23%
10%
31%
14%
9%
20% 19%
8%
31%
Marital Status(% Married)
+ + -
-
-
- MFour Underestimates
+ MFour Overestimates
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Benchmarking MethodMFour (geofenced) sample weighted to very similar dimensions as BRFSS
MFour (geofenced) estimate is within 95% CI of BRFSS (RDD) estimate (gold standard)
Three Tiers 1. Tier 1: “Nail on the head” (geofenced w/in BRFSS bounds)
2. Tier 2: “Close but no cigar“ (95% CIs overlap)
3. Tier 3: “Close only counts in horseshoes and hand grenades” (95% CIs do not overlap)
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Initial Results
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Tier 1: “Nail on the head” (Geofenced w/in BRFSS bounds)
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Tier 2: “Close but no cigar“(95% CIs overlap)
It’s a long list…
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Tier 2: “Close but no cigar“(95% CIs overlap)
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Tier 2: “Close but no cigar…Maybe a cigarillo?“(95% CIs overlap)
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Tier 3: “Close only counts in horseshoes and hand grenades” (95% CIs do not overlap)Smoker everydaySmokeless tobacco at least some daysAlcohol: Days/week drank; Avg. drinks/day past month, Average drinks is
binge drinking, Binge drank past monthSugar sweetened bev. Times per day, At least 1 serving juice per day, At
least 1 serving sugar sweetened per dayNo health care due to cost
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
When dissimilar, more “bad health behavior”(over or under as % of BRFSS estimate)
FedCASIC 2021 - Virtual
020406080
100120140160180
Geofenced sample overestimates
Geofenced sample underestimates
Double
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Next StepsExplore other benchmarking methods
When and where did people respond? When…how far after notification Where…at/near point of sampling v. at home Effects on estimate quality (i.e., “Are any of the Rs similar to BRFSS Rs”)
Compare to MTurk sample or other nonprobability sources
Seeking collaborators to provide sample sources
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Thank you!Matt Jans
matt.jans@icf.com Davia Moyse
davia.moyse@icf.comJamie Dayton
james.dayton@icf.comFedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Three Additional Implementation Case Studies?
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Hookah Establishments - Design Target population: Bars and restaurants in New York City that sell hookahGeography: New York CitySample size – 2,500 completes across 75+ NYC Hookah Bars Geofenced sample: ~50 Intercept sample: 2,250 completes Other sample – 200 completes from refusals - self administered cards with study URL provided to intercept
interviewers
Geofencing parameters: Upon panel member exit (3-5 meters from bar entrance) Limit potential for duplicates by turning off geofence when intercept interviews are being conducted Rescind invitation after 24 hours
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Hookah Establishments - LessonsWhat Worked Well? Geofencing is inexpensive Modest changes required between the three survey versions Few false positives Inexpensive project supplement
Challenges? Target audience very small - limited benefits Coordination of intercept interviewer schedule with geofence deactivation
Other Takeaways More effective if recall period exceeded 24 hours
– Geotracking can track geolocation prior to current survey date
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Hurricane Florence - Design Target population: Panel members in path of storm who also visited home improvement store to repair storm-related damage
Geography: Stores in North Carolina, South Carolina, Virginia and Coastal Georgia
Sample size Geofenced sample: 250
Geofencing use: Target households impacted by Hurricane/Tropical Storm Florence
– Determined nature of repairs needed– Insurance status– Financial hardship
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Hurricane Florence - LessonsWhat Worked Well? Existing geofence (used for commercial market research) Geo-targeting look-back to any visit between storm landfall and today Screening for storm-related visit Fast Linkage to pre and post surveys
Challenges? Larger N Not all home improvement locations geofenced Not all repairs completed by DIY
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Littering in NYC - Design Target population: NYC Residents age 18-65Geography: NYC’s 5 BorrowsSample size and response rate Geofenced sample: 75 completes from 20 bus shelters, 2 subway exits/entrances, and 3 poster locations Other sample: ~1,000 NYC residents sampled by ZIP Code
Geofencing use: Capture people likely to have seen ads posted about 9 months prior to survey Sampled people who had passed by anti-littering ads at bus shelters, subway stations, and along boardwalk
during the past year
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Littering in NYC - LessonsWhat Worked Well? Geofencing to target NYC residents who were likely to have seen anti-littering adds
Challenges? Small sample size for this method due to few geofenced points
– Supplementary, not primary sampling method– Had to increase radius of geofence to gain additional sample mid-field
FedCASIC 2021 - Virtual
ICF proprietary and confidential. Do not copy, distribute, or disclose.
Case Study SummaryGeofencing can be used in myriad contexts Primary data collection v. supplement
Most useful when sampling involves known geographic points Need a centroid around which to draw the geofence
Potentially representative Skewing younger (including age-related characteristics like marriage) Likely due to mobile device ownership and use Providing devices to participants could reduce this overrepresentation
FedCASIC 2021 - Virtual
top related