Carol A. Gotway Crawford, PhD - Home - AAPOR...Carol A. Gotway Crawford, PhD Director, Division of Behavioral Surveillance Centers for Disease Control and Prevention (CDC) May 19,

Carol A. Gotway Crawford, PhDDirector, Division of Behavioral Surveillance

Centers for Disease Control and Prevention (CDC)

May 19, 2013

Joint work with

Catherine Okoro, Satvinder Dhingra, Guixiang Zhao, William Garvin, DBS/CDCMeena Khare, NCHS/CDC

The Triple Constraints of Health andBehavioral Surveys: Cost, Quality, and Time

Background

Internet Opt-in panels:• A promising sampling methodology: fast, flexible, cost effective

• Most often used in market research and political polling

Goal: Assess the feasibility and accuracy forbehavioral health and risk factor surveillance

Pilot Objectives

• Compare sampling methodologies– Sample matching , source blending, and quota

• Assess feasibility and accuracy for public health needs

• Compare Internet panel estimates to other surveys and modes– CDC: BRFSS (phone), NHIS (Face-to-Face), HealthStyles (Mail and Internet)– NIH: Patient-Reported Outcome Measures Information System (PROMIS)– Cooperative Congressional Election Study (CCES)

• Evaluate across a range of parameters:– Cost, geographic granularity, and timeliness

Questionnaire Development

• Survey consists of ~80 questions (20 minutes)

• Questions drawn from:– CDC: BRFSS, NHANES, NHIS

– NIH: PROMIS

– SAMHSA: NSDUH

– ONC: Consumer Survey of Attitudes Toward the Privacy and SecurityAspects of EHR and HIE

– National Partnership for Women and Families (NPWF)

– Cooperative Congressional Election Study (CCES) (NIH-supported)

Surveys, Sample Size & Benchmarking

MSAs n States n National n Follow-upsAtlanta 1,000 Georgia 1,000 DBS National 3,500 DBS Follow-ups

Chicago 1,000 Illinois 1,000 NIH PROMIS (2006) 600 Emory WB PROMIS

Houston 1,000 Texas 1,000 CCES pre-election 1,000 CCES post-election

New York 1,000 New York 1,000 DBS Critical Health Issues

BRFSSSMART

BRFSS BRFSS 2011

NHIS ’10 NHIS 2011

NHIS ’11 Health Styles ’10-M

Health Styles ’12-I

Illinois

0 10 20 30 40 50 60

YouGov '13BRFSS '11

NHIS '11

YouGov '13BRFSS '11

NHIS '11

YouGov '13BRFSS '11

NHIS '11

YouGov '13BRFSS '11

NHIS '11

18-2

526

-34

35-6

4>=

65

Age

(Yea

rs)

0 10 20 30 40 50 60

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

Mal

eFe

mal

e

Sex

0 10 20 30 40 50 60 70 80

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

<HS

=HS

>HS

Educ

atio

n

0 10 20 30 40 50 60 70 80

YouGov '13BRFSS '11

NHIS '11

YouGov '13BRFSS '11

NHIS '11

YouGov '13BRFSS '11

NHIS '11

YouGov '13BRFSS '11

NHIS '11

Whi

te N

HBl

ack

NH

Hisp

anic

Oth

er

Race

/Eth

nici

ty% (95% CI) % (95% CI)

Illinois

0 10 20 30 40

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

Yes

Yes

Yes

Fair/

poor

Heav

y Dr

inki

ngCu

rren

t Sm

okin

gFl

u Sh

otSR

H

0 10 20 30 40

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

YouGov '13

BRFSS '11

NHIS '11

Yes

BMI≥

30kg

/m2

Yes

No

Disa

bilit

yO

besi

tyDi

abet

esHe

alth

Insu

ranc

e

% (95% CI) % (95% CI)

Preliminary Results

• Great deal of similarity– Results of sample matching comparable to BRFSS and NHIS (within error)– Variation among surveys consistent across states

• Differences can be attributed to:– Coverage effects (sample selection*outcome interaction)– Use of different control totals and weighting methods– Mode effects (Face-to-Face, telephone, Internet)– Question Differences and order effects– Temporal changes (2013 vs. 2011)– Sample size differences– Cross sectional differences

Preliminary Conclusions and Next Steps• Opt-in Internet panels

– 25-40% of the cost of a completed interview compared with other modes– Much less overhead, financially and logistically– Data available within weeks– State-level estimates consistent with other surveys

• Next Steps: Much to do!– Compare results to other surveys with the same mode (CCES and PROMIS)– Compare MSA and National estimates– Compare sampling methodologies (sample matching, quota, source blending)– Conduct longitudinal follow-ups– Assess non-coverage and coverage bias– Assess integration /blending with BRFSS? Or stand alone?

Collaborators

Name Institution AffiliationStephen Ansolabehere, PhD Harvard University CCES

David Cella, MD Northwestern University PROMIS

Steven Gittelman, PhD Mktg. Inc. Mktg. Inc.

Bob Kaplan, PhD NIH OBSSR

Bill Riley, PhD NIH NCI

Doug Rivers, PhD Stanford, YouGov YouGov

Carol A. Gotway Crawford: [email protected]

The findings and conclusions in this report are those of the authors and do not necessarily representthe official position of the Centers for Disease Control and Prevention.

Questions???