Steven B. Cohen Fred Rohde and William Yu Agency for Healthcare Research and Quality

Building Wave Response Rates in a Longitudinal Survey: Essential for Nonsampling Error Reduction

or Last In - First Out?

Steven B. CohenSteven B. CohenFred Rohde and William YuFred Rohde and William Yu

Agency for Healthcare Research and QualityAgency for Healthcare Research and Quality

Advancing Advancing Excellence in Excellence in Health CareHealth Care Purpose of DiscussionPurpose of Discussion

Need for essential longitudinal data on health care Need for essential longitudinal data on health care coverage, use and expenditures to inform health care coverage, use and expenditures to inform health care policy and practicepolicy and practice

Description of the Medical Expenditure Panel Survey Description of the Medical Expenditure Panel Survey (MEPS): purpose, longitudinal design and analytical (MEPS): purpose, longitudinal design and analytical capacitycapacity

Focus on field efforts to achieve target response ratesFocus on field efforts to achieve target response rates


Evaluations of the quality of the MEPS Evaluations of the quality of the MEPS nonresponse adjustment strategies nonresponse adjustment strategies

Determination of characteristics for Determination of characteristics for cases cases fielded at end of field period (EOF) & fielded at end of field period (EOF) & conversion of temporary refusals (TNR). conversion of temporary refusals (TNR).

Examine ROI for inclusion of these cases. Examine ROI for inclusion of these cases.


Examine impact on annual and longitudinal Examine impact on annual and longitudinal response rates; completion of self response rates; completion of self administered questionnaires (SAQ)administered questionnaires (SAQ)

Impact on key survey estimates of health Impact on key survey estimates of health insurance coverage and expendituresinsurance coverage and expenditures

Implications of alternative field strategiesImplications of alternative field strategies

Advancing Advancing Excellence in Excellence in Health CareHealth Care

Medical Expenditure Panel Medical Expenditure Panel Survey (MEPS)Survey (MEPS)

Annual Survey of 15,000 households:Annual Survey of 15,000 households: provides national estimates of health care use, expenditures, provides national estimates of health care use, expenditures, insurance coverage, sources of payment, access to care and health insurance coverage, sources of payment, access to care and health care qualitycare quality

Permits studies of:Permits studies of: Distribution of expenditures and sources of paymentDistribution of expenditures and sources of payment Role of demographics, family structure, insuranceRole of demographics, family structure, insurance Expenditures for specific conditionsExpenditures for specific conditions Trends over timeTrends over time


Key Features of MEPS-HCKey Features of MEPS-HC

Survey of U.S. civilian noninstitutionalized populationSurvey of U.S. civilian noninstitutionalized population Sub-sample of respondents to the National Health Interview Sub-sample of respondents to the National Health Interview

Survey (NHIS)Survey (NHIS)– Linkage to NHISLinkage to NHIS

Oversample of minorities and other target groupsOversample of minorities and other target groups Panel Survey – new panel introduced each yearPanel Survey – new panel introduced each year

– Continuous data collection over 2 ½ year periodContinuous data collection over 2 ½ year period– 5 in-person interviews (CAPI) 5 in-person interviews (CAPI) – Data from 1st year of new panel combined with data from Data from 1st year of new panel combined with data from

2nd year of previous panel2nd year of previous panel


MEPS Overlapping PanelsMEPS Overlapping Panels(Panels 8 and 9)(Panels 8 and 9)

MEPS Household Component MEPS Panel 8 2003-

2004

Round 2 Round 3 Round 4 Round 5

Round 1 Round 2 Round 3

MEPS Panel 92004-2005

1/1/2003 1/1/2004

Round 1NHISNHIS20022002

NHISNHIS20032003 Round 4 Round 5


MEPS Household ComponentMEPS Household ComponentSample DesignSample Design

Oversampling of policy relevant domainsOversampling of policy relevant domains19961996 Minorities (Blacks & Hispanics)Minorities (Blacks & Hispanics)19971997 MinoritiesMinorities

Low income Low income Children with activity limitationsChildren with activity limitations Adults with functional limitationsAdults with functional limitations Predicted high expenditure casesPredicted high expenditure cases ElderlyElderly

1998-20011998-2001 MinoritiesMinorities2002+2002+ Minorities, Asians, Low IncomeMinorities, Asians, Low Income

15,000 households; ~35,000 persons15,000 households; ~35,000 persons

Advancing Advancing Excellence in Excellence in Health CareHealth Care Design SpecificationsDesign Specifications

Target Precision Target Precision Specifications for national and Specifications for national and regional estimates; policy regional estimates; policy relevant subgroupsrelevant subgroups

Overall Design effect of 1.6Overall Design effect of 1.6 200 PSU design (Max)200 PSU design (Max) Overall/round specific survey Overall/round specific survey

response rate requirementsresponse rate requirements

Linkage to NHISLinkage to NHIS Multistage designMultistage design Disproportional samplingDisproportional sampling Longitudinal designLongitudinal design Minimize survey cost for Minimize survey cost for

fixed precisionfixed precision

MEPS, 1996–2006: Number of uninsured, under age 65

65.863.962.961.761.962.2 59.1 58.5 61.762.0

50.149.848.147.045.945.742.642.044.5 44.2 43.8

35.834.433.732.031.331.528.731.032.131.6

0

20

40

60

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Any time in year First half of year Full year

Num

ber i

n m

illio

ns

Source: Center for Financing, Access, and Cost Trends, AHRQ, Household Component of the Medical Expenditure Panel Survey, 1996–2005 Full-Year and 1996–2006 Point-in-Time Files

Advancing Advancing Excellence in Excellence in Health CareHealth Care Trends in ConcentrationTrends in Concentration

27

38

55

70

97

28

39

56

70

97

2838

56

69

97

2333

50

65

97

0102030405060708090

100

Top 1% Top 2% Top 5% Top 10% Top 50%Population ranked by expenditures

1977 1987 1996 2005

Per

cent

age

of e

xpen

ditu

res

Source: National Medical Care Expenditure Survey, 1977; National Medical Expenditure Survey, 1987; Medical Expenditure Panel Survey, 1996 and 2005.

Advancing Advancing Excellence in Excellence in Health CareHealth Care MEPS Field ForceMEPS Field Force

Westat is data collection organizationWestat is data collection organization 500 interviewers 500 interviewers Sample allocated in ~ 200 PSUs, spread across all Sample allocated in ~ 200 PSUs, spread across all

50 states50 states extensive training modulesextensive training modules Information on socio-demographic characteristics of Information on socio-demographic characteristics of

households available based on linkage with NHIShouseholds available based on linkage with NHIS Remuneration for survey participationRemuneration for survey participation


Tool Chest of Methods to Maximize Tool Chest of Methods to Maximize Survey ResponseSurvey Response

Recruitment of experienced and bilingual interviewerRecruitment of experienced and bilingual interviewer 10+ days training (including procedures for obtaining signed 10+ days training (including procedures for obtaining signed

consents)consents) Uses of MEPS data as reference materials for interviewersUses of MEPS data as reference materials for interviewers Periodic retraining and special trainings (e.g. methods to Periodic retraining and special trainings (e.g. methods to

improve response rates)improve response rates) Respondent remunerationRespondent remuneration Advance mailings from co-sponsors of surveyAdvance mailings from co-sponsors of survey Monthly planning calendar and MEPS DVDMonthly planning calendar and MEPS DVD Daily emails to interviewers regarding interviewing progressDaily emails to interviewers regarding interviewing progress Multiple contacts for refusal conversionsMultiple contacts for refusal conversions


MEPS Target Response Rates by Round MEPS Target Response Rates by Round and Overalland Overall

Response Rate

NHIS 90%*

Round 1 80%

Round 2 95%

Round 3 96%

Year 1 of Panel 66%

Round 4 97%

Round 5 98%

Pooled Response Rate-Two Panels 65%

*NHIS response rate among households designated for MEPS.Note: Year 1 and the Overall response rate include the NHIS response rate.


Person Level (survey attrition) Person Level (survey attrition) Nonresponse Adjustment CovariatesNonresponse Adjustment Covariates

Factors associated with survey attrition (after R1)Factors associated with survey attrition (after R1)– Indicator for initial refusal to R1 interview Indicator for initial refusal to R1 interview – Family sizeFamily size– AgeAge– MSA, census regionMSA, census region– Marital status (family reference person)Marital status (family reference person)– Race/ethnicityRace/ethnicity– Education of reference personEducation of reference person– Employment statusEmployment status– Health insurance statusHealth insurance status– Total expenditures (in yr 1 for yr 2 adj.)Total expenditures (in yr 1 for yr 2 adj.)– # doctor visits (in yr 1)# doctor visits (in yr 1)– Self reported health statusSelf reported health status


Person Level Adjustments:Person Level Adjustments:Annual EstimatesAnnual Estimates

Each MEPS panel weighted separatelyEach MEPS panel weighted separately Nonresponse adjustment for complete Nonresponse adjustment for complete

nonresponse and for survey attritionnonresponse and for survey attrition Final Poststratification adjustment –CPS Final Poststratification adjustment –CPS

12/31:12/31: age, race/ethnicity, sex, region, MSA age, race/ethnicity, sex, region, MSA

status, status, poverty statuspoverty status


Testing for Panel EffectTesting for Panel Effect


Capacity of MEPS to Produce Comparable Capacity of MEPS to Produce Comparable NHIS Estimates of Health Insurance NHIS Estimates of Health Insurance

CoverageCoverage


Characteristics of Respondents Fielded at Characteristics of Respondents Fielded at End of First Round or Temporary RefusalEnd of First Round or Temporary Refusal

Initial Refusals:Initial Refusals:Higher likelihood:Higher likelihood: MSA residence; Northeast region; white Non-MSA residence; Northeast region; white Non-

Hispanic; elderly; excellent health; some high Hispanic; elderly; excellent health; some high school; family size 2+; Attrite in future waves school; family size 2+; Attrite in future waves of data collection of data collection

End of Field Period CasesEnd of Field Period CasesHigher likelihood:Higher likelihood: Race: Asian or BlackRace: Asian or Black in excellent healthin excellent health Attrite in future waves of data collectionAttrite in future waves of data collection


Testing for Reluctant Response Effect Testing for Reluctant Response Effect on Coverage Estimateson Coverage Estimates

------------------------------------------------------- DF Wald F P-ValueDF Wald F P-Value

--------------------------------------------------------------------------------------------------------------OVERALL MODEL 22 107.98 0.0000OVERALL MODEL 22 107.98 0.0000

JULY INTERVIEW JULY INTERVIEW 1 2.38 0.1244 1 2.38 0.1244TEMP. REFUSALS 1 0.92 0.3393TEMP. REFUSALS 1 0.92 0.3393SEX SEX 1 98.02 <0.0001 1 98.02 <0.0001RACE/ETHNICITY 3 58.42 <0.0001RACE/ETHNICITY 3 58.42 <0.0001MARITAL STATUS 4 16.90 <0.0001MARITAL STATUS 4 16.90 <0.0001EDUCATION 4 10.94 <0.0001EDUCATION 4 10.94 <0.0001POVERTY STATUS 4 43.97 <0.0001POVERTY STATUS 4 43.97 <0.0001MSA STATUS 1 4.34 0.0382MSA STATUS 1 4.34 0.0382INDIVIDUAL INCOME 1 35.52 <0.0001INDIVIDUAL INCOME 1 35.52 <0.0001MEDICAL $ 1 35.79 <0.0001MEDICAL $ 1 35.79 <0.0001---------------------------------------------------------------------------------------------------------------2 * Normalized Log-Likelihood Full Model: 13037.78-2 * Normalized Log-Likelihood Full Model: 13037.78Pseudo RPseudo R2:2: :0.194167 :0.194167

Advancing Advancing Excellence in Excellence in Health CareHealth Care Mean Number of Contacts by MonthMean Number of Contacts by Month

R1 Interview Month

Mea

n N

umbe

r of C

onta

cts

5

10

15

20

< Mar Mar Apr May Jun Jul

Overall


O refusals


1+ refusals

Panel 9Panel 10


Mean Number of Temporary Mean Number of Temporary Refusals by MonthRefusals by Month

R1 Interview Month

Mea

n N

umbe

r of R

efus

als

0

2

4

6

8


Overall


O refusals


1+ refusals

Panel 9Panel 10


Conditional Response Rates by Month Conditional Response Rates by Month of Round 1 Response: Panel 9of Round 1 Response: Panel 9

R1 Interview Month

Res

pons

e R

ate

0.2

0.4

0.6

0.8


Overall


O refusals


1+ refusals

1st-year response rate1st-year SAQ response rate1st- and 2nd-year SAQ response rateLongitudinal response rate


Panel/temp. Panel/temp. refusalrefusal

InterviewInterviewdate date N persons N persons

Mean R1Mean R1contactscontacts

Mean R1Mean R1Temp.Temp.

refusalsrefusals

full year full year Cond.Cond.

responseresponse

LongitudLongitudCond.Cond.

responseresponseOverall P9Overall P9 OverallOverall 18,250 18,250 6.176.17 0.300.30 89.90%89.90% 85.40%85.40%P10P10 18,170 18,170 6.676.67 0.350.35 88.80%88.80% ..P9P9 < Jul< Jul 17,537 17,537 5.645.64 0.190.19 90.30%90.30% 85.90%85.90%P10P10 17,380 17,380 6.166.16 0.230.23 89.30%89.30% ..P9P9 JulJul 713 713 19.2119.21 2.942.94 79.20%79.20% 71.10%71.10%P10P10 790 790 17.7817.78 2.942.94 76.70%76.70% ..0 P90 P9 < Jul< Jul 16,631 16,631 5.425.42 0.000.00 91.00%91.00% 86.80%86.80%0 P100 P10 16,377 16,377 5.905.90 0.000.00 89.90%89.90% ..1+ P91+ P9 JulJul 272 272 17.2617.26 7.717.71 77.20%77.20% 62.50%62.50%1+ P101+ P10 328 328 16.5516.55 7.087.08 73.20%73.20% ..ExclusionsExclusions July July RR reduction RR reduction .041.041 .959.959

June refusalJune refusal July July .062.062 .938.938

MeasureStandard Field

Operation 50% sample 33% sample 25% sample Exclusion

Difficult Cases Difficult CasesDifficult

Cases Difficult Cases

Mean Expenditures 3479.01 3477.90 3478.24 3521.02 3543.21Standard Error 91.03 92.02 92.93 94.17 94.94

Responding sample 31589 30769 30495 30359 29948

July cases+June temp refusals 1641 821 547 411

RSE 0.026 0.026 0.027 0.027 0.027

Bias 0 0 0 0 64.2064.20MSE 8287.10 8467.37 8635.93 8868.86 13135.24RMSE 91.03 92.02 92.93 94.17 114.61Ratio of MSE to

Standard 1 1.022 1.042 1.070 1.586Mean no. hours per

complete 14.2 22 22 22 12.5

Add’l sample to meet precision goal 670 1,284 2,132

Impact on MSE of Mean Medical Expenditure Estimates for alternative field strategies

MeasureStandard Field

Operation 50% sample 33% sample 25% sample Exclusion

Difficult Cases Difficult CasesDifficult

Cases Difficult Cases

ProportionStandard Error

Responding sample

July cases+June temp refusals

RSE

Bias

MSE

RMSE

Ratio of MSE to Standard

Impact on MSE of Proportion with Medical Expenses in excess of $10,000 for alternative field strategies

0.0756 0.0756 0.0757 0.077 0.07740.0021 0.0022 0.0022 0.0023 0.0022

31589 30769 30495 30359 29948

1641 821 547 411

0.028 0.029 0.029 0.030 0.028

0 0 0 0 0.002

0.00000441 0.00000484 0.00000484 0.00000529 8.08E-06

0.0021 0.0022 0.0022 0.0023 0.0028

1 1.098 1.098 1.200 1.832


SummarySummary

Need for accurate and reliable national data on Need for accurate and reliable national data on health insurance coverage to inform policy and health insurance coverage to inform policy and practice practice

MEPS design features and analytical capacityMEPS design features and analytical capacity Statistical, methodological and operational design Statistical, methodological and operational design

features to adjust for nonresponse and attritionfeatures to adjust for nonresponse and attrition Evaluation of estimation strategies to correct for Evaluation of estimation strategies to correct for

survey attrition survey attrition Examination of ROI for inclusion of difficult cases Examination of ROI for inclusion of difficult cases Options identified for more efficient and effective Options identified for more efficient and effective

field strategiesfield strategies

Agency for Healthcare Research and QualityAgency for Healthcare Research and QualityAdvancing Excellence in Health CareAdvancing Excellence in Health Care • • www.ahrq.govwww.ahrq.gov

Steven B. Cohen Fred Rohde and William Yu Agency for Healthcare Research and Quality

Documents

longitudinal survey

health care policy

health care coverage

round 1nhis2002nhis2003round

year of new panel

meps field forcewestat

essential longitudinal

timekey features of