Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health Catherine Simile National Center for Health Statistics.

Operational and Methodological Lessons Learned Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Healthfrom the 2003 Joint Canada/U.S. Survey of Health

Catherine Simile Catherine Simile National Center for Health StatisticsNational Center for Health Statistics

Ed RamaEd RamaStatistics CanadaStatistics Canada

The findings and conclusions in this The findings and conclusions in this presentation are those of the author(s) and presentation are those of the author(s) and do not necessarily represent the views of do not necessarily represent the views of the Centers for Disease Control and the Centers for Disease Control and Prevention.Prevention.

OverviewOverview

BackgroundBackground

Lessons learnedLessons learned Questionnaire DesignQuestionnaire Design Sampling and Data CollectionsSampling and Data Collections Data Processing and ReleaseData Processing and Release

BACKGROUNDBACKGROUND

How It StartedHow It Started

Both NCHS and STC were involved in Both NCHS and STC were involved in international efforts to improve cross international efforts to improve cross national comparisons of health datanational comparisons of health data

NCHS/STC yearly interchanges to discuss NCHS/STC yearly interchanges to discuss common interestscommon interests

Idea for a joint survey launched at the Idea for a joint survey launched at the September 2000 InterchangeSeptember 2000 Interchange

ObjectivesObjectives

Produce highly comparable data on the Produce highly comparable data on the Canadian and American populations Canadian and American populations unaffected by difference in data collection unaffected by difference in data collection methodology on the following core methodology on the following core indicators:indicators: Health careHealth care Functional statusFunctional status Health statusHealth status Risk factorsRisk factors

Objectives (cont.)Objectives (cont.)

Influence content of the each country’s Influence content of the each country’s ongoing, national health surveys to ongoing, national health surveys to enhance comparability and data qualityenhance comparability and data quality

Develop a model for successful Develop a model for successful collaboration towards standardizing collaboration towards standardizing conceptsconcepts

Anticipated ResultsAnticipated Results

Greater understanding of the implications Greater understanding of the implications of survey standardizationof survey standardization

Generation of hypotheses for observed Generation of hypotheses for observed

differences, if anydifferences, if any

OrganizationOrganization

Project Team –Responsible for day to day Project Team –Responsible for day to day operationsoperations

Steering Committee – project oversightSteering Committee – project oversight

All interviews conducted from Statistics All interviews conducted from Statistics Canada’s Regional Offices using RDD and Canada’s Regional Offices using RDD and CATICATI

Survey tasksSurvey tasks Questionnaire design - questions taken from Questionnaire design - questions taken from

ongoing surveys in both countries -the Canadian ongoing surveys in both countries -the Canadian Community Health Survey and the National Community Health Survey and the National Health Interview SurveyHealth Interview Survey

Average duration of questionnaire – 25 minutesAverage duration of questionnaire – 25 minutes

Editing specifications specific to questionsEditing specifications specific to questions

Interviews conducted in English, Spanish (US), Interviews conducted in English, Spanish (US), and French (Canada)and French (Canada)

Population CoveredPopulation Covered

Residents of Canada and the US aged 18 Residents of Canada and the US aged 18 and older living in private dwellings with and older living in private dwellings with telephonestelephones

Canadian samples stratified by provinceCanadian samples stratified by province US samples stratified by 4 regionsUS samples stratified by 4 regions Sample designed to produce reliable Sample designed to produce reliable

national estimates for 3 age groups (18-national estimates for 3 age groups (18-44, 45-64, 65 and over) by gender44, 45-64, 65 and over) by gender

Sample SizeSample Size

Canada: 3,505Canada: 3,505

US: 5,183US: 5,183

TimelineTimeline

2001 – Questionnaire design and 2001 – Questionnaire design and Cognitive testingCognitive testing

June 2002 – Final content decidedJune 2002 – Final content decided July – Oct. 2002 – Design and testing of July – Oct. 2002 – Design and testing of

collection applicationscollection applications Oct. 2002 – Interviewer trainingOct. 2002 – Interviewer training Nov. 4, 2002 – Data collection startsNov. 4, 2002 – Data collection starts

Timeline (cont.)Timeline (cont.)

Mid-November 2002 – U.S. sample Mid-November 2002 – U.S. sample introduction letters sent introduction letters sent

February 2003 – U.S. sample re-screened February 2003 – U.S. sample re-screened to try to identify out-of-scopes among the to try to identify out-of-scopes among the ring no answersring no answers

March 2003 – Conversion letter sentMarch 2003 – Conversion letter sent

Timeline (cont.) Timeline (cont.)

March 31, 2003 – End of collection in CanadaMarch 31, 2003 – End of collection in Canada April 2003 - Data collection in US from Toronto April 2003 - Data collection in US from Toronto

office extended targeting specific casesoffice extended targeting specific cases follow-up letters sent to answering machines and ring follow-up letters sent to answering machines and ring

no answerno answer June 2003 June 2003

collection resumed in Vancouver to convert refusal collection resumed in Vancouver to convert refusal casescases

July 14, 2003 - collection officially endedJuly 14, 2003 - collection officially ended September – official response rates calculatedSeptember – official response rates calculated

LESSONS LESSONS LEARNEDLEARNED

Cognitive Testing: Window of Cognitive Testing: Window of Opportunity Opportunity

Normal practice:Normal practice:• U.S.: individual one-on-one English interviews in U.S.: individual one-on-one English interviews in

Washington D. C. agency officeWashington D. C. agency office• Canada: focus groups in English in Ottawa and in Canada: focus groups in English in Ottawa and in

French in MontrealFrench in Montreal

JCUSH:JCUSH:• Combined approach: one-on-one interviews and Combined approach: one-on-one interviews and

focus groups for both U.S. and Canada in agency focus groups for both U.S. and Canada in agency labs and off-sitelabs and off-site

Cognitive Testing, (cont’d)Cognitive Testing, (cont’d)

Forced us to think about comparability Forced us to think about comparability differently: More difference between differently: More difference between subsamples in country than between subsamples in country than between countriescountries

Led to a new way of doing cognitive testing in Led to a new way of doing cognitive testing in U.S.U.S.

Translation: Implementation of Translation: Implementation of New GuidelinesNew Guidelines

Languages used in this surveyLanguages used in this survey English (both countries)English (both countries) French (Canada only, required by law)French (Canada only, required by law) Spanish (U.S. only, customary)Spanish (U.S. only, customary)

Steps from the GuidelinesSteps from the Guidelines::

1.1. Pre-translation preparation--NOPre-translation preparation--NO2.2. Selection of contractor--YESSelection of contractor--YES3.3. Completion of translation from final text--NOCompletion of translation from final text--NO4.4. Review of translation—YESReview of translation—YES

• Bilingual review used survey and topic experts, U. S. Bilingual review used survey and topic experts, U. S. Census interviewers, translators, and French speaker from Census interviewers, translators, and French speaker from CanadaCanada

5.5. Adjudication--YESAdjudication--YES6.6. Development of survey instrument --YESDevelopment of survey instrument --YES7.7. Pretest of survey, including translators--NOPretest of survey, including translators--NO8.8. Selection of interviewers YESSelection of interviewers YES9.9. Training of Interviewers--NOTraining of Interviewers--NO10.10. Incorporation of feedback from the field--YESIncorporation of feedback from the field--YES

Data Collection: Plan for Data Collection: Plan for AssumptionsAssumptions

““Clean sample” and resolving cases Clean sample” and resolving cases

Cooperativeness of Canadian and U.S. Cooperativeness of Canadian and U.S. population differentpopulation different

Differences in implementing legal Differences in implementing legal requirements impacted discretion and requirements impacted discretion and authority of data collection staff authority of data collection staff

DATA COLLECTION RESULTS: Resolution, DATA COLLECTION RESULTS: Resolution, Cooperation and Final Response Rates, JCUSHCooperation and Final Response Rates, JCUSH

80

50

69.3 69.362.7

100

0

10

20

30

40

50

60

70

80

90

100

Resolution Cooperation Response

United States Canada

Resolving CasesResolving Cases

Sampling methodologist for each country Sampling methodologist for each country determined number of telephone lines necessary determined number of telephone lines necessary to reach intended sample sizesto reach intended sample sizes

# of lines selected# of lines selected Targeted Targeted

SampleSample sizesize

United StatesUnited States 32,00932,009 5,0005,000

CanadaCanada 10,33410,334 3,5003,500

Resolving Cases (Cont’d)Resolving Cases (Cont’d)

Definition of clean sample differentDefinition of clean sample different GENYSIS removed 1/3 of original U.S. GENYSIS removed 1/3 of original U.S.

samplesample The remaining 2/3 sent to Statistics Canada The remaining 2/3 sent to Statistics Canada

assumed to be “clean”assumed to be “clean”

Working residential numbers cannot be Working residential numbers cannot be verified in the U.S. like they can in Canadaverified in the U.S. like they can in Canada

Cooperation: ResolutionCooperation: Resolution

Evident in 20% U.S. unresolved cases:Evident in 20% U.S. unresolved cases:

• Ring no answerRing no answer 31%31%

• Unresolved answering machineUnresolved answering machine 9% 9%

• Hang up during introduction (likely)Hang up during introduction (likely) 8% 8%

Cooperation: RefusalsCooperation: Refusals

Percentage of all eligible cases:Percentage of all eligible cases:

U.S.U.S. CanadaCanada• RefusalsRefusals 21% 21% 14% 14%• BreakoffsBreakoffs 11% <.01% 11% <.01%

TOTALTOTAL 33%33% 14% 14%

Collection MonitoringCollection Monitoring Not necessary to monitor unresolved cases where Not necessary to monitor unresolved cases where

100% of the cases can be relatively easily resolved100% of the cases can be relatively easily resolved

The monitoring system used works well for a Canadian The monitoring system used works well for a Canadian sample, but not for a U.S. sample where much of sample, but not for a U.S. sample where much of resolution work is done by field staff resolution work is done by field staff

Throughout the data collection period, the staff was Throughout the data collection period, the staff was always uncertain as to what was happening, and so had always uncertain as to what was happening, and so had difficulty ascertaining how to allocate resourcesdifficulty ascertaining how to allocate resources

Did not help that U.S. staff could not easily travel to Did not help that U.S. staff could not easily travel to CanadaCanada

Discretion and Authority of Data Discretion and Authority of Data Collection StaffCollection Staff

Restrictions and delays to convert non Restrictions and delays to convert non response (all communications approved response (all communications approved by NCHS Institutional Review Board)by NCHS Institutional Review Board)

Data ReleaseData Release

Followed Statistics Canada’s usual Followed Statistics Canada’s usual practice. Preparation for release included practice. Preparation for release included both:both:

• editing and review of microdata for public releaseediting and review of microdata for public release• collaborative analytical report released at the same collaborative analytical report released at the same

timetime

Collaborative analysis hampered by legal Collaborative analysis hampered by legal restrictions on data accessrestrictions on data access

SummarySummary

Windows of Opportunity Provided Windows of Opportunity Provided New ways of doing cognitive testingNew ways of doing cognitive testing First opportunity to implement new translation guidelinesFirst opportunity to implement new translation guidelines

Communication and Clarifying Assumptions CrucialCommunication and Clarifying Assumptions Crucial ““Clean sample” and resolving casesClean sample” and resolving cases Differences in cooperativeness of Canadian and US population Differences in cooperativeness of Canadian and US population

require different monitoring and collection strategiesrequire different monitoring and collection strategies

Difference in implementing legal requirements not Difference in implementing legal requirements not insignificant in their impactsinsignificant in their impacts

impacted the discretion and authority of interviewersimpacted the discretion and authority of interviewers difficult to collaborate in analysisdifficult to collaborate in analysis

Dream lofty dreams, and as you dream, so Dream lofty dreams, and as you dream, so shall you become.shall you become.

James AllenJames Allen

Failure is the opportunity to begin again more Failure is the opportunity to begin again more intelligently.intelligently.

Henry Ford Henry Ford 1863-1947, American Industrialist1863-1947, American Industrialist

Access Data and ReportsAccess Data and Reports

NCHS website at NCHS website at www.cdc.gov/nchs/nhis.htmwww.cdc.gov/nchs/nhis.htm

Statistics Canada website at Statistics Canada website at www.statcan.cawww.statcan.ca

Contact InformationContact Information

Catherine M. Simile, Ph.D.Catherine M. Simile, Ph.D.National Center for Health StatisticsNational Center for Health StatisticsDivision of Health Interview StatisticsDivision of Health Interview Statistics

3311 Toledo Road, Room 21153311 Toledo Road, Room 2115Hyattsville, MD 20782Hyattsville, MD 20782

Phone: (301) 458-4499Phone: (301) 458-4499Email: [email protected]: [email protected]