Cognitive Testing of Survey Translations: Does Respondent Language Proficiency Matter? Patricia Goerman and Mikelyn Meyers, U.S. Census Bureau Hyunjoo Park and Mandy Sha, RTI International Alisu Schoua Glusberg, RSS Presented at the 70 th annual conference of the American Association for Public Opinion Research (AAPOR) Hollywood, FL: May 14-17, 2015 1
17
Embed
Cognitive Testing of Survey Translations: Does Respondent ... · Cognitive testing: One-on-one interviews to evaluate whether respondents interpret, comprehend and respond to survey
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Cognitive Testing of Survey
Translations: Does Respondent
Language Proficiency Matter?
Patricia Goerman and Mikelyn Meyers, U.S. Census BureauHyunjoo Park and Mandy Sha, RTI International
Alisu Schoua Glusberg, RSS
Presented at the 70th annual conference of the American Association for Public Opinion Research (AAPOR)
Hollywood, FL: May 14-17, 2015
1
Overview of the Issue
Rule of thumb: survey translations should be tested with monolingual respondents
Assumptions:
Monolinguals are the intended users
Bilinguals maybe more likely to understand “bad” or overly literal translations:
For example: group home (hogar de grupo)
home schooling (enseñanza en el hogar)
2
Why Does it Matter?
More costly and time consuming to restrict testing to only monolingual respondents
Difficulties with recruiting/interviewing monolinguals
Distrust/lack of understanding of purpose
Correlation with lower income status
3
Review of the Literature
Survey Pretesting with monolinguals v. bilinguals -very little empirical research
(Park, et al. 2014)
Monolinguals: Difficulty with cognitive interview process (Park, et al. 2013; Pan et al. 2010; Goerman, 2006)
Differences between monolinguals and bilinguals in cognitive processes
(Bialystok, 2007; Hernandez and Bates 2001; Marian et al. 2009)
Monolingual status may be a correlate of other variables, e.g., educational attainment
(Ridolfo and Schoua Glusberg, 2011)
Important to include bilinguals in testing to account for differences in acculturation and other demographic characteristics
(Willis and Zahnd 2007)
4
Definitions
Cognitive testing: One-on-one interviews to evaluate whether respondents interpret, comprehend and respond to survey questions as intended.
Usability testing: One-on-one interview to study whether online questionnaire can be answered effectively, efficiently and with satisfaction by target respondents (Wang, 2015)
5
2014 Census Test Spanish Internet
Cognitive/Usability Testing
Origin of present study
Quick turnaround project: Test Spanish-language Census 2014 test instrument with 10 Spanish speakers
Recruitment goals Different national origins
Internet experience
Monolingual or Spanish-dominant, bilingual respondents to best test the translation
6
Preliminary Findings/ Impressions
Correlation between bilingual ability, education level and internet/computer experience
Accidental recruitment:
Non-computer literate, monolingual
Computer savvy, fluent English speakers
Issues of mode interacting with other respondent characteristics. How does language proficiency fit in with other considerations?
7
Design of Empirical Research
Decennial Census testing across modes and languages, large ongoing contract
Double the number of respondents so that half are monolingual, half bilingual
Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.