Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany
Mar 27, 2015
Asking Survey Questionsin Many Tongues
Paper presented at theESS Launch Conference
Brussels, November 25-26, 2003
Janet A Harkness, ZUMAGermany
Overview
• English as survey lingua franca• The ESS translation context• ESS requirements and procedures• Common survey practice
and where the problems lie• The ESS and lessons learned
English as survey lingua franca
• We can all speak English• Source questionnaires often in English• Texts anchored to language• Language-driven challenges (two way)
The ESS Context: Facts and Figures• Round One fielded in 22 countries• Fielded in 20-21 languages, depending on how you count• All languages used as first by 5% and
more
• 5 countries fielded in more than one language (12 languages)
• Most often shared languages (to date): French Italian German.
• 9 countries shared one or more languages
The WP Context: Resources & Researchers
• ESF• EC and Partners• CCT • Scientific Advisory
Board• Participating
countries• Translation
Workpackage
Translation Expert Panel Dr. Hans Hönig,
Dr. Paul Kussmaul,
Beth-Ellen Pennell, MA,
Dr. Alisú Schoua-Glusberg
Key Workpackage Component
TRAPD
A framework for ESS translation and assessment to deal with the complex requirements
Translation framework TRAPD
Translation, Review, Adjudication, Pre-testing and Documentation
5 interrelated processes with protocols involved in producing a final translated version
TRAPD combines translation with assessment
Key Activities for Round One
• Develop framework for ESS translation and assessment: TRAPD
• Overview report on TRAPD
• Guidelines for NCs on translation and assessment procedures, team members ‘ roles.
• Strategies for languages shared across countries
• Documentation framework and guidelines
Additional practical measures
• Basic guidelines on organising translation and assessment, selecting & training translators
• Documentation template for problems, divergences, sharing harmonisation
• Source questionnaire annotated to help translators
• Consultation hotline
Ongoing work--Round One into Round Two
• Probe interviews with (external) translators on the source questionnaire
• Analysis of (external) translators‘ appraisals of selected questions from ESS translations
• Comparisons of selected translations for shared languages
In Sum
Funds, Guidelines, Specifications, Requirements, Illustrations and Examples Templates, Annotations, Hot-line, Presentations and Reports
Ongoing: cognitive-based work on the source questionnaires and translations
Isn’t this an unusual amount of effort?
Aren’t questionnaires easy to translate?
Questionnaires at first glance
• Easy questions, simple wording• Response categories often one or two
words• Simple and repeated instructions• Repetition of lexical and structural
elements throughout • Similar or identical questions in other
surveysAnd the work team often knows English
Questionnaires at second glance
• Questionnaires are complex texts•They look easier than they are• Good questions not easy to construct• Measurement-in-text poses special problems• Cross-disciplinary requirements (e.g., measurement, translation)• Interdisciplinary expertise neededA key problem: resulting challenges under-estimated
A Complex Text Type
• Text-with-blanks genre• Often largely generic • Texts within the text• Low Context- jump from one topic to
another• Cross-national aims encourage general
formulations• Computer applications can technically
complex source formats
One example of not-so-easy features
• Measurement properties lead to ‘surveyspeak’ and ‘scalespeak’
Surveyspeak and Scalespeak are textual traces of measurement(Harkness 1996)
Surveyspeak
Q: All things considered, how satisfied are you with your life as a whole nowadays ? (ESS, B13)
Q: How‘s life?
Q: When did you last visit a dentist?
Would you say that you are //very certain, somewhat certain, neither certain nor uncertain, somewhat uncertain or very uncertain// about the date you just gave?
A:
Conversational Follow-up: Uh-HuhReally ?Are you sure?
Scalespeak -- response categories
How important or unimportant is ----
Extremely ImportantSomewhat importantNeither important/or unimportantSomewhat unimportantExtremely unimportant
Scalespeak
Extremely unimportant (ESS E13)
Somewhat importantSomewhat unimportant
Somewhat difficult
Somewhat ugly
Response scales are difficult to translate
Differences across languages/ cultures in• quantification structures & potential negation
• saliency of concept (satisfaction, justness) • disclosure norms• response styles• numeracy and related notions• scaling norms (1-6; 1-10?)
Known Measurement Effects
vs. Allow/Not allow
Allow/ Forbid
So what about differences such as:
vs.Agree/
Not agree
Agree/ Disagree
Agree/Reject
or
Translations may not be able to match English scale properties
German: ‘Agree’ / ‘Not agree’/ ‘Reject’
English : Agree/ Not agree / Disagree
Despite these complexitiesviews on translationtranslatorsassessment and testingare often pre-theoreticaland procedures followed not grounded in research
Not by chance that the ESS work package focused on Survey translation process procedures
Specifications, briefing, & training materials for translators and for those hiring translators
Survey translation evaluation procedures
Providing access to know-how to implement translation components
These are what the survey community in general still lacks
Thus the ESS effort and activities meet real needs
It is a necessary amount of effort
In addition
• No design costs, only translation and assessment • Core questionnaire designed for replication
• Even ESS translations (in a deluxe version) are a very small part of total national survey costs.
Lessons learned from:
Contact with national teams
Documentation from national teams
Discussions within the Expert Panel and CCT
Cognitive interviews with external translators
Related work in other projects(eg SHARE, EUYOUPART, ISSP)
Some lessons learned
1. Procedures and guidelines crucial Team work Iterative translation and review Sharing and harmonisation
2. More hands-on input would help more On the job training Training opportunities and materials
Lessons learned continued
3. Pretesting –TRApD –more needed on source questionnaires
4. Version documentation is essential – but the Ugly Duckling of the work package. Need to help it have its proper place in a user-friendly form
5. Version harmonization raises scheduling issues. No harmonization discussion raises comparability issues
6. More intensive cross-cultural input neededQuestionnaire contentQuestionnaire formulations
[Raises planning, timing and procedural questions]
Thank you very much for your time and attention