Top Banner
Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany
32

Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Mar 27, 2015

Download

Documents

Morgan Kerr
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Asking Survey Questionsin Many Tongues

Paper presented at theESS Launch Conference

Brussels, November 25-26, 2003

Janet A Harkness, ZUMAGermany

Page 2: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Overview

• English as survey lingua franca• The ESS translation context• ESS requirements and procedures• Common survey practice

and where the problems lie• The ESS and lessons learned

Page 3: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

English as survey lingua franca

• We can all speak English• Source questionnaires often in English• Texts anchored to language• Language-driven challenges (two way)

Page 4: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

The ESS Context: Facts and Figures• Round One fielded in 22 countries• Fielded in 20-21 languages, depending on how you count• All languages used as first by 5% and

more

• 5 countries fielded in more than one language (12 languages)

• Most often shared languages (to date): French Italian German.

• 9 countries shared one or more languages

Page 5: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

The WP Context: Resources & Researchers

• ESF• EC and Partners• CCT • Scientific Advisory

Board• Participating

countries• Translation

Workpackage

Translation Expert Panel Dr. Hans Hönig,

Dr. Paul Kussmaul,

Beth-Ellen Pennell, MA,

Dr. Alisú Schoua-Glusberg

Page 6: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Key Workpackage Component

TRAPD

A framework for ESS translation and assessment to deal with the complex requirements

Page 7: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Translation framework TRAPD

Translation, Review, Adjudication, Pre-testing and Documentation

5 interrelated processes with protocols involved in producing a final translated version

TRAPD combines translation with assessment

Page 8: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Key Activities for Round One

• Develop framework for ESS translation and assessment: TRAPD

• Overview report on TRAPD

• Guidelines for NCs on translation and assessment procedures, team members ‘ roles.

• Strategies for languages shared across countries

• Documentation framework and guidelines

Page 9: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Additional practical measures

• Basic guidelines on organising translation and assessment, selecting & training translators

• Documentation template for problems, divergences, sharing harmonisation

• Source questionnaire annotated to help translators

• Consultation hotline

Page 10: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Ongoing work--Round One into Round Two

• Probe interviews with (external) translators on the source questionnaire

• Analysis of (external) translators‘ appraisals of selected questions from ESS translations

• Comparisons of selected translations for shared languages

Page 11: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

In Sum

Funds, Guidelines, Specifications, Requirements, Illustrations and Examples Templates, Annotations, Hot-line, Presentations and Reports

Ongoing: cognitive-based work on the source questionnaires and translations

Page 12: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Isn’t this an unusual amount of effort?

Aren’t questionnaires easy to translate?

Page 13: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Questionnaires at first glance

• Easy questions, simple wording• Response categories often one or two

words• Simple and repeated instructions• Repetition of lexical and structural

elements throughout • Similar or identical questions in other

surveysAnd the work team often knows English

Page 14: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Questionnaires at second glance

• Questionnaires are complex texts•They look easier than they are• Good questions not easy to construct• Measurement-in-text poses special problems• Cross-disciplinary requirements (e.g., measurement, translation)• Interdisciplinary expertise neededA key problem: resulting challenges under-estimated

Page 15: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

A Complex Text Type

• Text-with-blanks genre• Often largely generic • Texts within the text• Low Context- jump from one topic to

another• Cross-national aims encourage general

formulations• Computer applications can technically

complex source formats

Page 16: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

One example of not-so-easy features

• Measurement properties lead to ‘surveyspeak’ and ‘scalespeak’

Surveyspeak and Scalespeak are textual traces of measurement(Harkness 1996)

Page 17: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Surveyspeak

Q: All things considered, how satisfied are you with your life as a whole nowadays ? (ESS, B13)

Q: How‘s life?

Page 18: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Q: When did you last visit a dentist?

Would you say that you are //very certain, somewhat certain, neither certain nor uncertain, somewhat uncertain or very uncertain// about the date you just gave?

A:

Conversational Follow-up: Uh-HuhReally ?Are you sure?

Page 19: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Scalespeak -- response categories

How important or unimportant is ----

Extremely ImportantSomewhat importantNeither important/or unimportantSomewhat unimportantExtremely unimportant

Page 20: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Scalespeak

Extremely unimportant (ESS E13)

Somewhat importantSomewhat unimportant

Somewhat difficult

Somewhat ugly

Page 21: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Response scales are difficult to translate

Differences across languages/ cultures in• quantification structures & potential negation

• saliency of concept (satisfaction, justness) • disclosure norms• response styles• numeracy and related notions• scaling norms (1-6; 1-10?)

Page 22: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Known Measurement Effects

vs. Allow/Not allow

Allow/ Forbid

Page 23: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

So what about differences such as:

vs.Agree/

Not agree

Agree/ Disagree

Agree/Reject

or

Page 24: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Translations may not be able to match English scale properties

German: ‘Agree’ / ‘Not agree’/ ‘Reject’

English : Agree/ Not agree / Disagree

Page 25: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Despite these complexitiesviews on translationtranslatorsassessment and testingare often pre-theoreticaland procedures followed not grounded in research

Page 26: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Not by chance that the ESS work package focused on Survey translation process procedures

Specifications, briefing, & training materials for translators and for those hiring translators

Survey translation evaluation procedures

Providing access to know-how to implement translation components

Page 27: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

These are what the survey community in general still lacks

Thus the ESS effort and activities meet real needs

It is a necessary amount of effort

Page 28: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

In addition

• No design costs, only translation and assessment • Core questionnaire designed for replication

• Even ESS translations (in a deluxe version) are a very small part of total national survey costs.

Page 29: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Lessons learned from:

Contact with national teams

Documentation from national teams

Discussions within the Expert Panel and CCT

Cognitive interviews with external translators

Related work in other projects(eg SHARE, EUYOUPART, ISSP)

Page 30: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Some lessons learned

1. Procedures and guidelines crucial Team work Iterative translation and review Sharing and harmonisation

2. More hands-on input would help more On the job training Training opportunities and materials

Page 31: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Lessons learned continued

3. Pretesting –TRApD –more needed on source questionnaires

4. Version documentation is essential – but the Ugly Duckling of the work package. Need to help it have its proper place in a user-friendly form

5. Version harmonization raises scheduling issues. No harmonization discussion raises comparability issues

6. More intensive cross-cultural input neededQuestionnaire contentQuestionnaire formulations

[Raises planning, timing and procedural questions]

Page 32: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.

Thank you very much for your time and attention