Adapting Credentialing Exams for Use in Multiple Languages and Cultures

1

Adapting Credentialing Exams for Use in Multiple Languages and Cultures

Ronald K. HambletonUniversity of

Massachusetts

2

Three Reasons for Three Reasons for Translating and Adapting Translating and Adapting ExamsExams

1. Knowledge and skills of interest are often the same across language groups--exam adaptation ensures consistency of content and structure.

2. Often more efficient to translate and adapt an exam than to build a new exam.

3. Easier to insure exam equivalence and fairness (than construct new exam in a second language).

3

Goals of the Presentation1. To describe several myths

about test adaptation.2. To review steps in the test

adaptation process.3. To present the ITC guidelines

for test translation/adaptation.

4

Test Translation vs. Test Adaptation?“Test Adaptation” is more

descriptive of the process that usually takes place—directions, formats, contexts, etc.

“Test Translation” is sometimes only a small part of the process.

5

Examples of Test Examples of Test AdaptationsAdaptations

Intelligence, Aptitude, and Personality Tests

Quality of Life Surveys TIMSS, OECD/PISA, and More SAT, GED, NAEP, State

Assessments Credentialing Exams

6

Example 1

Out of sight, out of mind(Back translated from French)

invisible, insane

7

Are these words similar in meaning?

Pessimistic -- Sanguine

Example 2 (IEA Study in Example 2 (IEA Study in Reading)Reading)

8

Pessimistic -- SanguineAdapted to

Pessimistic -- Optimistic

9

Alex reads his book for 1 hour and then used a book mark to keep his place. How much longer will it take him to finish the book?

A. ½ hourB. 2 hoursC. 5 hoursD. 10 hours

Example 3 (1995 TIMMS Example 3 (1995 TIMMS Pilot)Pilot)

10

Example 4Parker Pen in Mexico: “It

won’t leak in your pocket and embarrass.”

Became:“It won’t leak in your pocket

and make you pregnant.”

11

Four Common Myths About Four Common Myths About Adapting Credentialing Adapting Credentialing ExamsExams1. Know two languages and

you can be a translator.

12

Selection and Training of Translators

Knowledgeable in the languages Knowledgeable in the cultures Knowledgeable in the subject

matter Knowledgeable in the principles

of test development, item writing, and scoring rubrics, etc.

13

Four Common Myths About Four Common Myths About Adapting Credentialing Adapting Credentialing ExamsExams2. A good translation guarantees validity.

14

Four Common Myths About Four Common Myths About Adapting Credentialing Adapting Credentialing ExamsExams3. Judgmental reviews are

sufficient to identify problems in a exam translation/adaptation.

15

Four Common Myths About Four Common Myths About Adapting Credentialing Adapting Credentialing ExamsExams4. The common strategy of a back-translations design and the use of a bilingual design to compile empirical data is sufficient to justify exam use.

16

8 Steps for Adapting Exams1. Checking exam content

and format equivalence.-content may not be generalizable-item formats are not universally used

17

8 Steps for Adapting Exams2. Deciding on the desirability of

a translation/adaptation.-unsuitability of content and/or the exam format?-technical expertise to produce and validate a new exam?

18

8 Steps for Adapting Exams3. Choosing translators.

-Four qualifications given earlier are important.-Some technical expertise may be especially helpful.

19

8 Steps for Adapting Exams4. Translating and adapting

the credentialing exam.-Should often be more than a single translator.-Several designs and variations available.

20

Judgmental Designs Judgmental Designs for Assessing for Assessing

Suitability of Suitability of a a Translation/AdaptationTranslation/Adaptation

1. Backward Adaptations2. Forward Adaptations

21

Advantages1. Some problems can be identified.2. Researchers do not need to be

bilingual.Disadvantages1. No review of target language

version!

Backward Translation Backward Translation DesignDesign

22

Advantages1. Judgments made directly about

language versions of an exam. Disadvantages1. Bilingual judges make

assessments of an exam for monolingual examinees.

Forward Translation Forward Translation DesignDesign

23

8 Steps for Adapting Exams5. Reviewing the adapted

version of the exam.-Multiple reviewers are often very valuable.-”Smoothing” of language is possible at this step.

24

8 Steps for Adapting Exams6. Conducting a small tryout

of the adapted exam-item analysis, reliability analysis, candidate review, etc.

25

8 Steps for Adapting Exams7. Carrying out a more

ambitious study of the adapted exam (that is, checking for exam validity).-construct equivalence, method equivalence, item equivalence

26

Statistical DesignsStatistical Designs1. Bilingual Designs2.Monolingual Designs

A. Source Language Examinees Take Original and Backward Adapted VersionsB. Monolingual Examinees Take Versions in Their Own Languages

27

Checking for Construct Equivalence:

Factor 1 Factor 2

Item Language A Language B Language A Language B

1 .56 .49 -.07 .14 2 .49 .40 .59 .41 3 .43 .57 .11 -.04 4 .48 .52 .63 .68 5 .54 .46 .38 .45 … … … … … 19 -.03 .22 .49 .44 20 .25 .13 .53 .56 21 .29 .08 .27 -.13 22 .09 .26 .02 .22

28

4

6

8

10

12

14

16

18

4 6 8 10 12 14 16 18

Delta (English)

Del

ta (E

S1)

29

Checking for Item Equivalence: Delta Plots

4

6

8

10

12

14

16

18

4 6 8 10 12 14 16 18

Delta (English)

Del

ta (R

LO)

30

Checking for Item Equivalence: IRT Methods (change in difficulty)

-3 -2 -1 0 1 2 3

Proficiency

Prob

abili

ty o

f C

orre

ct R

espo

nse

.

-1.00

-0.50

0.00

0.50

1.00

-3 -2 -1 0 1 2 3

Proficiency

Freq

uenc

y1.0

0.5

0.0

Language ALanguage B

0

0

2010

10 20

40 5030

30 40 50 60

60

A B

31

IRT Methods (change in discrimination)

-3 -2 -1 0 1 2 3

Proficiency

Prob

abili

ty o

f C

orre

ct R

espo

nse

.

-1.00

-0.50

0.00

0.50

1.00

-3 -2 -1 0 1 2 3

Proficiency

Freq

uenc

y1.0

0.5

0.0

Language ALanguage B

0

0

2010

10 20

40 5030

30 40 50 60

60

A B

32

Ref.Foc.

Figure 1-1 ICCs: No DIF

Ability Score

3210-1-2-3

Pro

babi

lity

1.0.9

.8

.7

.6

.5

.4

.3

.2

.10.0

Ref.Foc.

Figure 1-2 ICCs: DIF in bs

Ability Score

3210-1-2-3

Pro

babi

lity

1.0.9

.8

.7

.6

.5

.4

.3

.2

.10.0

Ref.Foc.

Figure 1-3 ICCs: DIF in as

Ability Score

3210-1-2-3

Pro

babi

lity

1.0.9

.8

.7

.6

.5

.4

.3

.2

.10.0

Ref.Foc.

Figure 1-4 ICCs: DIF in as, bs, and cs

Ability Score

3210-1-2-3

Pro

babi

lity

1.0.9

.8

.7

.6

.5

.4

.3

.2

.10.0

F igu re 1-1 I tem ch aracter istic cu rves for possib le D IF pattern s

33

Checking for Method Equivalence1. Any biases due to directions,

exam format, scoring?2. Preparation of exam

administrators?3. Equivalence of exam taking

skills?

34

8 Steps for Adapting Exams8. Documenting the process.

-Steps in exam adaptation, details, evidence for validity.

35

International Test International Test Commission Guidelines for Commission Guidelines for Adapting Educational and Adapting Educational and Psychological InstrumentsPsychological InstrumentsPrepared ByPrepared ByInternational Test CommissionInternational Test Commissionand and Seven International Seven International OrganizationsOrganizations(EAPA, ETP, IACCP, IAAP, IEA, ILTA, IUPsyS)(EAPA, ETP, IACCP, IAAP, IEA, ILTA, IUPsyS)

36

Instrument Adaptation Instrument Adaptation GuidelineGuideline

A practice which is judged as important for conducting and evaluating the adaptation or parallel development of psychological and education instruments for use in different populations.

37

Organization Organization of the 22 ITC Guidelinesof the 22 ITC Guidelines

Context (2) Instrument Development

and Adaptation (10) Administration (6) Documentation/Score

Interpretations (4)

38

Organization of Each of the Organization of Each of the 22 ITC Guidelines22 ITC Guidelines

Description of the Guideline Rationale/Explanation Steps to Meet the Guideline Common Errors References for Additional

Study

39

D.3Test developers/publishers should provide evidence that the choice of testing techniques, item formats, test conventions, and procedures are familiar to all intended populations.

40

D.8Test developers/publishers

should provide information on the evaluation of validity in all target populations for whom the adapted versions are intended.

41

D.9Test developers/publishers

should provide statistical evidence of the equivalence of questions for all intended populations.

42

Three Main Conclusions 1. Has progress been made in

test adaptation methodology?-Definitely!-Moved from single unqualified

translator and bilingual design to considerably more sophisticated methodologies.

43

Conclusions, cont.-Emergence of ITC guidelines;

methodological advances in judgmental designs and statistical methods (e.g., DIF, SEM);more resources and time being allocated.

44

Conclusions, Cont.2. What lies ahead for the field of

test adaptations?-I’m very optimistic about the future.-Needed is commitment of resources

and time, and implementation of existing methods.

45

Conclusions, Cont.3. What are the most important

areas for improvement?-choose multiple qualified

translators and give them sufficient time

-aggressively implement empirical methods

-build on past experiences and knowledge base

46

For more information about exam translation and adaptation, contact Ron Hambleton at

[email protected]