Cross-Cultural Measurement of Protective Factors · Cross-Cultural Measurement of Protective Factors ... –Mixed methods ... rephrasing may threaten comparability of scores)

Cross-Cultural Measurement of Protective Factors

Fons J. R. van de Vijver

www.fonsvandevijver.org

Background

• How can we assess protective factors from three domains (Zautra, Hall, & Murray, 2010): (1) individual characteristics, such as intelligence and personality; (2) characteristics of families, such as family support; (3) the larger social context, such as cultural norms

• Special foci– Mixed methods

• Combination of qualitative and quantitative methods

– Combination of emic and etic approach– Linking conceptualization, methods, and analyses as

procedure to enhance quality management– Test adaptations

Contents

1. Emic? Etic? Emic or etic? Emic and etic?

– There may be light at the end of the tunnel…

– SAPI as an example of combining the two

2. Options in instrument choice in cross-cultural studies

3. Quality enhancement in cross-cultural studies using mixed methods

4. Conclusions

Part 1Emic? Etic? Emic or etic? Emic and

etic?

Two Approaches to Studying Culture

Etic Approaches to Personality

•Five-Factor Model (Costa, McCrae)

–Evidence for invariance of five factors across many

countries

–Neuroticism, Extroversion, (Openness),

Agreeableness, Conscientiousness

Example Inventory (public domain)

Blue difficult to translate

Double barreled

Double barreled

Double barreled

Emic Approaches to Personality

• Chinese concept of interpersonal relatedness

• South-African concept of Ubuntu

– "Humanity towards others"

– "I am because we are"

– "A person is a person because of other persons"

Interim Conclusion

• Evidence for universality of Five-Factor Model

• Evidence for emic perspective

• Yet, only case studies, not accumulating

• “Emic Etic War” disappearing

• Need to integrate etic and emic approaches

SAPI and Cross-Cultural Psychology

• Goals of Cross-Cultural Psychology:

1. To transport and test (hypotheses and findings)

2. To explore other cultures in order to discover psychological variation

3. To integrate findings in order to generate a more inclusive, globally applicable psychology

• Goals ≈ History of cross-cultural psychology

SAPI Project • Aim

– No test available with demonstrated validity in all groups

– Develop a personality instrument for South Africa that can accommodate its diversity

• Stages1. Qualitative Stage

– Identifying personality structure in an indigenous, comparative study

– Generating and piloting items/scales2. Quantitative Stage

Method

•Participants •Adults from all eleven language groups•Sample guidelines: Socio-economic Status, Education (low, medium, High), Males/Females, Age (18-35 yrs & 35 older)

•Interview done in participant’s first language•Instrument

•Person descriptions of familiar persons•Your own personality – Describe yourself •A parent •….

Stages in Data Analysis

• Phase 1– Preparing Data

• Translations

• Removing synonyms, …

• Phase 2 – Iterations of

• Condensing data

• Consulting language and cultural experts

• Phase 3:

– Conceptual Cluster Analysis

14

Condensing the Data

Facets Categorising Cleaning Response in English Response in Ndebele

abusive

(34) abusive abused He is abused Uhlukulumezekile epilweni

abusive abused him

he abused him even when abuse

was not there kade ambuser lokha ingekho

abusive abuses me He abuses me Uyangi hlukumeza

abusive Abusive person An abusive person Umuntu ohlukumezako

Meddlesomeness Conflict SeekingInterpersonal Relatedness

Amiability

Egoism

Empathy

Gratefulness

Approachability

FairnessAchievement Orientation

Thoughtlessness

DedicationSociability

Integrity

Dominance

Epistemic Curiosity

MaterialismEmotional Control

Emotional Sensitivity

Ego Strength

Self-discipline

Active support

EXTRAVERSION

INTEGRITYExpressiveness

Positive Emotionality

Orderliness

Neuroticism

Encouraging others

Guidance

FACILITATING

Broad-mindedness

Openness to Experience

Skillfulness

Aesthetics Reasoning

Social Intellect

SOFT-HEARTEDNESS

Hostility

RELATIONSHIP HARMONY

Courage

Balance

CONSCIENTIOUSNESS

INTELLECT

OPENNESS

EMOTIONAL STABILITY

Subcluster Facet Example Response (Language)

Approachability Accommodating Addressed us in English so we could understand (Xhosa)

ApproachableShe is approachable, I could speak to her about anything (S Sotho)

Arrogant He thinks he is better than all the other people (N Sotho)

Flexible Flexible to situation (Tswana)

Humble She is a humble and down to earth person (Ndebele)

Open for Others Accepts people for who and what they are (English)

Proud Is proud and thinks of herself better than others (Swati)

Stubborn Was stubborn, did not listen to anybody (Tswana)

Tolerant Tolerant (Afrikaans)

Welcoming Welcoming – to everyone (Venda)

Conflict-Seeking Argumentative Likes to quarrel (Xhosa)

Provoking Provocative and calls people names (Swati)

Troublesome Creates tension for nothing (Zulu)

Cluster: Relationship Harmony

Active Support Community Involvement (11/143)There is one person who is always looking after the community (Zulu)

Heedful She listens when you talk to her (S Sotho)

Helpful Is helpful when you are in need (Swati)

Protective Protective (Xhosa)

Solving Problems of Others If I have a problem, she knows how to solve it (Ndebele)

Supportive I like to give people my support (Tswana)

Amiability Friendly She is a friendly person (Tsonga)

Irritating He is annoying and irritating (S Sotho)

Kind Kind (Venda)

Likeable He is loved by everyone (S Sotho)

Pleasant He was a nice person to live with (Zulu)

Stern Always serious, not smiling (Xhosa)

Egoism Generous One who is generous and gives food when asked (Swati)

Greedy Greedy (Afrikaans)

Jealous A person who is jealous of other people’s possessions (Zulu)

Self-Centered All revolves around her, she thinks (English)

Selfish Wants everything for himself (Xhosa)

Cluster: Soft-Heartedness

Subcluster Facet Example Response (Language)

Approachability Accommodating Addressed us in English so we could understand (Xhosa)

ApproachableShe is approachable, I could speak to her about anything (S Sotho)

Arrogant He thinks he is better than all the other people (N Sotho)

Flexible Flexible to situation (Tswana)

Humble She is a humble and down to earth person (Ndebele)

Open for Others Accepts people for who and what they are (English)

Proud Is proud and thinks of herself better than others (Swati)

Stubborn Was stubborn, did not listen to anybody (Tswana)

Tolerant Tolerant (Afrikaans)

Welcoming Welcoming – to everyone (Venda)

Conflict-Seeking Argumentative Likes to quarrel (Xhosa)

Provoking Provocative and calls people names (Swati)

Troublesome Creates tension for nothing (Zulu)

Cluster: Relationship Harmony

Active Support Community Involvement (11/143)There is one person who is always looking after the community (Zulu)

Heedful She listens when you talk to her (S Sotho)

Helpful Is helpful when you are in need (Swati)

Protective Protective (Xhosa)

Solving Problems of Others If I have a problem, she knows how to solve it (Ndebele)

Supportive I like to give people my support (Tswana)

Amiability Friendly She is a friendly person (Tsonga)

Irritating He is annoying and irritating (S Sotho)

Kind Kind (Venda)

Likeable He is loved by everyone (S Sotho)

Pleasant He was a nice person to live with (Zulu)

Stern Always serious, not smiling (Xhosa)

Egoism Generous One who is generous and gives food when asked (Swati)

Greedy Greedy (Afrikaans)

Jealous A person who is jealous of other people’s possessions (Zulu)

Self-Centered All revolves around her, she thinks (English)

Selfish Wants everything for himself (Xhosa)

Cluster: Soft-Heartedness

Conclusion

• Initial analyses suggest the presence of both emic and etic factors

– Emic notably in the social-relational domain

• Five Factor Model well replicated for N, E, O, C

• Agreeableness to social-relational cluster

• Integrity also found

Links with Other Instruments

• History of (claimed) culture-specific concepts: amae(Japan), filotimo (Greece)…

• CPAI-2: Expanding the Big Five with Interpersonal Relatedness (IR)

– Relational Orientation

– Social Sensitivity

– Discipline

– Harmony

– Thrift vs. Extravagance

– Traditionalism vs. Modernity

• First identified in a collectivistic context (China), later replicated in other contexts (Asian countries, US)

SAPI, Big Five, CPAI-2

• SAPI items: Input from original responses

• Item criteria– Short, simple, and clear

– Written in 1st person, starting with “I” followed by concrete behaviors, object, and context

– No negations in item stems (e.g. “I do not express my opinion” vs. “I do what others expect without expressing my opinion”)

– Single activity, habit, or preference (terms such as like/dislike were avoided)

– Temporal qualifiers were excluded, e.g., often, always, sometimes, etc.

• Hierarchical factor analysis replicated subclusters

Scales

• SAPI SOCREL: 10 scales, 91 items (mean α = .81 in both groups;

all items paraphrased)

• Facilitating (10 items, e.g., “I give guidance to people in their life decisions”)

• Integrity (11 items, e.g., “I acknowledge my mistakes”)

• Relationship Harmony (10 items, e.g., “I help people live in peace”)

• Active Support (13 items, e.g., “I support others when they need it”)

• Empathy (6 items, e.g., “I consider how others feel”)

• Unreliability (7 items, e.g., “I fail to meet others’ expectations”)

• Harmony Breach (8 items, e.g., “I cause fights”);

• Arrogance (6 items, e.g., “I show that I am better than others”)

• Hostility (10 items, e.g., “I make people feel vulnerable”)

• Egoism (10 items, e.g., “I only think about my own interests”).

• Big Five: Basic Traits Inventory (BTI) developed in SA; 24 facets, 193 items (mean α = .71 B, .80 W)

• CPAI-2 IR: 6 scales, 74 items (mean α = .58 B, .59 W)

– “Usually when I talk with people, I take great care not to offend them”

– “I strongly support the principle that if a family lives in harmony all things will prosper”

Blacks Whites

Males / Females 429 / 613 126 / 311

Mean age in years (SD) 24.33 (4.79) 19.55 (2.23)

Sample

SAPI & Big Five

aTucker’s phi :

.93, .95, .93, .94,

.92, .92, .41.

BTI: Taylor & De

Bruin (2005)

SAPI & Big Five

Part 2Options in Instrument Choice in

Cross-Cultural Studies

What Does “Equivalent” Mean?

• Eusebius Hieronymus (St. Jerome, famous bible

translator from Greek and Hebrew to Latin; ±347—

419/420):

– 2 types of translations: “words” and “meanings” (he

favored the latter)

Options• Adoption (Close “literal” translation)

– Advantage: maintains metric equivalence– Disadvantage: adequacy (too) readily assumed, should be

demonstrated

• Adaptation (changing contents of one or more items so as to increase cultural appropriateness)– Advantage: more flexible, more tailored to the context– Disadvantage: fewer statistical techniques available to

compare scores across cultures

• Assembly (composing a new instrument)– Advantage: very flexible– Disadvantage: almost no comparability maintained

What is the Best Option?

• One type is not intrinsically better or worse than another

• Main question is

NOT

• What is globally the best choice?

BUT

• What is the best choice in a specific case?

Construct equivalence

• Similarity of construct in source and target culture

Cultural equivalence

• Norms about interaction (modes of address)

• “Cultural fact sheet”

Linguistic equivalence

• Translation accuracy: Retention of denotation and connotation

Measurement equivalence

• Retention of psychometric features (response styles)

• Similarity of factors measured by a test and comparability of scores

Four Important Perspectives (Harkness &

Van de Vijver, in preparation):

• A good translation/adaptation combines equivalence perspectives

• What is a good translation/ adaptation?

–A translation or adaptation is good when it combines high levels of construct, cultural, linguistic, and measurement equivalence.

Is There a Best Way to Translate an Instrument?

• Simple items often straightforward to translate – Close translations will do well, various kinds of

equivalence jointly maximized

• More complex items often require choices about which equivalence will be maximized:– Maximizing comparability or cultural

appropriateness ?

• Different perspectives on equivalence often, but not always compatible

• Example: cross-cultural differences in modes of address – Maximizing linguistic equivalence may challenge

cultural appropriateness (e.g., requests may be too direct)

– Maximizing cultural appropriateness may challenge statistical equivalence (e.g., rephrasing may threaten comparability of scores)

Taxonomy of Adaptations

Need for adaptation can be

• Construct-driven

• Culture-driven (communication style)

• Language-driven

• Measurement-driven (familiarity/recognizability)

A Sample of Possible Procedures (after

Harkness, 2003)

• Translation back translation

• Committee approach (forward translations)

• Mixed approaches (e.g., independent forwards)

Translation stage

•Think alouds, focus groups•Feedback from mono- and bilinguals•Comprehension and readability checks

Pretesting stage (qualitative)

• Equivalence and bias analyses (DIF, structural equivalence)

Pretesting or actual administration (quantitative)

Strength and Weakness of Translations Back Translations

Main strengths

• Well accepted quality check; standard procedures well known in scientific community (incl. researchers, grant institutions and journal boards)

• No knowledge of target language required

Main weaknesses

• Capitalizes on linguistic, cultural, and item-writing skills of (usually) a single person

• Can produce stilted language

• Readability and comprehensibility in target language may be problematic

• Study of intelligence among urban slum children in India

• Influence of supplementation of micronutrients

• Source: Malda, Van de Vijver, Srinivasan, Transler (2008): Adapting a Western Cognitive Test for a Non-Western Context: The KABC-II in Bangalore, India

Kaufman Assessment Battery for Children (second edition)

Subtests:

• Atlantis

• Number Recall

• Rover

• Triangles

• Word Order

• Pattern Reasoning

• Story Completion replaced by adaptation of

WISC(/-R/-III) Picture Arrangement

1. Example Construct-Driven

• Problems with the behaviors or attitudes associated with the construct or with communication norms pertaining to these behaviors or attitudes

• Usage of somatic and psychological symptoms in depression inventories

• Differential norms in allowance to express psychological symptoms across cultures

• Patel, Abas, Broadhead, Todd, & Reeler (2001) – In Zimbabwe, multiple somatic complaints such as headaches and

fatigue are the most common presentations of depression. On inquiry, however, most patients freely admit to cognitive and emotionalsymptoms. Many somatic symptoms, especially those related to the heart and the head, are cultural metaphors for fear or grief. Most depressed individuals attribute their symptoms to “thinking too much” (kufungisisa), to a supernatural cause, and to social stressors. Our data confirm the view that although depression in developing countriesoften presents with somatic symptoms, most patients do not attributetheir symptoms to a somatic illness and cannot be said to have “pure” somatisation. This means that it is vital to understand the culture specific terminology used by patients and to assess mood in thosewith multiple somatic complaints.

• Consequence – Common western measures of depression will under-diagnose

depression in Shona speakers.

47

2. Example Culture-Driven

Example: ‘Burglar’ (Picture Arrangement; adapted for use

in low-SES children in Bangalore, India )

Problems: 1. Unclear whether the burglar was getting in

or getting out;

2. Man not recognized as burglar;

3. Window was not recognized (vertically moving windows are uncommon in India)

Malda, Van de Vijver, Srinivasan, Transler (2008): Adapting a Western Cognitive Test for a Non-Western Context: The KABC-II in Bangalore, India

• Example: Do you often feel distressed? • Translation to Dutch:

– “Distressed” does not have an equivalent word in Dutch – Possible solutions

• Composite of different emotions in Dutch; ask for frequency of composite (“how often do you feel X and Y?”). Problem: composite may not be recognizable

• Choose a single emotion that is as close as possible; problem: change of item content if no close match can be found

• Describe the emotion in the item (e.g., vignette); problem: may require a similar description in English original

– Need to check adequacy of chosen solution in statistical analysis – Combination of judgmental and statistical evidence crucial in

instruments that are more difficult to translate/adapt

3. Example of Language-Driven Adaptation

• Language and test content:

– Adaptation of words in subtest Atlantis:

• Kannada nonsense words

(e.g., English ‘Dablee’ Kannada ‘Ribu’)

• Important: number of syllables

– Adaptation of digits in subtest Number Recall

• based on number of syllables (1 in English version; first 2 and then 3 in Kannada version)

50

4. Example of Measurement-Driven Adaptation (Unfamiliarity)

• Kaufman ABC used in Bangalore (Kannada-speaking children)

• Adaptation of words in subtest Word Order based on:

• Unfamiliarity and ambiguity of objects and words

• Number of syllables

Original version

Kannada version

Malda, Van de Vijver, Srinivasan, Transler (in review): Adapting a Western Cognitive Test for a Non-Western Context: The KABC-II in Bangalore, India

Original version

Kannada version

Problem: word for star in Kannada is too long, English word “star” is well known but too short (monosyllabic)

Original version

Problems:

(1) Key was often called ice cream;

(2) English word “key” was often used, which is too short (monosyllabic)

Kannada version

Original version

Problem: original drawing was not easily recognized as house, distinguishing features added

Kannada version

Example: ‘Painting’

Problem: mirror was not recognized

Rover

Test content:• Additional instructions in subtest Rover

• One additional instruction in subtest Pattern Reasoning

• Slight change of subtest composition and item order in subtest Triangles

Sample item Original version

Sample item Indian version

Problem: original sample item was too difficult; this item has been added as actual test item

Part 3Quality Enhancement in Cross-Cultural Studies Using Mixed

Methods

Mixed Methods: The “newest kid on the block”

• Recent interest

• Attempt to overcome QUAL—QUAN dichotomy

Examples of books

• Brewer, J., & Hunter, A. 1989. Multimethod Research: A Synthesis of Styles. Newbury Park: Sage.

• Cook, T. D., & Reichardt, C. S. (Eds.). 1979. Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: Sage.

• Creswell, J. W. 2003. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (2nd Ed).Thousand Oaks, CA: Sage.

• Frechtling, J., & Sharp, L. (Eds.). 1997. User-Friendly Handbook for Mixed Method Evaluations. Washington, DC: National Science Foundation.

• Greene, J. C., & Caracelli, V. J. (Eds.). 1997. Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms (New Directions for Evaluation, No. 74). San Francisco: Jossey-Bass.

• Newman, I., & Benz, C. R. 1998. Qualitative-Quantitative Research Methodology: Exploring the Interactive Continuum. Carbondale, IL: Southern Illinois University Press.

• Tashakkori, A., & Teddlie, C. 1998. Mixed Methodology: Combining the Qualitative and Quantitative Approaches (Applied Social Research Methods, No. 46). Thousand Oaks, CA: Sage.

• Tashakkori, A., & Teddlie, C. (Eds.). 2003. Handbook of Mixed Methods in Social & Behavioral Research.Thousand Oaks, CA: Sage.

From http://personal.bgsu.edu/~earleym/MIXEDMETHODS/resources.htm

More than Just Two Methods…

• Qual and quan have shown their own independent developments– Have paradigmatic properties

• The debate often has ideological undertones – The quantitative paradigm

• Associated with positivism: there is an objective truth that can be known

• Theories and methods are more adequate when they give a better representation of this truth

– The qualitative paradigm • Multiple realities or truths exist • These realities are constructed

Conceptual backgroundQUAL methods

• Background QUAL is heterogeneous, rangingfrom methods that are very close to QUAN topostmodernism and constructivism/socialconstructionism

• Examples: – Critical ethnography, Critical theory, Dialectical research, Discourse

analysis, Ethnomethodology, Ethnography, Focus group, Grounded theory, Hermeneutics, Participatory action research, Phenomenology (science), Semiotics

QUAL in this workshop

• QUAL not as a rebellion, a method for criticizing empiricial research or positivistic approach

• QUAL as a method for collecting information about the world that is replicable, accessible for others

Most common QUAL methods

• Participant observation – appropriate for collecting data on naturally occurring behaviors in

their usual contexts.

• In-depth interviews – optimal for collecting data on individuals’ personal histories,

perspectives, and experiences, particularly when sensitive topics are being explored.

• Focus groups – effective in eliciting data on the cultural norms of a group and in

generating broad overviews of issues of concern to the cultural groups or subgroups represented.

(from Overview Family Health International)

Some Highlights

• QUAL– Various approaches available

– Popular approach: Grounded Theory (Glaser & Strauss)

• is a systematic methodology in the social sciences emphasizing generation of theory from data in the process of conducting research. (Wikipedia)

• Rather than beginning by researching and developing a hypothesis, the first step is data collection, through a variety of methods.

• Theory as output (rather than input)

Grounded Theory: Four Stages of Analysis

Stage Purpose

CodesIdentifying anchors that allow the key points of the data to be gathered

ConceptsCollections of codes of similar content that allows the data to be grouped

CategoriesBroad groups of similar conceptsthat are used to generate a theory

TheoryA collection of explanations that explain the subject of the research

• QUAN

– Well established quality criteria

– Focus on reliability and validity

– Often aimed at hypothesis testing

(Fro

m F

am

ily H

ealth I

nst

itute

)

However…

• Many differences between QUAL and QUANmethods are overrated

– Most QUAN studies have QUAL stages (e.g., instrument design)

• What is needed?

– Discussion of which method is more adequate in which conditions

– How to combine methods (Tashakkori & Teddlie, 2003)?

Strengths and Weaknesses of Qualitative and Quantitative Methods

• Qual methods display their main strength in the context of discovery – E.g., helpful to get information about various

cultural characteristics of an ethnic group we are dealing with for the first time, to build models, and to generate hypotheses.

• Quan methods are particularly strong in context of justification, testing procedures/hypotheses.

• So, there is complementarity

2. Mixed Methods

• Involves the use of both qual and quan in a single study– (also qual—qual and quan—quan possible)

• Definition– the collection or analysis of both quantitative

and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration of the data at one or more stages in the process of research

(Creswell, Plano Clark, Gutmann, & Hanson, 2003, p. 212)

Example• Poverty survey in KwaZulu-Natal (Adato, Lund, &

Mhlongo, 2006) – First stage: a longitudinal quantitative survey – Second stage: a qualitative study (using focus group

discussions and key informant interviews) building on quantitative part

• Covering various aspects of poverty that were not covered by the survey

– the phenomenology of poverty– identifying mechanisms how members of communities cope

with adverse conditions– finding useful information that could be brought back to the

community so that the community would profit from the study.

Taxonomy of Mixed Methods

• Three underlying dichotomies Leech and Onwuegbuzie (2008):

– Level of mixing • Partially mixed versus fully mixed

– Time orientation• Concurrent versus sequential

– Emphasis of approaches• Equal status versus dominant status

3. Some Myths Regarding the Relation between Qualitative and Quantitative

Methods

Myth 1: • Structured data collection methods such as

observations, psychological tests, questionnaires, and (structured) interviews, belong to the realm of quantitative methods, whereas unstructured data collection methods such as archival analysis and semi-structured or unstructured interviews can only be studied using qualitative methods

Myth 2: • There is a close link between the study of

cultural specifics and qualitative methods on the one hand and the study of cross-cultural universals and quantitative methods on the other hand.

Myth 3:

• The use of mixed methods will resolve all problems of qual and quan methods• Not a “third movement” that solves all problems

but “best of both worlds”

Myth 4:• All qualitative and quantitative methods can

be combined• In postmodern tradition the criterion of validity is

abandoned and is replaced by the extent to which other researchers are convinced by the arguments proposed by an author

• Incommensurable with methods proposed here (both qual and quan)

4. Challenges

• Challenge 1: Establishing Quality Criteria

• Sale and Brazil (2004)– A meta-analysis of mixed-method studies – No specific criterion to evaluate such studies

reported– Needed?

• Probably not. – Lincoln and Guba (1986):

• Set of criteria that apply to both qual and quan

Concept Description Quan Qual

Truth value Adequacy of study inferences

Internal validity Credibility

Applicability Specification of the context in which the information was obtained and new contexts in which the same information would hold

External validity Transferability or fittingness

Consistency Extent other researchers or procedures would yield similar outcomes

Reliability Dependability

Neutrality The influence of researcher and his or her ideas on study outcomes

Objectivity Confirmability

• Challenge 2. Triangulation

– Usually combining qual and quan information

• Yet, the concept of triangulation is broader and could also refer to combining qualitative information (or quantitative information, for that matter)

– More work on integration of information needed

Kinds of Combinations

• QUAL—QUAN:– Often transformed to qual—qual or quan—quan

combination

• QUAN—QUAN: – monotrait—multimethod matrices

• QUAL-QUAL:– Procedures less formalized.

– Triangulation of qualitative evidence is a fairly common procedure in ethnographic research.

• Common issue:– Accounts of the same events provided by various

informants or sources can have three relations with each other

– Possible outcomes: Information can be• 1. converging

– usually interpreted as strengthening the quality of the observations and inferences based on them

• 2. complementary when different sources address different aspects of past events

• 3. incompatible– Suggesting inadequacy of at least one of the

sources and makes it impossible to draw any firm conclusions (memory error, dependability of informant, etc. )

Example of Mixed Methods:QUAL + QUAN

Summary

Example of Mixed Methods:QUAN + QUAL

• Benítez Baena, Van de Vijver, & Padilla García (2014): A study of adequacy of adaptations/translations

• Well-being instruments in large-scale surveysinvolving Spain and the Netherlands (English versions as reference version)

Study design

EVS EQLS WVS ESS SHARE

All these studies• Assessed well-being• Used large probability samples• Likert-type of items

Study 1: Quantitative Phase

• Confirmatory Factor Analysis used to compare factor models across the two countries

• Bias found in several items

– More bias in agreement than in frequency format

– Uniform bias related to topic

– Non-uniform bias related to other structural item characteristics, such as use of extreme qualifiers or terms (such as “never” or “strongly”)

Study 2: Experts Information

• We asked experts to compare translations (all combinations were examined)

– English – Spanish

– Spanish – Dutch

– English – Dutch

• Check of translation accuracy

Study 2: Experts Information (independent assessment)

Spanish

Podría decirme cómo de satisfecho está con su

vida familiar?

English

Could you tell me how

satisfied you are with your

family life?

Dutch

Kunt u aangeven hoe

tevreden u bent met uw familieleven?

“The reference to family is problematic because in Dutch there are two words for referring to family members: Gezin (nuclear family) and familie (extended family)… So, Dutch version refers to more

people than the Spanish version”

Study 3: Cognitive Interviews

• General and follow-up probes were included in the protocol for all the items assessed.

• Interview protocols were developed taking expert appraisal evidence

Materials

• Retrospective design: first participants responded to all the scales, and then the general and follow-up probes were applied.

Procedure

• “Pyramid model”: multi-stage approach (Miller et al., 2014)

• Q -Notes: Data Collection and Analysis software for cognitive interviews developed by National Center for Health Statistics (https://wwwn.cdc.gov/qnotes/Login.aspx)

Analyses

Study 3: Cognitive Interviews, InvolvingMonolingual Respondents

• Check of interpretation of questions, key concepts

Integration of Results

• 1. Some participants more inclined to useextreme categories and verbalizations

extremity as personality characteristic

Integration of results: 2. Problems with specific topics

Items Item Stem

ESS_A_3 How satisfied are you with your present standard of living?

EQLS_1 How satisfied you are with your education?

WVS_3 How important it is in your life: Leisure time

WVS_5 How important it is in your life: Work

EconomicIssues

Economic and social aspects

Manners and educational

level

Educationallevel

Hobbies and activitiesoutside

Householdactivities

NecessityFinancialresource

Overall Conclusions• Increasing interest in mixed methods • Beyond the stage of “believers”• There seem to be two main reasons for their less than

enthusiastic reception in the social and behavioral sciences: – Paradigm clash between qualitative and quantitative studies – Mixed methods are not widely known among editorial boards

and reviewers• No reason for reluctance in adopting methods

– Most cross-cultural survey research includes both qual and quan stages (e.g., cognitive pretesting usually qual, survey usually quan)

• We need to move from “if “ to “how”.

Taxonomy of Bias

Type Source

Construct bias Theoretical construct

Method bias Measurement aspects (e.g., sample, test, administration)

Item bias Specific item aspects (e.g., poor translation)

Construct Bias

• Partial non-overlap of behaviors defining construct

– Definition of happiness in individualistic and collectivistic countries?

• Example: Uchida, Norasakkunkit and Kitayama (2004):

Types and Sources of Method Bias

TTyyppee SSoouurrccee

SSaammppllee bbiiaass CCoonnffoouunnddiinngg ssaammppllee ddiiffffeerreenncceess ((ee..gg..,,

eedduuccaattiioonn))

IInnssttrruummeenntt bbiiaass TTeesstt cchhaarraacctteerriissttiiccss ((ee..gg..,, ssccoorriinngg ooff

ooppeenn eenndd rreessppoonnsseess,, rreessppoonnssee sseettss))

AAddmmiinniissttrraattiioonn

bbiiaass PPrroocceedduurraall aassppeeccttss ((ee..gg..,, iinntteerrvviieewweerr

eeffffeeccttss,, llaacckk ooff ssttaannddaarrddiizzaattiioonn ooff

aaddmmiinniissttrraattiioonn))

Method bias tends to have a global influence on cross-

cultural score differences (e.g., increment due to social

desirability)

Item Bias(also known as differential item functioning, DIF)

• Informal description Differences in psychological meaning of stimuli, due to anomalies at item level

• More formal definition:An item of a scale (e.g., measuring anxiety) is said to be biased if persons with the same trait anxiety, but coming from different cultures, are not equally likely to endorse the item.

Example of Biased Item

Types of (un)biased items

(a) Unbiased item

0

1

2

3

4

5

Very low Low Medium High Very high

Score level

Mean

sco

re

Culture A Culture B

(b) Item with uniform bias

0

1

2

3

4

5

Very low Low Medium High Very high

Score level

Mean

sco

re

Culture A Culture B

(c) Item with non-uniform bias

0

1

2

3

4

5

Very low Low Medium High Very high

Score level

Mean

sco

re

Culture A Culture B

(d) Item with both uniform and non-

uniform bias

0

1

2

3

4

5

Very low Low Medium High Very high

Score level

Me

an

sc

ore

Culture A Culture B

Analysis of Variance and Item Bias

• Item behavior examined per item

• We do not test for cultural differences, but we test whether scores are identical for persons from different groups with an equal proficiency

• Note: regression approach quite similar (illustrated later)

Taxonomy of Equivalence

• Refers to level of comparability

• More flexible than identity– Identity as ultimate type of equivalence

• Is related to bias:Highest level of equivalence obtained for bias-free measurement

• Hierarchical concept

Types of EquivalenceThree types:

–1. “Structural” or “functional equivalence”

–2. “Metric equivalence” or “measurement unit equivalence”

–3. “Scalar equivalence” or “full score equivalence”

(a) “Structural” or “Functional Equivalence”

• Measurement of the same traits

• Various statistical tools available, e.g.,– exploratory factor analysis (with target rotation)

– confirmatory factor analysis

– nomological networks (particularly relevant when items/questions are not identical across cultures)

• Qualitative equivalence can be firmly established

(b) “Metric Equivalence”, “Measurement Unit Equivalence”

• Difference in offset of scales of cultural groups, equal measurement units

• Individual differences have a different meaning within and across cultures:

no problems with offset in intra-cultural comparison, offset has to be added in cross-cultural comparison

• Statistical tool: structural equation modeling (confirmatory factor analysis)

(c) “Scalar Equivalence” or “Full Score Equivalence”

• Complete comparability of scores, both within and across cultures; seamless transfer of scores across cultures

• Frequently taken as the aim of cross-cultural research

Comparability and Equivalence Levels

Equivalence Comparability

Structural Underlying construct

Metric Same plus score metric

Scalar Same plus origin of scale

• Many statistical procedures available for testing structural equivalence

• Common approach:

– Apply dimensionality-reduction technique

– Compare underlying dimensions across cultures

– Similarity of underlying dimensions is criterion for similarity of meaning

Most commonly used

• Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA)

• CFA gradually replaces EFA

EFA (from Kline)

EFA

• Conduct factor analysis in each group

• Compare factor loadings

– Target rotations

– Compute factorial agreement

– If sufficiently high agreement: factors identical (underlying constructs invariant)

CFA

Regress Item (Observed) on Test Scores (Latent)

0

2

4

6

Low High

Ite

m S

core

No BiasGroup A

Group B

0

2

4

6

Low High

Ite

m S

core

Nonuniform Bias: SlopeGroup A

Group B

0

2

4

6

Low High

Ite

m S

core

Uniform Bias: InterceptGroup A

Group B

0

2

4

6

Low High

Ite

m S

core

CombinedGroup A

Group B

Main Differences

• CFA: structure defined at beforehand

• CFA: no rotation

• CFA: Fit statistics available

Difference with Exploratory Factor Analyses

• Starts from covariance matrices– Use metric information

• More parameters tested for cross-cultural similarity; examples– Factor loadings

– Factor correlations/covariances

– Error component of latent variables

– Error component of observed variables

• Enables the testing of a hierarchy of models

• Configural invariance: same constellation (pattern), both uniform and non-uniform bias could be present

• Measurement weights: regression weights in the measurement part of the model. In the case of a factor analysis model, these are the "factor loadings".– Metric invariance (no non-uniform bias)

• Measurement intercepts: intercepts in the measurement part of the model– Scalar invariance (no bias at all)

• (Structural residuals: variances and covariances of residual (error) variables in the structural part of the model.

• Measurement residuals: variances and covariances of residual (error) variables in the measurement part of the model)

• EFA less and less popular

• CFA works well when number of cultures and items is small

– Parceling

– Partial measurement invariance

• DIF frequently tested in educational and cognitive tests, much less so in attitude and personality measurement

Comparing Approaches

• Note that these approaches cannot identify construct and method bias

• Important issue in DIF

– In huge samples most or even all items may be biased

– Shift from significance to effect size (e.g., only medium and high flagged as biased)

Example

EFA results

• The CYRM-28 was administered to a purposive sample of 497 youth – concurrent users of multiple services (child welfare,

mental health, juvenile justice, special educational supports and community programs)

– from rural and urban communities of Atlantic Canada participating in the Pathways to Resilience study

– mean age = 16.85 years (SD = 1.87)

– Subsamples• 281 (56.5%) of the participants were male and 220 (44.3%)

participants self-identified as visible minorities.

CFA Results

• Results multigroup CFA

Part 4Conclusions

Conclusion

• Tool kit of cross-cultural research has expanded and will continue to expand

• Cross-cultural studies as balancing between standardization (with the aim of comparability) and contextualization (with the aim of cultural appropriateness)– Many QUAN tools available

• Focus on standardized procedures; yet, more flexible procedures available

– QUAL procedures less standardized, yet important• Documentation of quality assurance in manuscript

• We need to move beyond “Whatever the problem, Method X is the answer” to “What is the best procedure to address a specific problem?”

• Methodological flexibility

– Adjust methods and procedures to the problem at hand