Degree to which inferences made using data are justified or supported by evidence Some types of validity ◦ Criterion-related ◦ Content ◦ Construct.

Validity

Degree to which inferences made using data are justified or supported by evidence

Some types of validity◦ Criterion-related ◦ Content◦ Construct

All part of unitarian view of validity Constructs - theoretical abstractions aimed

at organizing and making sense of our environment; they are LATENT

Validity

Importance of Criteria A criterion is any variable you wish to

explain and/or predict They are the key to well-developed theory,

good measurement, and strong research design

Ultimate criterion Multidimensional nature of criteria Intermediate criteria

Criterion-Related Validity Process of establishing a relationship

between variables Predictive, concurrent, postdictive Usually based on correlation or

regression equation Low reliability will attenuate or mask

relationships

Selection Ratio – proportion of the individuals in the sample who are selected of the total number

Base rate – percent of successful individuals under random selection

Range Restriction Differential Prediction for different

subgroups

Usefulness of criterion-related validity estimate

DecisionsSR,BR=.50

FN

FP

VP

VN

Xc

YcFN+VP=BRVN+FP=1-BR

VP+FP=SRFN+VN=1-SR

Successful

Unsuccessful

Reject Accept

False Negatives

False Positives

DecisionsSR=.15,BR=.50

FN

FP

VP

VN

Xc

Yc

FN+VP=BRVN+FP=1-BR

VP+FP=SRFN+VN=1-SR

Successful

Unsuccessful

Reject Accept

False Negatives

False Positives

DecisionsSR=.85,BR=.50

FN

FP

VP

VN

Xc

Yc

FN+VP=BRVN+FP=1-BR

VP+FP=SRFN+VN=1-SR

Successful

Unsuccessful

Reject Accept

False Negatives

False Positives

DecisionsSR=.50,BR=.80

FN

FP

VP

VN

Xc

Yc

FN+VP=BRVN+FP=1-BR

VP+FP=SRFN+VN=1-SR

Criterion-Related Validity Even low correlations can lead to large

increases in selection efficiency SR and BR have strong influences When SR is small (choose few), fewer FP and

more FN When SR is large, fewer FN and more FP When BR is large (many can be successful), SR

and validity have little effect on selection efficiency

Most gains in success ratio when BR = .50 and SR is small (e.g., .10)

The tradeoffs depend on purpose of selection

Range Restriction

X

Y

- Direct- Indirect- Ambiguous

Differential PredictionIntercept Bias

X

Y

Same prediction for each group

Differential PredictionSlope Bias

X

Y

Different prediction for each group

Content Validity (Logical Analysis)

Extent to which items or measures cover the content area the test purports to measure◦ Expert judges determine if a measure came from

a particular content domain◦ Scoring and content is based upon theory◦ If measures are from same content domain,

should demonstrate high reliability ◦ If low internal consistency reliability, low content

validity

Construct Validity

Validity of inferences about latent unobserved variables on the basis of observed variables

Does a measure assess what it is intended to assess? Do the variables relate in theoretically meaningful ways?

Low reliability will make it difficult to assess the nature of a particular construct and attenuate relationships with other constructs

Construct ValidityConstruct Validity

Can we generalize to the constructs from the measures?

Theory What you think

CauseConstruct

EffectConstruct

Measure orManipulation

ObservedOutcomes

ObservedRelationship

TrueRelationship

What you see

Construct Validity

Anxiety

Test Score(Y)

Measureof Anxiety

(X)

Abilityto Learn

1

2

3

4

SaladsEaten (Z)

Vegetarianism

5

Internal Structure Analysis Cross Structure Analysis Nomological network (Cronbach & Meehl)

Ways to Establish Construct Validity

Internal Structure Analysis

Factor Analysis◦ Used to identify factors or dimensions that underlie

relations among observed variables Exploratory - Useful When:

◦ No info on internal structure available◦ Factor structures may look different than original scale◦ You have reservations about previous factor analyses

Confirmatory - Useful When:◦ You have some idea of the internal structure◦ Confirming factor structures from previous studies

Necessary but not sufficient to establish construct validity

Internal Structure Analysis

Ability to Learn

Z1 Z2

Z3

Anxiety

X1 X2 X3

X4

e1e3

e4 e5 e6e7e2

Cross-Structure Analysis Embedded in nomological network

(nomological validity) Test of hypotheses by examining

relationships between different indicators of underlying constructs ◦ e.g., leadership style based on reports from

subordinates and leadership self-report inventory

Relies on multiple methods of measurement

A representation of constructs of interest in a study, their observable manifestations (measures), and the interrelationships among and between them

Cronbach & Meehl said this is necessary to establish construct validity

Elements include:◦ Specify linkage between constructs (hypotheses)◦ Operationalize constructs (specify measurement)

Nomological Network

Convergent and Discriminant Validity Convergent validity - Convergence

among different methods designed to measure the same construct

Discriminant validity - Distinctiveness of constructs, demonstrated by divergence of methods designed to measure different constructs

Multi-Trait Multi-Method

Heterotrait-Monomethod◦ Different traits, same method

Heterotrait-Heteromethod◦ Different traits, different methods

Monotrait-Heteromethod◦ Same trait, different methods◦ Validity diagonals

Monotrait-Monomethod◦ Same trait, same method◦ Reliability diagonals

MTMM

Method1 Method2 Method3 A1 B1 C1 A2 B2 C2 A3 B3 C3

M1 A1 (.89) B1 .51 (.89) C1 .38 .37 (.76)

M2 A2 .57 .22 .09 (.93) B2 .22 .57 .10 .68 (.94) C2 .11 .11 .46 .59 .58 (.84)

M3 A3 .56 .22 .11 .67 .42 .33 (.94) B3 .23 .58 .12 .43 .66 .34 .67 (.92) C3 .11 .11 .45 .34 .32 .58 .58 .60 (.85)

MTMM

Steps to Establish Construct Validity Specify the nomological net (expected +

and - relationships) of expected relations Establish reliability Check convergence with other preexisting

measures of the construct (convergent validity)

Factor analysis Empirical studies of relatedness Empirical studies of discriminability

Take the hypotheses you developed in assignment 2 and the variables that were included in them. ◦ Draw a picture of what you believe the nomological

network of these variables would look like◦ What alternative measures of each variable might

you use (different than those specified in Assignment 3) to establish convergent validity?

◦ Draw what an MTMM construct validity chart would look like that includes each variable in your study and the original and alternative measures you identified for each construct. Specify whether each correlation would be expected to be Hi, Low or Moderate.

Assignment 4