Validity and Item Analysis Chapter 4. Validity Concerns what the instrument measures and how well it does that task Not something an instrument has or.

Validity and Item Analysis

Chapter 4

Validity Concerns what the instrument measures

and how well it does that task

Not something an instrument has or does not have

Informs counselor when it is appropriate to use the instrument and what can be inferred from the results

Reliability is a prerequisite for validity

Movement Away from Traditional Categories of Validity

Traditional types of validity Content-Related

Criterion-Related

Construct-Related

Evidence Based on Instrument Content The degree to which the evidence indicates

the items, questions, or tasks adequately represent the intended behavior domain.

Central focus is typically on how the instrument’s content was determined

Content-related validation evidence should not be confused with face validity

Evidence Based on the Response Processes

Information processing

Examining the response processes used by different groups

Computer simulated performances

Evidence Based on Internal Structure

Examining internal structure using factor analysis

Examining the internal structure of an assessment with groups (e.g., gender, race or ethnicity)

Evidence Based on Relations to Other Variables

Correlational method Convergent evidence/Discriminant evidence Multitrait-multimethod matrix

Prediction or instrument-criterion relationship Concurrent and predictive validity Regression Decision Theory

Validity Generalization

Correlational Method

Correlational method-examine relationship between instrument and other pertinent variables or a criterion

Validity coefficient(s)

ABCScale

Mean ratingson criteria

Tyrone 87 4.5Steve 63 2.4Hannah 75 3.7Jeannie 54 1.6Andy 32 .5Benjamin 44 4.9

Prediction or instrument-criterion relationship

Concurrent validity

Predictive validity

Regression

ABC

.... ..

10 20 30 40 50 60 70 80 90 100

1

2

3

4

5

..

… …

… .

. .

…

.

. .

. .

.

… . .

.

. .

.

..Ratings

Regression Line

ABC

.... ..

10 20 30 40 50 60 70 80 90 100

1

2

3

4

5

..

… …

… .

. .

…

.

. .

. .

.

… . .

.

. .

.

..Ratings

a = intercept b = slopeX = score

Standard Error of Estimate

xySS ryest

21

The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity.

Decision Theory: Expectancy Table

ABC Scale

Ratings

10 20 30 40 50 60 70 80 90 100

5 1 1 4 5

4 1 1 2 1 2 1

3 1 3 2 2 3 3 2

2 2 3 2 1 2

1 3 1 1

Expectancy Table (cont.)

10 20 30 40 50 60 70 80 90 100

5 1 1 4 5

4 1 1 2 1 2 1

3 1 3 2 2 3 3 2

2 2 3 2 1 2

1 3 1 1

Expectancy Table (cont.)

10 20 30 40 50 60 70 80 90 100

5 1 1 4 5

4 1 1 2 1 2 1

3 1 3 2 2 3 3 2

2 2 3 2 1 2

1 3 1 1

Validity Generalization

A method of combining validation studies to determine if the validity evidence can be generalized Must be a substantial number of studies Meta-analysis

Evidence-based consequences of testing

Social consequences of racial/ethnic group differences

Consider social implications when examining the validation evidence

Conclusion on validation evidence

Gradual accumulation of evidence

Counselor must evaluate the information to determine if it appropriate for which client under what circumstance

Validation evidence should also be considered in informal assessments

Item Analysis

Focus is on examining and evaluating each item on an instrument.

Item difficulty:p = # who answered correctly

total number

Item discrimination:d = upper % - lower %(or correlational method)

Item Response Theory or Latent Trait Theory

Focus is on each item and on establishing items that measure the individual’s ability or level of a latent trait.

Involves examining the item characteristic function (or characteristic curve) and the calibration of each individual item.

Parameters: Difficulty Slope Basal level

Item Characteristic Curve

4 5 6 7 8 9

1.00

.50

.00

Grade

%