Top Banner
Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity issues from the literature Discussion of validity issues with respect to student projects
35

Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Jan 15, 2016

Download

Documents

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

ValidityLecture Overview

• Overview of the concept• Different types of validity• Threats to validity and strategies for

handling them• Examples of validity issues from the

literature• Discussion of validity issues with respect to

student projects

Page 2: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Validity Descriptors

• “Hypothesis” validity

• “Construct” validity

• “Content” validity

• “Convergent” validity

• “Ecological” validity

• “Internal” validity

• “Statistical conclusion”validity

• “Concurrent” validity

• “External” validity

• “Predictive” validity

• “Criterion-related” validity

• “Discriminant” validity

Page 3: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Taxonomy of Validity

• Validity as it pertains to assessment

• Validity as it pertains to causal inference

• Validity as it pertains to generalization of findings to real-world phenomena

Page 4: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Validity Issues Surrounding Assessment/Measurement

Page 5: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Validity of an Assessment Tool

• Validity represents an overall judgment of the degree to which both empirical evidence and theoretical considerations support the interpretation of the score and the implications for action that this interpretation entails (Cronbach, 1971).

Page 6: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Validity of an Assessment Tool

• Score validation is an empirical evaluation of the meaning and consequences of measurement (Messick, 1989).

Page 7: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Validity Features

• Validity applies to all assessments, including performance/behavioral assessments

• Validity is not a property of the “test” per se, but rather of the meaning of the test score

• Validation is an ongoing process

Page 8: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Features

• Validity is not just a measurement principle, it is a social value that has powerful implications whenever evaluative judgments and decisions are made

Page 9: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of “Assessment” Validity

• Content validity– Degree to which “Test” items adequately

sample the universe of relevant items for a given domain

Page 10: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of “Assessment” Validity

• Criterion-related validity– Degree to which a “Test” score relates to some

relevant external criterion• Concurrent validity

• Predictive validity

Page 11: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of “Assessment” Validity

• Construct validity*– Ongoing, integrated summary of the evidence

supporting the interpretation and utility of “Test” scores

– Combines information from content validity, criterion-related validity, and discriminant/convergent validity

Page 12: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Convergent/Discriminant Construct Validation

• Convergent validity– Empirical evidence demonstrating communality

between the test score and other indicators of the same construct

• Discriminant validity– Empirical evidence demonstrating a lack of

communality with the test score and indicators of a different construct

Page 13: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Assessment Validity

• Construct underrepresentation– Exists when the assessment fails to include

important facets of the construct (i.e., assessment is too narrow)

– Examples?

Page 14: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

INSTRUCTIONS: Circle the one phrase that best represents the extent to which you agree with the item. If any of the items concern something that is not part of your experience (e.g., “it scares me when I feel shaky” for someone who has never trembled or had the “shakes”), answer on the basis of how you think you might feel if you had such an experience. Otherwise, answer all items on the basis of your own experience. 1. It is important to me not to appear nervous. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 2. When I cannot keep my mind on a task, I worry that I might be going crazy. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 3. It scares me when I feel 'shaky' (trembling). VERY LITTLE A LITTLE SOME MUCH VERY MUCH 4. It scares me when I feel faint. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 5. It is important to me to stay in control of my emotions. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 6. It scares me when my heart beats rapidly. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 7. It embarrasses me when my stomach growls. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 8. It scares me when I am nauseous. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 9. When I notice that my heart is beating rapidly, I worry that I might have a heart attack. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 10. It scares me when I become short of breath. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 11. When my stomach is upset, I worry that I might be seriously ill. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 12. It scares me when I am unable to keep my mind on a task. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 13. Other people notice when I feel shaky. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 14. Unusual body sensations scare me. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 15. When I am nervous, I worry that I might be mentally ill. VERY LITTLE A LITTLE SOME MUCH VERY MUCH 16. It scares me when I am nervous. VERY LITTLE A LITTLE SOME MUCH VERY MUCH

Page 15: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Assessment Validity

• Construct-irrelevant variance– Exists when the assessment contains reliable

variance associated with other distinct constructs (i.e., assessment is too broad)

– Examples?

Page 16: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Evidence for Assessment Validity

• Evidence of content relevance and representativeness

• The extent to which test scores are consistent with theoretical predictions

• Evidence examining the extent to which score properties and interpretations generalize to and across groups, settings, and tasks

Page 17: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Evidence for Assessment Validity

• Evidence on the fidelity of the scoring structure to the structure of the construct being tapped

• Evidence from criterion-related studies including convergent and discriminant studies

• Evidence pertaining to the consequential aspect of test use and score interpretation, especially as it relates to issues of bias, and fairness

Page 18: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Strategies for Enhancing Assessment Validity

• Avoid sole reliance on measures that lack validation data (e.g., new author-constructed measures)

• Employ multiple indicators of the focal construct whenever possible

• Employ indicators from more than one assessment modality domain

• Discussion?

Page 19: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Drawing Valid Inferences about Causal Relationships

Page 20: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of Validity Pertinent to Drawing Causal Inferences

• Internal validity– Degree to which causal inferences can be made

between a measured or manipulated variable (i.e. independent variable) and another measured variable (dependent variable)

Page 21: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of Validity Pertinent to Drawing Causal Inferences

• Statistical conclusion validity– Concerned with sources of random error and with the

appropriate use of statistics and statistical tests (as opposed to systematic bias as in the case of internal validity)

Page 22: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of Validity Pertinent to Drawing Causal Inferences

• External validity– Refers to the degree to which the observed

causal relationship is generalizable across persons, settings, and occasions

– Important distinction between generalizing to a specified population (or setting or occasion) vs. generalizing across populations

Page 23: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Types of Validity Pertinent to Drawing Causal Inferences

• Construct validity*– The degree to which causal inferences

concerning one variable’s effect on another can be generalized to examplars of the constructs in question

– In every day practice, this form of validity deals with the issue of “confounds”

Page 24: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Internal Validity

• History

• Maturation

• Testing

• Instrumentation

• Statistical regression

Page 25: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Hypothetical Example of Maturation Threat

05

101520253035404550

Jan Feb March April

ProzacLuvox

Page 26: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Hypothetical Example of Maturation Threat

05

101520253035404550

Jan Feb March April

ProzacLuvox

Page 27: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Hypothetical Example of Maturation Threat

10

15

20

25

30

35

40

45

Jan Feb March April

ProzacLuvoxPlacebo

Page 28: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Example of a“Testing” Threat

0

10

20

30

40

50

60

% w

ho P

an

ick

ed

CBT EST WL

PrePost

Data from Jaimez & Telch (in preparation)

Page 29: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Internal Validity

• Selection

• Mortality

• Ambiguity about the direction of causal influence

Page 30: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Internal Validity

• Diffusion of treatments

• Compensatory equalization of treatments

• Compensatory rivalry by respondents

• Resentful demoralization of respondents

Page 31: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Internal Validity

• Compensatory equalization of treatments

• Compensatory rivalry among participants

• Resentful demoralization

• Mortality

Page 32: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Construct Validity About Cause and Effect

• Construct underrepresentation– Mono-operation bias– Mono-method bias– Confounding constructs and levels of

constructs

Page 33: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to Construct Validity About Cause and Effect

• Construct irrelevancies (i.e., confounds)– Interaction of different treatments– Hypothesis-guessing within experimental

conditions– Evaluation apprehension (demand

characteristics)– Experimenter expectancies– Interaction of testing and treatment

Page 34: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Threats to External Validity

• Interaction of selection and treatment

• Interaction of setting and treatment

• Interaction of history and treatment

Page 35: Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.

Strategies for Enhancing External Validity

• Employ random sampling to obtain a representative sample if time, resources, and feasibility permit

• Employ heterogeneous samples whenever possible

• Conduct analyses to determine whether the causal relationship holds across characteristics of subjects, settings, etc