Chapter 6 class version

Chapter 6: RESEARCH VALIDITY

9/27/2012

Roadmap

• Research study announcement• Exam 1 update• Reflection Assignment #1• Quick review & Ch. 5 wrap-up• Begin Chapter 6: Research Validity

Reflection Assignment #1

• Due Tuesday (next class)• Hard copy due at beginning of class– STAPLE YOUR BUSINESS. I’m so serious.

• Cover page: Assignment document (rubric for grading)

Quick Review

Random SamplingSimple Random

Stratified Random

Cluster Random

Systematic Sampling

Nonrandom SamplingConvenience

Quota

Purposive

Snowball

Quick Review

• Random selection vs. Random assignment

• Difference = purpose

How do we determine sample size?

• If population <100, measure them all– Special term for this?

• In general, get as big a sample as possible

• Sample size calculator: G*Power

• Depends on lots of factors

RESEARCH VALIDITYChapter 6

MORE Validity—yay!

• Remember: validity has to do with drawing accurate inferences

• So far: validity in the context of measuring variables

• Now: validity in the context of setting up studies

Research Validity

• Refers to the truthfulness of inferences made from a research study

• Think of validity on a continuum rather than all-or-none

• 4 major types of research validity

• Must prioritize

Types of Research Validity

• Statistical Conclusion Validity• Construct Validity• Internal Validity• External Validity

Statistical Conclusion Validity

• Validity with which we can infer that the IV and DV covary– Covary = vary together

• The validity of the inferences we make from our analyses

Stats Refresher: “Statistically Significant”

• p <.05

• What does it mean?– The observed relationship is probably

NOT due to chance alone

• Sometimes our stats are just wrong• Chance• Too little power (sample size)• Type 1 / Type 2 error

Construct Validity

• Refresh: construct = ?

• Validity of the inferences we make about constructs based on how we measure them

• What does this sound like?– The chapter 5 validity topics!

• Which constructs/operationalizations do we need to consider for construct validity?

• All of them!– IV– DV– Population– Setting

How do we assess Construct Validity?

• Content validity

• Criterion validity

• Convergent validity

ACTIVITY

Group Activity: 5-7 minutes• You’re applying for a grant to fund a

research project

• Identify research idea– IV - operationalize– DV - operationalize– Hypothesis

• Explain how you will gather evidence of construct validity in your measurements

Threats to Construct Validity

• Factors that impact how well our operationalizations actually represent constructs

• Pg 171, Table 6.2 – long list of threats

• We will focus on two major ones:– Participant reactivity to the experimental

situation– Experimenter effects

Reactivity to the Experimental Situation

(From the participant angle)

• Participants’ motives and perceptions

• Demand characteristics

• Positive self-presentation

Instruction set #1

We want to see how well you are able to learn the following sets of letters. Letters will appear in groups of 3 to 7, and each letter will appear on the screen for 1 second. Following the presentation of the letters, …

Instruction Set #2

In the following task, you will be presented with groups of letters ranging from 3 to 7 letters. Each letter will appear on the screen for one second. Your task is to…

Experimenter effects

• Researcher actions and characteristics that influence the responses made by the research participant

• Expectancies– Clever Hans

• Attributes– Biosocial– Psychosocial– Situational factors

Clever Hans

I Math

Internal Validity

• The extent to which we can accurately infer that the independent and dependent variables are causally related

Observed Effect (DV)

Independent Variable

Causally Related

Independent Variable

Observed Effect (DV)

Cause must precede effect

Cause and effect are related (covary)

No other explanation is plausible

No other explanation is plausible

Special Considerations

• Extraneous variables

• Confounding extraneous variables

Threats to Internal Validity

• History • Maturation • Instrumentation • Testing • Regression Artifact • Attrition • Selection • Additive and Interactive Effects

History

• Any event occurring after the study begins that could produce the observed outcome

• Differential history: only one group experiences history event

Maturation

• Changes in biological and psychological conditions that occur with the passage of time – Factors within the individual

• Example: Head Start program and achievement over a school year

Instrumentation

• Changes in the assessment/measurement of the dependent variable

• Example: multiple observers and interviewers

Testing

• Changes in a person’s score on the second administration of a test a result of previously having taken the test

• Example: pre-test and post-test on memory task

Regression Artifact

• A.k.a. regression toward the mean

• The tendency for extreme scores to become less extreme on a second assessment

Attrition

• Participant drop-out– Don’t show up for appointment– Decide to discontinue study

• Differential attrition is especially problematic

Selection

• The choice of participants for the various treatment groups based on different criteria – NOT random assignment

Additive & Interactive Effects

• The combined effect of several threats to internal validity