Lecture 09: Internal Validity and Experiments. Outline Causal Relations Internal Validity Threats to Internal Validity Video Games and Violence Designs.

Lecture 09: Internal Validity and Experiments

Outline

Causal Relations Internal Validity Threats to Internal Validity Video Games and Violence Designs

To Show Cause and Effect (J. S. Mill)

X must precede Y in Time X must covary with Y No other explanation besides X causes Y is

plausible. “Typically, we infer from an effect to a cause

by eliminating other possible causes.” (Mackie, 1974, p. 67)

Experiments (Shadish et al., 2002)

Definition: A study in which an intervention is deliberately introduced to observe its effects.

Randomized Experiment: An experiment in which units are assigned to received the treatment or alternative condition by a random process (e.g., coin flip, table of random numbers)

Design 1: Randomized Two-Group Design

Pool of Participants

Treatment

Control

Outcome (DV)

Outcome (DV)

Internal Validity: Concerns the validity of inferences

about whether an association between two variables is causal.

Shadish et al. (2002, p. 508)

Classic Ideas

Campbell (1957) defined internal validity as this question: “Did the experimental stimulus make some significant difference in this specific instance?”

Later, Cook and Campbell (1979) argued the internal validity referred to whether observed associations result from a causal relation.

Working Definition of Internal Validity (Source: Shadish, Cook, & Campbell, 2002, p. 38)

Internal Validity refers to inferences about whether the observed covariation between X (the presumed treatment) and Y (the presumed outcome) reflects a causal relation from X to Y.

Threats to internal validity are the other possible causes of the X and Y association.

Validity is not a property of a method, it is a characteristic of inferences.

Threats to Internal Validity

Remember definitions of independent and dependent variables.

Selection

Preexisting differences between individuals in the different conditions that may influence the dependent variable.

When is selection a particular problem?

Maturation

Naturally occurring processes within persons that could cause a change in their behavior.

Setting: You want to evaluate whether or not a parenting program promoted knowledge of child development and parenting efficacy.

Pretest – Intervention – Posttest Design Also known as the O1 – Intervention – O2

design (Campbell & Stanley, 1966)

History

An event that coincides with the independent variable that could affect the dependent variable. Often we think of major historical events in the social, economic, or political lives of people.

Setting: A study about attitudes toward capital punishment occurs right after a well-publicized and brutal murder has occurred.

A big problem if experimental conditions are run at different times.

Instrumentation

Changes that occur over time in measurement procedures or devices.

Basic Rule: Never switch measures in the middle of a study. Willett, Singer, and Martin (1998): “the time for instrument modification is during pilot work, not data collection” (p. 411).

Still the meaning of measures can change. What if raters get more experienced between pretest

and posttest? What if measures lack invariance?

Attrition (called Mortality in Book)

People leave the study. Attrition is always a threat to external validity. When is it a threat to internal validity?

Differential Attrition: Attrition is different for each experimental condition. Think of this as selection (out of a study) after random assignment.

How can you account for this?

Example: Video Games and Violence(Source: Anderson & Dill, 2000)

Theory: General Affective Aggression Model

Basic Idea: Enactment of Aggression is based on knowledge structures created by social learning processes.

Playing violent video games increases the accessibility of aggressive cognitions by semantic priming. So the primary path is from Exposure to Cognitions – Not from Exposure to Affect or Exposure to Arousal.

How can we test this?

Pretest Video Games

First Step: Select a pair of games that differ in amount of violence NOT arousal and NOT affect.

Measure Blood Pressure and Heart Rate: No differences.

Measure Ratings of Difficulty, Enjoyment, Frustration, and Action Speed: No differences

Measure Excitement: Men thought Wolfenstein was more exciting than Myst. Difference not evident for women.

Wolfenstein 3D versus Myst.

Study Details

210 Undergraduate Participants (49.5% Women) IV: Random Assignment to Video Game Condition. DV: B-data - Competitive Reaction Time Task. Use Duration of White Noise Blasts after the Lose

Trials.– Participants in Violent Condition Gave Longer Blasts than

Participants in Non-Violent Condition (6.81 versus 6.65)

We Could Also Study This Question with a Correlational Design

Study Details

227 Undergraduate Participants (54% Women). Reported playing video games an average of 2.14 hours per week. At least 90% reported that they currently play video games.

Most popular games: Super Mario Brothers, Tetris, Mortal Kombat

Measured` Aggressive Behavior, Aggressive Personality, Time Spent Playing Games, Violent Video Game Exposure, and GPA

Gen AB EVG Time AP GPA

Gender -

Aggressive Behavior

.20 -

Exposure to Violent Games

.43 .46 -

Time Spent .35 .20 .28 -

Aggressive

Personality

.19 .36 .22 .16 -

GPA -.18 -.11 -.08 -.20 -.15 -

Designs

Design 2: Pretest-Posttest Two-Group Design

Treatment

Control

Posttest (DV: Post -

Pre)

Posttest (DV: Post-

Pre)

Pretest

Pretest

Design 3: Solomon Four-Group

Treatment

Control

Posttest

Posttest

Pretest

Pretest

Treatment

Control

“Posttest”

“Posttest”

Design 4: Between-Participants Factorial Design

Z1

Z2

Outcome (DV)

Outcome (DV)

X1

X1

Z1

Z2

Outcome (DV)

Outcome (DV)

X2

X2