Quasi-Experiments – Outline 1. True Experiments a. Characteristics b. Threats to validity controlled by experiments c. Threats not controlled by experiments.

Quasi-Experiments – Outline

1. True Experimentsa. Characteristicsb. Threats to validity controlled by experimentsc. Threats not controlled by experimentsd. Obstacles to true experiments in the field

2. Quasi-experimentsa.The logic of quasi-experiments b.Non-equivalent control group design

• Example – Langer & Rudin (1976)c. Interrupted time-series design

• Example – Campbell (1969)

Quasi

True Experiments - Characteristics

• True experiments are characterized by:• A manipulation• A high degree of control• An appropriate comparison

(the major goal of exerting control)

• Manipulation in the presence of control gives you an appropriate comparison.

Quasi

Threats to validity controlled by true experiments

• History • occurrence of an event other than the treatment

Quasi


• Maturation • participants always change as a function of time. Is change in behavior due to something else?

Quasi


• Testing • improvement due to practice on a test (familiarity with procedure, or with testers expectations)

Quasi


• Instrumentation • especially if humans are used to assess behavior (fatigue, practice)

Quasi


• Regression • when first observation is extreme, next one is likely to be closer to the mean.

Quasi


• Selection • if differences between groups exist from the outset of a study

Quasi


• Mortality • if exit from a study is not random, groups may end up very different

Quasi


• Interactions of selection… • with History• with Maturation• with Instrumentation (ceiling

effects)

Quasi

Note difference between these threats:

• Maturation• One group; performance

better on post-test than on pre-test

• Interaction of Maturation & Selection• Two or more groups• Performance difference

larger on post-test than on pre-test

Quasi

Threats to validity not controlled by experiments

• Contamination• communication of

information about the experiment between groups of subjects

• Cook & Campbell (1979):• resentment• ‘compensatory rivalry’• diffusion of treatment:

control subjects use information given to others to change their own behavior.

Quasi

Contamination – an example

• Craven, Marsh, Debus, & Jayasinghe (2001)

• Journal of Educational Psychology

• Teachers trained to improve students’ academic self-concept through praise

• Internal control• External control

Quasi

Contamination – an example

• Craven, Marsh, Debus, & Jayasinghe (2001)

• Next slide shows T2 (post-test) academic self-concept scores as a function of T1 scores for control children only.

Quasi

External controlInternal control

Internal high focusInternal low focus

1.0

0.5

0.0

-0.5

-1.0Low Medium High T1 acad self concept

Diffusion

No diffusion

T2 acad self concept

Low focus group consistently higher than external control

Resentful demoralization?Overzealous cooperation?


• Threats to external validity • best way to deal with this is replication

Quasi


• Hawthorne effects • changes in a person’s behavior due to being studied rather than the manipulation.

• a special kind of reactivity.

Quasi

Hawthorne effects

• Demand characteristics • cues communicated by researcher

• subject’s under-standing of their role

Quasi

Hawthorne effects

• Role of “research subject” • Is subject behaving the way he thinks a person in that role should behave?

• (E.g., hypnotized person)

Quasi

Hawthorne effects

• Orne (1962) • ‘good subjects’ think they are contributing to science by complying with researcher’s demands

Quasi

Hawthorne effects

• What to do about Hawthorne effects?

• Orne (1962): Use quasi-control subjects as “co-investigators”

• They do your task, reflect on demand characteristics of the experiment.

Quasi

Obstacles to true experiments in the field

• Sometimes, we cannot bring the phenomenon we want to study into the lab, so we have to work in the field.

• Can we do experiments in the field?

Quasi


• Can’t get permission from individuals in authority?

• Your study may involve some time and effort on their part. But what’s in it for them?

• In schools, parents also have to agree.

Quasi


• Can’t assign subjects to groups randomly?

• have to work with intact groups (e.g., classes in a school)

Quasi

Quasi-Experiments

• Quasi-experiments resemble true experiments… • usually include a

manipulation, and provide a comparison.

• …but they are not true experiments.• lack high degree of control

that is characteristic of true experiments.

Quasi

Quasi-Experiments

• Quasi-Experiments are compromises

• They allow the researcher some control when full control is not possible.

Quasi

Quasi-Experiments

• Because full control is not possible, there may be several “rival hypotheses” competing as accounts of any change in behavior observed.

• How do we convince others that our hypothesis is the right one?

Quasi

The Logic of Quasi-Experiments

• Eliminate any threats you can• Show how each threat to

validity on list given above is dealt with in your study.

• Argue that others don’t apply.• using evidence or logic

Quasi

Two kinds of quasi-experiments

• Non-equivalent control group

• “non-equivalent” because not randomly assigned

Quasi


• Interrupted time-series design

• a series of observations over time, interrupted by some treatment

Quasi

Non-equivalent Control Group design

• Control group is “like” the treatment group.

• Chosen from same population

• Pre- and post-test measures obtained for both groups, so similarity can be assessed.

Quasi


• Control group is not equivalent

• subjects are not randomly-assigned to control & treatment groups

• so best you can do is argue that comparison is appropriate.

Quasi


• If the groups are comparable to begin with, this design potentially eliminates threats to internal validity due to:

• History• Maturation• Testing• Instrumentation• Regression

Quasi

Problems with the NECG design

• Threats to validity due to interactions with selection may not be eliminated using the NECG design.

• Selection and maturation

• Most likely when treatment group is self-selected (as in psychotherapy cases – people who sought help).

Quasi


• Selection and history • Does one group experience some event that has a positive or negative effect (e.g., teacher of one class leaves)?

Quasi


• Selection and instrumentation

• Does one group show ceiling or floor effects?

Quasi


• Regression to the mean • Are one group’s pretest scores more extreme than the other group’s?

Quasi

Possible NECG study outcomes

• both experimental and control groups show improve-ment from pretest to posttest

• appears not to be any effect of the treatment

Quasi

Pretest Posttest

Control group


• Looks like a treatment effect, but there may be a threat due to• selection and maturation, • selection and history

Quasi

Pretest Posttest

Control group


• Selection and maturation could be a threat

• Or interaction of selection and • history• testing• instrumentation• or mortality.

Quasi

Pretest Posttest

Control group


• Interaction of selection and regression looks like a serious threat here

• Selection and maturation probably not a threat here.

Quasi

Pretest Posttest


• Crossover effect• Clearest evidence for an

effect of the program of any of these graphs.

• Selection and instrumentation not a problem – no ceiling or floor effects

Quasi

Pretest Posttest

Quasi-experiment example

• Langer & Rudin (1976)• Research conducted in

retirement home.

• Residents on one floor given more control over their daily lives

• Residents of another floor given same interaction with staff, but no increased control.

Quasi

Langer & Rudin (1976) – Measures

• Ratings• Self-report of feeling of

control from residents• Staff assessments of mental &

physical well-being, by ‘blind’ assessors

• Objective measures • record of movie attendance• participation in “Guess how

many jelly-beans” contest on each floor

Quasi

L & R (1976) – limits on control

• L & R had no control over • who entered the home• who was assigned to either

floor.• no control over staff hiring

or firing / resigning.

Quasi

L & R (1976) – Possible Problems

• Interaction of Selection and Maturation• even if groups have similar

pretest scores, they may differ on things pretest didn’t measure

• probably not a problem here – people on both floors had similar SES

• assigned to floors randomly, not by health status.

Quasi


• Selection and history • suppose a popular (or unpopular) nurse left one of the floors during the study. That might influence well-being.

• L & R did not address this issue.

Quasi


• Selection and instrumentation

• did one group show ceiling or floor effects?

• L & R say, no.

Quasi


• Regression • were one group’s pretest scores more extreme than the others?

• L & R say, no.

Quasi


• Observer bias and Contamination

• observers in the L & R study were not aware of the hypothesis.

• L & R reported there was little communication between floors.

Quasi


• Hawthorne Effect • cannot be ruled out, but L & R took care to give both floors same attention.

• Message varied between floors, but “face time” was the same.

Quasi


• External Validity • might be an issue. • home involved was rated

“one of the finest” in the state

• subjects may have been atypical in their desire for control

Quasi


• Non-equivalent control group

• Interrupted time-series design

• a series of observations over time, interrupted by some treatment

Quasi

Time-Series Designs

• In T-S designs, performance is measured both before and after a treatment.

• If there is an abrupt change in performance at time of treatment, we conclude that treatment worked.

Quasi

Time-series designs example

• Campbell (1969)• Effect of speed limit

reduction on traffic fatalities in Connecticut

• incidence of traffic fatalities in years before and after the speed limit reduction,

• conclusion: speed limit change had a modest effect.

Quasi

Campbell (1969)

• Any threat to internal validity?

• other explanations for any change in traffic fatality incidence:• Changes in car safety• Weather• Record keeping

Quasi

Campbell (1969)

• Any threat to internal validity?

• Such effects should be similar in neighboring states

• Campbell found no change in fatality incidence in those states.

Quasi

Campbell (1969)

• Any threat to external validity?

• E.g., would treatment have same effect in other states, or are people in Connecticut more law-abiding?

Quasi

Campbell (1969)

• Time-series design eliminates most other threats to validity – e.g., maturation, testing, regression.

• For example, maturation would probably not produce a sudden change in performance of the kind found in Time-Series Designs.

Quasi

Quasi-Experiments – Outline 1. True Experiments a. Characteristics b. Threats to validity controlled by experiments c. Threats not controlled by experiments.

Documents

experiments threats

true experiments selection

treatment quasi slide

pretest quasi slide

practice quasi slide

study quasi slide

true experiments regression

true experiments mortality