KNR 295 Research Design Slide 1 Design Ch. 6 – Internal Validity.

KNR 295Research DesignSlide 1

Design

Ch. 6 – Internal Validity

KNR 295Research DesignSlide 2 Internal Validity

Cause and effect Cause…

“Function: transitive verbInflected Form(s): caused; caus·ingDate: 14th century1 : to serve as a cause or occasion of : MAKE2 : to effect by command, authority, or force- caus·er noun”

*from Merriam-Webster online


Cause and effect Effect…

“Function: nounEtymology: Middle English, from Middle French & Latin; Middle French, from Latin effectus, from efficere to bring about, from ex- + facere to make, do -- more at DODate: 14th century1 something that inevitably follows an antecedent (as a cause or agent) ”

*from Merriam-Webster online


Cause and effect

Internal validity is a “zero-generalizability concern”


Cause and effect So…did the treatment you used

cause the measurement you took to be what it was?

Or…to what extent did the treatment you used influence the measurement you took?


Establishing cause and effect Example: Does beer make you

happy? Temporal precedence

beer

happiness



happy? Covariation of cause and effect

If beer then happy

If not beer then not happy

If more beer then more happy

If less beer then less happy



happy? No plausible alternative explanations“Never drink alone”…social

causes?To drink, you need to be flush…it’s an economic difference?

Drinkers experience smoke too…it’s all down to ciggies?

When you drink you go to the loo more often…something to do with bladder swelling/emptying?

Ran

k –

most

p

lau

sib

le?

- least

p

lau

sib

le?


Single group threats Sticking with the beer idea…

Imagine you take a group of students, and give them a dose of beer. You might measure their happiness just once at the end, or you might measure them both before and after the beer dose…



Give beer

Measure happiness

Measure happiness

Measure happiness

Give beer



History Threat It’s not the beer. It’s some other major historical

event that influenced your group’s happiness… Perhaps on the day of the test, Saddam decided

to pack it in, and everyone simultaneously found out that the recession was a big mass-induced dream state, awaking to find that the world was infinitely happier than they had first thought?

Pla

usi

ble

?



Maturation Threat As people get older, they get happier. You took

your first measurement during your group’s teenage angst years, then waited until the wisdom, wonder and general delight of 40ish life settled in before taking the second measurement

[can be anything related to general “internal” change in people – any “big event” external to the person causing the change is generally regarded as a history threat]

Pla

usi

ble

?



Testing Threat (Pre-post only) Measuring happiness made participants focus on

their degree of happiness – if they were unhappy, they resolved to buckle up, and cheer up. If they were happy they were made aware of this and were therefore delighted to be so well-adjusted

Pla

usi

ble

?



Instrumentation Threat (pre-post again) A couple of possibilities…

The group are scored with different instruments on each occasion. The second is more sensitive than the first. The group’s happiness doesn’t change, but the scores arising from the instruments differ anyway

The person giving out the happiness instrument is such a happy-go-lucky lass that she infects everyone else with her sense of “joi de vivre” (experimenter as instrument)- the longer she’s around the happier you feel

Pla

usi

ble

?



Mortality Threat As the study wears on and the beer starts to

have its inevitable effect, many people are past the stage of being happy and are approaching that other unfortunate stage of influence…but they aren’t around for the happiness measurement at the end of the studyP

lau

sib

le?



Regression Threat [The Fugawe Tribe] When you look at the scores of your group on

the pre-test, you notice that they are unusually low… This situation, where for any reason scores are just

by chance suppressed on the first measurement, will probably result in artificially higher scores (in comparison to the first) on the second reading

It’s not that the treatment is having an effect, it’s that the scores are just going back to normal

Normally occurs when selecting a group by pre-test scores…

Pla

usi

ble

?



Regression Threat You want to find out if the effect even works on

really unhappy people. So you get 200 grad students to fill out a happiness questionnaire, and select only the lowest 25% for your study.

Lo and behold, after a good dose of beer, they all score higher on the hap-o-meter!

Why?

Pla

usi

ble

?


Solving single group threats Add a control group The only difference between the

control group and the treatment group should be the presence or absence of the treatment

Sometimes this means you need multiple control groups

Often, it still fails…


Multiple group threats All are selection bias threats The critical question is: “were the

groups equal (on the measure of choice) at the start of the study”

The general case is simply a selection bias. Other specific differences follow


Multiple group threats

Single-group threats – taken care of now

Multiple-group threats – still a problem


Multiple group threats More examples with the hoppy

one… Selection-history threat

Say you got your 2 groups from grad classes On the day of the test, one of the groups’

classes has a nasty test while the other has a normal class

What is causing the difference in happiness now?



one… Selection-maturation threat

Say you’re looking at the long-term effects of beer on happiness (not temporary as now)

Say also that one group (the drinkers) is selected from among the student population, while the other is selected from among the faculty

It might be easy to imagine that the older group is somewhat more stable with respect to the happiness measure than the younger



one… Selection-testing threat

Generally, that one group is for some reason more sensitive to be influenced by the first test than the second

Say you get the beer group to volunteer from a pub, while the non-beer group volunteers from a teetotal social club

The pub group also (unknown to you) score higher on a neuroticism scale. This leads them to be more easily influenced by knowledge of their score on a happiness scale…and thus alters their responses second time around



one… Selection-instrumentation threat

Say we have our happy-go-lucky experimenter from the single group threats operating here too, but she can’t do both groups (schedule can’t allow), so she gets her partner (who is rather gloomy…funny how that seems to happen) to monitor the other group



one… Selection-mortality threat

Even if you add another (non-beer) group, it is quite probable that your non-beer group will be intact at the end of the study, while your beer group might suffer substantial drop-out



one… Selection-regression threat

Say you select your volunteers from a pub on a weekday afternoon. The others are selected at random from grad research classes

Trouble is, folk who drink at a pub on a weekday afternoon tend to be a depressed lot…and thus are more likely to improve their scores on a subsequent post-test

[this is fiction, folks!]


Solving Multiple group threats Randomly assign to groups Random assignment ensures no

systematic difference between groups

We’re still not done yet though…


Social interaction threats

Single-group threats – taken care of by adding control group

Multiple-group threats – taken care of by random assignment

Social interaction threats


Social interaction threats One more time with the amber

nectar (as Ben Franklin once said, “beer is proof that God loves us and wants us to be happy”) Diffusion or imitation of treatment

Imagine you pluck all your participants from a college hall of residence. They are aware of each other’s participation, and the other group’s treatment. On seeing the beer group go off for their treatment, the non beer group think “that’s a good idea” and troop off to the liquor store…


Social interaction threats One more time…

Compensatory rivalry In some way the groups become aware of each

other’s treatments and of the measures, and something about them sets up a different behavior that influences the final measurement

For example, the non-beer set may for some reason not want to be judged as less happy” than the beer group second time around.



Resentful demoralization Say the non-beer group are aware the beer

group is getting the beer Say also the non beer group enjoy a beer They’re jealous This unhappiness finds its way into the post-test



Compensatory equalization Say the non-beer group are aware the beer

group is getting the beer Say also the non beer group enjoy a beer Say the experimenters are aware of this They treat the other group with extra kindness

and comfort to offset the perceived slight at the lack of beer


Social interaction threats And one extra I’d add here

(repeated from construct validity) Experimenter bias

The belief in the treatment in some way causes the experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy

KNR 295Research DesignSlide 34 Introduction to design

KNR 295Research DesignSlide 35 Types of designs

KNR 295Research DesignSlide 36 Types of designs

KNR 295 Research Design Slide 1 Design Ch. 6 – Internal Validity.

Documents

effectif beer

beer ideaimagine

dose of beer

beer ideahistory threatits

covariation of cause

group of students

general internal change

middle english