Experimental Design. Experimental design is the part of statistics that happens before you carry out an experiment Proper planning can save many headaches.

Experimental Design

Experimental Design

• Experimental design is the part of statistics that happens before you carry out an experiment

• Proper planning can save many headaches

• You should design your experiments with a particular statistical test in mind

Why do experiments?

• Contrast: observational study vs. experiments

• Example: – Observational studies show a positive

association between ice cream sales and levels of violent crime

– What does this mean?

Why do experiments?

• Contrast: observational study vs. experiments

• Example: – Observational studies show a positive

association between ice cream sales and levels of violent crime

– What does this mean?

Alternative explanation

Hot weather

Ice cream

Violentcrime

Alternative explanation

Hot weather

Ice cream

Violentcrime

Correlation is not causation

Why do experiments?

• Observational studies are prone to confounding variables: Variables that mask or distort the association between measured variables in a study– Example: hot weather

• In an experiment, you can use random assignments of treatments to individuals to avoid confounding variables

Goals of Experimental Design

• Avoid experimental artifacts• Eliminate bias

1. Use a simultaneous control group2. Randomization3. Blinding

• Reduce sampling error1. Replication2. Balance3. Blocking





Experimental Artifacts

• Experimental artifacts: a bias in a measurement produced by unintended consequences of experimental procedures

• Conduct your experiments under as natural of conditions as possible to avoid artifacts





Control Group

• A control group is a group of subjects left untreated for the treatment of interest but otherwise experiencing the same conditions as the treated subjects

• Example: one group of patients is given an inert placebo

The Placebo Effect

• Patients treated with placebos, including sugar pills, often report improvement

• Example: up to 40% of patients with chronic back pain report improvement when treated with a placebo

• Even “sham surgeries” can have a positive effect

• This is why you need a control group!

Randomization

• Randomization is the random assignment of treatments to units in an experimental study

• Breaks the association between potential confounding variables and the explanatory variables

Experimental unitsC

onfo

undi

ng v

aria

ble

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

Without randomization, the confounding variable differs among treatments

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

Experimental unitsC

onfo

undi

ng v

aria

ble

Treatments

With randomization, the confounding variable does not differ among treatments

Blinding

• Blinding is the concealment of information from the participants and/or researchers about which subjects are receiving which treatments

• Single blind: subjects are unaware of treatments

• Double blind: subjects and researchers are unaware of treatments





Replication

• Experimental unit: the individual unit to which treatments are assigned

Experiment 1

Experiment 2

Experiment 3

Tank 1 Tank 2

All separate tanks

Replication


Experiment 1

Experiment 2

Experiment 3

Tank 1 Tank 2

All separate tanks

2 Experimental Units



Replication


Experiment 1

Experiment 2

Experiment 3

Tank 1 Tank 2

All separate tanks




Pseudoreplication

Why is pseudoreplication bad?

• problem with confounding and replication!

• Imagine that something strange happened, by chance, to tank 2 but not to tank 1

• Example: light burns out

• All four lizards in tank 2 would be smaller

• You might then think that the difference was due to the treatment, but it’s actually just random chance

Experiment 2

Tank 1 Tank 2

Why is replication good?

• Consider the formula for standard error of the mean:

SEY

s

n

Larger n Smaller SE

Balance

• In a balanced experimental design, all treatments have equal sample size

Better than

Balanced Unbalanced

Balance

• In a balanced experimental design, all treatments have equal sample size

• This maximizes power

• Also makes tests more robust to violating assumptions

Blocking

• Blocking is the grouping of experimental units that have similar properties

• Within each block, treatments are randomly assigned to experimental treatments

• Randomized block design

Randomized Block Design

Randomized Block Design

• Example: tanks in a field

Very sunny

Not So Sunny

Block 1

Block 4

Block 2

Block 3

What good is blocking?

• Blocking allows you to remove extraneous variation from the data

• Like replicating the whole experiment multiple times, once in each block

• Paired design is an example of blocking

Experiments with 2 Factors

• Factorial design – investigates all treatment combinations of two or more variables

• Factorial design allows us to test for interactions between treatment variables

Factorial Design

5.5 6.5 7.5

25 n=2 n=2 n=2

30 n=2 n=2 n=2

35 n=2 n=2 n=2

40 n=2 n=2 n=2

Tem

pera

ture

pH

Interaction Effects

• An interaction between two (or more) explanatory variables means that the effect of one variable depends upon the state of the other variable

Interpretations of 2-way ANOVA Terms

0

10

20

30

40

50

60

70

25 30 35 40

Temperature

Gro

wth

Rate

pH 5.5

pH 6.5

pH 7.5

Effect of pH and Temperature,No interaction

0

5

10

15

20

25

30

35

40

45

25 30 35 40

Temperature

Gro

wth

Rate

pH 5.5

pH 6.5

pH 7.5

Interpretations of 2-way ANOVA Terms

Effect of pH and Temperature,with interaction





What if you can’t do experiments?

• Sometimes you can’t do experiments

• One strategy:– Matching– Every individual in the treatment group is

matched to a control individual having the same or closely similar values for known confounding variables

What if you can’t do experiments?

• Example: Do species on islands change their body size compared to species in mainland habitats?

• For each island species, identify a closely related species living on a nearby mainland area

Power Analysis

• Before carrying out an experiment you must choose a sample size

• Too small: no chance to detect treatment effect

• Too large: too expensive

• We can use power analysis to choose our sample size

Experimental Design. Experimental design is the part of statistics that happens before you carry out an experiment Proper planning can save many headaches.

Documents

experimental design

variable slide

experimental units

variable treatments

blocking slide

experimental study

causation slide

mind slide