Evaluating scientific data

Evaluating Scientific Data

BIO 101LFall 2014

Why be cautious?

• Author’s motivations are not the same as your own– Sell you on a drug,

therapy, hospital, etc– Sell you a magazine– ‘Publish or perish’

• Applies more broadly than health claims Eric

Poehlman

What is pattern here?

Turner & Spillich 1997

Questions to ask:

1. What is the motivation of the person making the claim?

2. Is evidence based on correlation or causation?

3. What is the inference space?4. How large is the effect?5. How large of a population was tested?6. Is the effect statistically significant?

Correlation or causation?

Correlation • shows a pattern• might suggest a cause• pattern may instead result from unmeasured

3rd variable

Correlation or causation?

Do nightlights cause nearsightedness (myopia)?

• What is pattern?

• What is explanation?

Do nightlights cause nearsightedness (myopia)?

• What is pattern?

• What is explanation?

Inference space?

• What is the population being studied?

Inference space

• s

Inference space

• What is the range of treatments being studied?

How large is the effect?

• ‘Increase’ or ‘Decrease’ alone does not indicate whether it is enough to matter.

• What is measurement without treatment?• How much does measurement change with

treatment?

Cell phones and brain cancer

•

How large of a population was tested?

•

How large of a population was tested?

•

Is effect statistically significant?

• Did results arise by chance?• Or is the effect itself actually important?• Often indicated by P-values• Lower P-values means that chance is less likely

to explain pattern• P < 0.05 is a common cut-off point• Does article specify “statistically significant?”