Why most published research findings are false Aur´ elien Madouasse Context Introduction Modelling Framework Hypothesis testing Bias Multiple testing Comments Corollaries Conclusion Why most published research findings are false Article by John P. A. Ioannidis (2005) Aur´ elien Madouasse November 4, 2011
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Why most published research findings are falseArticle by John P. A. Ioannidis (2005)
• Consider a parameter measured in a population ofindividuals with a disease:
• Before treatment
• After treatment (Here assuming the treatment has an effect)
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis testing
• Consider a parameter measured in a population ofindividuals with a disease:
• Before treatment• After treatment (Here assuming the treatment has an effect)
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis
• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis
• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect
• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• For a given hypothesis, whether we get it wrong dependson:
• Whether the hypothesis is true• The magnitude of the effect• The values we choose for α and β
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper
• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses
• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True
• Hypothesis testing can be seen as testing for a disease inEpidemiology
• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology
• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity
• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity
• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
PPV =p(1− β)
p(1− β) + (1− p)α
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R
1+R × (1− β)R
1+R × (1− β) + 11+R × α
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− β)
R(1− β) + α
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
• Among the studies that should have been reported asnegative
• A proportion u are reported as positive because of bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
• Among the studies that should have been reported asnegative
• A proportion u are reported as positive because of bias
The smaller the studies conducted in a scientific field, theless likely the research findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 2
The smaller the effect sizes in a scientific field, the lesslikely the research findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 3
The greater the number and the lesser the selection oftested relationships in a scientific field, the less likely theresearch findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 4
The greater the flexibility in designs, definitions, outcomesand analytical modes in a scientific field, the less likely theresearch findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 5
The greater the financial and other interests and prejudicesin a scientific field, the less likely the research findings areto be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 6
The hotter a scientific field (with more scientific teamsinvolved), the less likely the research findings are to be true