To P or not to P???: Does a statistical test holds what it promises? Probably more than 70% of all medical and biological scientific studies are irreproducible! There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims. ” ,” Ioannidis (2005, PLoS Medicine)
31
Embed
To P or not to P???: Does a statistical test holds what it promises?
To P or not to P???: Does a statistical test holds what it promises?. There is increasing concern that in modern research , false findings may be the majority or even the vast majority of published research claims . ” ,” Ioannidis (2005, PLoS Medicine ). - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
To P or not to P???:Does a statistical test holds what it promises?
Probably more than 70% of all medical and biological scientific studies are irreproducible!
There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims.” ,” Ioannidis (2005, PLoS Medicine)
We compare the effects two drugs to control blood
pressure in two groups of patients
We test the effect of a drug to control blood pressure
against a null control group
We test for a significant correlation
𝑃 (𝑡 )=𝑃 (𝑡=𝑒𝑓𝑓𝑒𝑐𝑡 𝑠𝑖𝑧𝑒
𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑𝑒𝑟𝑟𝑜𝑟)<0.001
We compare two observationsWe use the t-test to calculate a
probability of difference.
We compare an observation against a null expectation
We use the t-test to assess the validity of a hypothesis.
We compare an observed statistic against an unobserved
null assumption.
)
Formally, we test H1: r2 = 0.57 against the alternative H0 : r2 = 0
This is not the same as to test the hypothesis that X and Y are correlated against the hypothesis that X and Y are not correlated.
X and Y might not be correlated but have a r2 ≠ 0.This appears if X and Y are jointly constraint by marginal settings.
An intuitive significance test
We test an observation against a specific null assumption.We compare two specific values of r2.
Number of storks and reproductive rates in Europe (Matthews 2000)
Urbanisation
Storks Birth rate
Storks
Pseudocorrelations between X and Y arise whenX = f(U)Y = g(U)f = h(g)
The sample spaces of both variables are constraint by one or more hidden variables that are itself correlated.
Birth rate
r2 = 0.25; P < 0.05
Excel gets plots at log-scales
wrong.
Country Area No. stork pairs Stork density Inhabitants Annual no.
Fisher argued that hypotheses can be tested using likelihoods.
𝜒2 (𝑘 )←∑1
𝑘
𝑍 𝑖❑2 𝜒2 (𝑘 )←∑
1
𝑘 𝑍12
𝑍22 →
𝐹𝑘
The quotient of two normally distributed random variates is c2 distributed.
-2ln( l is normally distributed. -2ln(l) is c2 distributed.
Classical frequentist hypothesis testing
We throw 100 times a coin and get 59 times the head. Is the coin fair?
𝑝 (𝑥=59|12 )=(10059 )( 12 )100
=0.016
𝑝 (𝑥=59| 59100 )=(10059 )( 59100 )59
(1− 59100 )100−59
=0.081
P = 1/2 P = 59/100
Likelihood with P = ½ and P = 59/100 estimates
Fisher would contrast two probabilities of a binomial process given the outcome of 59 heads.
Fisher:• The significance P of a test is the probability of the hypothesis given the data!• The significance P of a test refers to a hypothesis to be falsified.• It is the probability to obtain an effect in comparison of a random assumption.• As a hypothesis P is part of the discussion of a publication.
Λ=−2 ln( 0.0160.081 )=−2 ln (0.20 )=3.26
𝑃 ( Λ )=𝑃 ( 𝜒2=3.26 ;𝑑𝑓 𝜒 2=1 )=0.93
The odds in favour of H1 is 0.016/0.081 = 0.2. H1 is five time more probable than H0. The probability that the observed binomial probability q1 = 59/100 differs from q1 = ½ given
the observed data is Pobs = 0.93.The probability in favour of the null assumption is therefore P0 = 1-0.93 = 0.07.
In Fisher’s view a test should falsify a hypothesis with respect to a null assumption given the data.
This is in accordance with the Popper - Lakatos approach to scientific methodology.
According to Fisher the test failed to reject the hypothesis of P = 59/100.
Pearson-Neyman asked what is the probability of the data given the model!
The significance value a of a test is the probability (the evidence) against the null assumption.
1-P
Q 1-Q
P
H1 true H0 true
Reject H0
Reject H1
Type I error
Type II error
P is the probability to reject H0 given that H0 is true (the type I error rate).
It is not allow to equal P and Q, the probability to reject H1 given that H1 is true
(the type II error rate).
0 1 2 3 4 50
0.01
0.02
0.03
0.04
0 1 2 3 4 50
0.20.40.60.8
1P P
Test valueb b
H1H0
P(H1)
Distribution of b under H0 Cumulative distribution of b under H0
Classical frequentist hypothesis testing
Pearson-Neyman:• The significance P of a test is the probability that our null hypothesis is true in comparison
a to precisely defined alternative hypothesis.• This approach does not raise concerns if we have two and only two contrary hypotheses
(tertium non datur).• As a result P is part of the results section of a publication.
In the view of Pearson and Neyman a test should falsify a null hypothesis with respect to the observation.
For 50 years Pearson and Neyman won because their approach is simpler in most applications.
P is not the probability that H0 is true!!1-P is not the probability that H1 is true!!
Rejecting H0 does not mean that H1 is true.Rejecting H1 does not mean that H0 is true.
A test aims at falsifying a hypothesis.
• The test does not rely on prior information.• It does not consider additional hypothesis.• The result is invariant of the way of testing.
Fisher: Pearson-Neyman:
A test aims at falsifying a null assumption.
We test against assumed data that have not been measured.
We test for differences in the model parameters.
P values are part of the hypothesis development.
P values are central to hypothesis testing.
We test the observed data.We test against something that has not been
measured.
Modus tollens
𝐴→¬𝐵 If Ryszard is from Poland he is probably not a member of Sejm.Probably Ryszard is a member of Sejm.Thus he is probably not a citizen of Poland.
If P(H1) > 0.95 H0 is probably false.H0 is probably true.P(H1) < 0.95.This does not mean that H1 is probably false.It only means that we don’t know.
A word on logic
If multiple null assumptions are possible the results of classical hypothesis testing are difficult to interpret.
If multiple hypotheses are contrary to a single null hypothesis the results of classical hypothesis testing are difficult to interpret.
Pearson-Neyman and Fisher testing works always properly if there are two and only two truly contrasting alternatives.
The pattern of co-occurrences of the two species appeared to be random (P(H0) > 0.3).(we cannot test for randomness)
We reject our hypothesis about antibiotic resistences in the Bacillus thuringiensis strains P(H1) > 0.1. (we can only reject null hypotheses)
The two enzymes did not differ in substrate binding efficacy (P > 0.05). (we do not know)
Time of acclimation and type of injection significantly affected changes in Tb within 30 min after injection (three-way ANOVA: F5;461 = 2:29; P<0.05). (with n = 466, time explains 0.5% variation)
The present study has clearly confirmed the hypothesis that non-native gobies are much more aggressive fish than are bullheads of comparable size... This result is similar to those obtained for invasive round goby in its interactions with the native North American cottid. (F1,14 = 37.83); (if others have found the same, we rather should test for lack of difference. The present null assumption is only a straw man).
Examples
conditional priori(A)posterior
priori(B)
Thomas Bayes (1702-1761)
Abraham de Moivre (1667-1754)
𝑝 (𝐴⋀ 𝐵 )=𝑝 (𝐴|𝐵 )𝑝 (𝐵)𝑝 (𝐵⋀ 𝐴 )=𝑝 (𝐵|𝐴 )𝑝 ( 𝐴)
𝑝 (𝐴⋀ 𝐵 )=𝑝 (𝐵⋀ 𝐴)¿
𝑝 (𝐴|𝐵 )𝑝 (𝐵)=𝑝 (𝐵|𝐴 )𝑝 (𝐴)
𝑝 (𝐴|𝐵 )=𝑝 (𝐵|𝐴)𝑝 (𝐴)
𝑝 (𝐵)=𝑝 (𝐵|𝐴 )𝑝(𝐵)
𝑝 (𝐴)
Theorem of Bayes
The Bayesian philosophy
The law of conditional probability
Theorem of Bayes
0 0.1 0.90.5 0.99
A frequentist test provides a precise estimate of probability
0 0.1 0.90. 5 0.99
P PDP
P
A Bayesian interpretation of probability
Under a Bayesian interpretation a statistical test provides an estimate of how much a test shifted an initial assumption about the level of probability in favour of our
hypothesis towards statistical significance.Significance is the degree of belief based on prior knowledge.
Under a frequentist interpretation a statistical test provides an estimate of the probability in favour of our null hypothesis.
In the frequentist interpretation probability is an objective reality.
𝑝 (𝑝𝑜𝑠𝑡|𝑝𝑟𝑖𝑜𝑟 )=𝑝 (𝑝𝑜𝑠𝑡)
𝑝 (𝑝𝑜𝑠𝑡|𝑝𝑟𝑖𝑜𝑟 )=𝑝 (𝑝𝑟𝑖𝑜𝑟|𝑝𝑜𝑠𝑡 )𝑝 (𝑝𝑟𝑖𝑜𝑟 )
𝑝 (𝑝𝑜𝑠𝑡)
Post is independent of prior
Post is mediated by prior
𝑝 (𝑝𝑜𝑠𝑡|𝑝𝑟𝑖𝑜𝑟 )≤𝑝 (𝑝𝑜𝑠𝑡)
P
The earth is round: P < 0.05 (Goodman 1995)
0 0.9 0.10. 5 0.01
P P
We perform a test in our bathroom to look whether the water in the filled bathtub is curved according to a globe or to a three-dimensional heart.Our test gives P = 0.98 in favour of earth like curvature (P(H0) < 0.05).Does this change our view about the geometry of earth? Does this mean that a heart model has 2% support?
P PThe Bayesian probability level in favour of H0
The frequentist probability level in favour of H0 that the earth is a heart
The higher our initial guess about the probability of our hypothesis is, the less does any new test contribute to further evidence.Frequentist tests are not as strong as we think.
Often null hypotheses serve as straw man only to „support” or hypothesis (fictional testing)
0.001 0.0001 0.00001 0.00000001
Confirmatory studies
Bayesian prior and conditional probabilities are often not known and have to be guessed.
Frequentist inference did a good job, we have scientific progress.
𝑝 (𝑝𝑜𝑠𝑡|𝑝𝑟𝑖𝑜𝑟 )=𝑝 (𝑝𝑟𝑖𝑜𝑟|𝑝𝑜𝑠𝑡 )𝑝 (𝑝𝑟𝑖𝑜𝑟 )
𝑝 (𝑝𝑜𝑠𝑡)
Our test provides a significance level independent of prior information only if we are quite sure about the hypothesis to be tested.
A study reports that potato chips increase the risk of cancer. P < 0.01.
P(H1) = 0.99However, previous work did not find a relationship. Thus we believe that p(H1) < 0.5.
Our test returns a probability of P = (0.0 < P < 0.5) * 0.99 < 0.5
The posterior test is not as significant as we believe.
Tests in confirmatory studies must consider prior information.
𝐵𝐹=𝑝( 𝐴)𝑝 (𝐵)
=𝑝 (𝐴|𝐵 )𝑝 (𝐵|𝐴)
Bayes factor, odds
𝐵𝐹=𝑝 (𝑡|𝐻 1 )𝑝 (𝑡|𝐻 0 )
We have 59 heads and 41 numbers. Does this mean that head has a higher probability?
𝑝 (𝑥≥59|12 )=∑𝑖=59
100
(100𝑖 )( 12 )100
=0.044
The frequentist approach
Under Bayesian logic the observed result is only 5 times less probable
than any other result.
𝐾=0.0440.0099
=4.44
The odds for a deviation is 4.44.1/4.44 = 0.23
Bayesian inference
The Bayes approach asks what is the probability of our model with respect to any other possible
Number of species co-occurrences in comparison to a null expectaction (data are simple random numbers)
The variance of the null space decreases due to statistical averaging.Nobs
Null distribution
𝑡=𝑒𝑓𝑓𝑒𝑐𝑡 𝑠𝑖𝑧𝑒
𝑆𝐸
𝑆𝐸→0→ 𝑡→∞
• Any test that evolves randomisation of a compound metric will eventually become significant due to the decrease of the standard error.
• This reduction is due to statistical averaging.
The null model relies on a randomisation of 1s and 0s in the matrix
At very large sample sizes (N >> 100) classical statistical tests break down.
Instead of using a predefined significance level use a predefined effect size or r2 level.
The T-test of Wilcoxon revealed a statistically significant difference in pH of surface water between the lagg site (Sphagno-Juncetum) and the two other sites.
Every statistical analysis must at least present sample sizes, effect sizes, and confidence limits.Multiple independent testing needs independent data.
Pattern seeking or P-fishing
PersonBlood presure Gender Age class Smoker
1 80 m 30 y
2 133 f 40 y
3 64 m 60 n
4 139 f 40 y
5 63 m 80 n
6 105 f 70 y
7 114 f 60 y
Variables SS df MS F PGender 1 1 1.37 0.00 0.97Age class 15183 8 1897.85 2.32 0.02Smoker 4062 1 4061.61 4.97 0.03Gender*Age class 6507 7 929.57 1.14 0.34Gender*Smoker 1168 1 1167.74 1.43 0.23Age class*Smoker 8203 7 1171.81 1.43 0.19Gender*Age class*Smoker 4083 5 816.58 1.00 0.42
Error 790913 968 817.06
Simple linear random numbers Of 12 trials four gave significant results
Using the same test several times with the same data needs a Bonferroni correction.
Single test
)(1)( sigpnsigp
)()(
))(1(1
))(1(1)(
))(1()(
signpsigp
signp
sigpsigp
sigpnsigp
testExp
test
ntestExp
ntestExp
n independent tests
nn TestTestExp
05.005.0
The Bonferroni correcton is very conservative.
False discovery rates (false detection error rates): The proportion of erroneously declared significances.
• Common practise is to screen the data for significant relationships and publish these significances.
• The respective paper does not mention how many variables have been tested.
• Hypotheses are constructed post factum to match the „findings”. • „Results” are discussed as if they corroborate the hypotheses.
P-fishing
If the data set is large• Divide the records at random into two or more parts.• Use one part for hypothesis generation, use the other parts for testing.• Use always multiple testing corrected corrected significance levels.• Take care of non-independence of data. Try reduced degrees of freedom.
• Hypotheses must come from theory (deduction), not from the data.• Inductive hypothesis testing is critical.• If the hypotheses are intended as being a simple description, don’t use P-values.
P < 0.000001
Possibly data are non-independent due to sampling
sequence
No clear hypothesis
Final guidelines
Don’t mix data description, classification and hypotheses testing.Provide always sample sizes and effect sizes . If possible provide confidence limits.
Data description and model selection:• Rely on AIC, effect sizes, and r2 only.• Do not use P-values.• Check for logic and reason.
Hypothesis testing:• Be careful with hypothesis induction.
Hypotheses should stem from theory not from the data.
• Do not develop and test hypotheses using the same data.
• Do not use significance testing without a priori defined and theory derived hypotheses.
• Check for logic and reason.• Check whether results can be reproduced. • Do not develop hypotheses post factum
(telling just so stories)Testing for simple differences and relationships:• Be careful in the interpretation of P-values. P does not provide the probability that a
certain observation is true. • P does not provide the probability that the alternative observation is true.• Check for logic and reason.• Don’t use simple tests in very large data sets. Use effect sizes only.• Use predefined effect sizes and explained variances.• If possible use a Bayesian approach.