The use & abuse of tests • Statistical significance ≠ practical significance • Significance ≠ proof of effect (confounds) • Lack of significance ≠ lack of effect
Jan 04, 2016
The use & abuse of tests
• Statistical significance ≠ practical significance
• Significance ≠ proof of effect (confounds)
• Lack of significance ≠ lack of effect
Factors that affect a hypothesis test
• the actual obtained difference
• the magnitude of the sample variance (s2)
• the sample size (n)
• the significance level (alpha)
• whether the test is one-tail or two-tail
X
Why might a hypothesis test fail to find a real result?
Two types of error
We either accept or reject the H0. Either way, we could be wrong:
Two types of error
False positive rate
False negative rate
“sensitivity” or “power”
We either accept or reject the H0. Either way, we could be wrong:
Error probabilities
When the null hypothesis is true:
P(Type I Error) = alpha
When the alternative hypothesis is true:
P(Type II Error) = beta
Two types of error
False positive rate
False negative rate
“sensitivity” or “power”
Type I error
The “false positive rate”
• We decide there is an effect when none exists; we reject the null wrongly
• By choosing an alpha as our criterion, we are deciding the amount of Type I error we are willing to live with.
• P-value is the likelihood that we would commit a Type I error in rejecting the null
Type II error
The “false negative” rate
• We decide there is nothing going on, and we miss the boat – the effect was really there and we didn’t catch it.
• Cannot be directly set but fluctuates with sample size, sample variability, effect size, and alpha
• Could be due to high variability… or if measure is insensitive or effect is small
Power
The “sensitivity” of the test
• The likelihood of picking up on an effect, given that it is really there.
• Related to Type II error: power = 1-
A visual example
(We are only going to work through a one-tailed example.)
We are going to collect a sample of 10 highly successful leaders & innovators and measure their scores on scale that measures tendencies toward manic states.
We hypothesize that this group has more tendency to mania than does the general population ( and )50 5
µ0 = 50
Rejection region
Step 1: Decide on alpha and identify your decision rule (Zcrit)
Z = 0 Zcrit = 1.64
null distribution
Step 2: State your decision rule in units of sample mean (Xcrit )
null distribution
Rejection region
µ0 = 50
Z = 0 Zcrit = 1.64
Xcrit = 52.61
Rejection regionRejection regionAcceptance region
Step 3: Identify µA, the suspected true population mean for your sample
µ0 = 50 Xcrit = 52.61 µA = 55
alternative distribution
Rejection region
power
beta
Xcrit = 52.61 µA = 55µ0 = 50
Step 4: How likely is it that this alternative distribution would produce a mean in the rejection region?
Z = 0Z = -1.51
alternative distribution
beta
µ0 XcritµA
alpha
Power & Error
Power is a function of
The chosen alpha level ()
The true difference between 0 and A
The size of the sample (n)
The standard deviation (s or )
standard error
beta
µ0 XcritµA
alpha
Changing alpha
beta
µ0 XcritµA
Changing alpha
alpha
beta
µ0 XcritµA
alpha
Changing alpha
beta
µ0 XcritµA
Changing alpha
alpha
beta
µ0 XcritµA
alpha
Changing alpha
• Raising alpha gives you less Type II error (more power) but more Type I error. A trade-off.
Changing distance between 0 and A
beta
µ0 XcritµA
alpha
beta
µ0 XcritµA
alpha
Changing distance between 0 and A
beta
µ0 XcritµA
alpha
Changing distance between 0 and A
beta
µ0 XcritµA
alpha
Changing distance between 0 and A
beta
µ0 XcritµA
alpha
Changing distance between 0 and A
• Increasing distance between 0 and A lowers Type II error (improves power) without changing Type I error
Changing standard error
beta
µ0 XcritµA
alpha
beta
µ0 XcritµA
alpha
Changing standard error
beta
µ0 XcritµA
alpha
Changing standard error
beta
µ0 XcritµA
alpha
Changing standard error
beta
µ0 XcritµA
alpha
Changing standard error
• Decreasing standard error simultaneously reduces both kinds of error and improves power.
To increase power
Try to make really different from the null-hypothesis value (if possible)
Loosen your alpha criterion (from .05 to .10, for example)
Reduce the standard error (increase the size of the sample, or reduce variability)
For a given level of alpha and a given sample size, power is directly related to effect size. See Cohen’s power tables, described in your text