chapter 7 Statistical Issues in Research Planning and Evaluation
Chapter Outline
• Probability• Meaningfulness• Power• Using information in context of the study• Reporting statistical data
Probability
• What are the odds that a certain thing(s) will happen?– Equally likely events– Relative frequency
• Alpha (α): – Level of chance occurrence set by the researcher
prior to the study– Typical: p < .05 or p < .01
• Odds that findings are due to chance are either 5 or 1 in 100
– Controls for Type I error
Truth Table forthe Null Hypothesis
Ho true Ho false
Accept Correct decision Type II error (beta)
Reject Type I error (alpha)
Correct decision
Probability
• Beta (β):– Magnitude of a Type II error– Acceptance of null hypothesis when it is FALSE
• As alpha is set increasingly smaller, beta becomes larger
Sampling for Null Hypothesis
From Experimental procedures for behavioral science, 3rd ed., by R.E. Kirk © 1995. Reprinted with permission of Brooks/Cole, an imprint of the Wadsworth Group, a division of Thomson Learning. Fax 800-730-2215.
Meaningfulness
• The practical significance of an effect or relationship
• Effect Size (ES)– Standardized value that is the difference between
the means divided by the standard deviation
• Formula:
– ES = (M1 – M2)/s
Estimating Effect Size
• ES allows comparison between studies using different dependent variables because it puts data in standard deviation units.
• Scale:– ES of 0 is no difference,
– ES of 0.2 or less is small,
– ES of ~0.5 is medium, and
– ES of 0.8 or more is large
Power
• The probability of rejecting the null hypothesis when it is false (detecting a real difference), or
• The probability of making a correct decision• This concepts answers the following
questions:– How large a difference is important in theory or
practice?– How many participants are needed to declare an
important difference as significant?
Using Information in the Context of the Study
• How do findings from the study fit within the context of– Theory– Practice
• Context is what matters with regard to “meaningfulness”– Estimates of significance are driven by sample size– Estimates of meaningfulness are driven by the size
of the difference– Context is driven by how the findings will be used
Summary
• Information needed when planning research1. Alpha
• Establishes the acceptable magnitude of Type I error• Usually .05 or .01
2. Effect size• The outcome of a study expressed in standard deviation
units
3. Power• The chance of rejecting a false null hypothesis• 1 - beta
4. Sample size• Number of participants in the study
Using the Power Calculator WhenReading a Research Study
When reading research, often sample size, means, and standard deviations are supplied. You can calculate the effect size by the formula in chapter 7. Using this data and the Power Calculator at the Web site below, you can estimate the power to detect a difference or relationship.
http://calculators.stat.ucla.edu/powercalc/
Using the Power Calculatorto Plan Research
If you are planning your own research, you can often estimate the effect size from other studies. By setting your alpha (say .05) and power (say .8), you can use the Power Calculator at the website below to estimate the sample size you need.
http://calculators.stat.ucla.edu/powercalc/