Effect Size - balkinresearchmethods.com · compute Cohen’s d or Cohen’s f, which measure effect size in standard deviation units. However, they are ... Omega squared ω2 and Eta-squared

Balkin, R. S.(2008). 1

Effect Size

Rick Balkin, Ph.D., LPCDepartment of Counseling

Texas A&M [email protected]

Balkin, R. S.(2008). 2

Statistical vs. PracticalSignificance

Statistical significance refers to the probabilitythat the rejection of the null hypothesisoccurred outside the realm of chance (alphalevel).

Practical significance refers to themeaningfulness of the differences, byspecifying the magnitude of the differencesbetween the means or the strength of theassociation between the independentvariable(s) and the dependent variable.

Balkin, R. S.(2008). 3

Practical significance: why weneed it.

A school counselor wants to compare aset of scores on the SAT to the nationalnorm. The population has a mean of500 and a standard deviation of 100.

Balkin, R. S.(2008). 4


If the school counselor has 25 students in the sample

with a mean (

X

) of 520, then the z-test would be

conducted as follows: 00.1

20

20

25

100

500520==

!=

!=

!=

n

XXz

X"

µ

"

µ

With an alpha level of .05 (non -directional) and z crit = 1.96, there is no statistic ally significant difference

between the sample group and the population ( z = 1.00,

p > .05).

Balkin, R. S.(2008). 5


Now, take the same scores, but increase the sample size

to 100.

00.210

20

100

100

500520==

!=

!=

!=

n

XXz

X"

µ

"

µ

With an alpha level of .05 (non -directional) and z crit =

1.96, there is a statistically significant difference

between the sample group and the population ( z = 2.00,

p < .05). The observed value is greater than the critical

value (2.00> 1.96).

Balkin, R. S.(2008). 6


Did you notice that with the smaller samplesize you did not have statistical significancebut with the larger sample size you did?

Although the magnitude of the meandifferences did not change, the interpretationof the results changed strictly based on theincrease in sample size. When sample sizeincreased, the error decreased.

Balkin, R. S.(2008). 7


Thus, statistically significant differencesare more likely to occur when largesamples are utilized.

Nearly any null hypothesis can berejected when a large enough sample isattained.

Balkin, R. S.(2008). 8


Practical significance is importantbecause it addresses the magnitude ofa treatment effect without thecomplication of sample size, therebyproviding more meaningful informationthat has usefulness to practitioners andresearchers (Kirk, 1995).

Balkin, R. S.(2008). 9


The following procedures are utilized toprovide measures of effect size todetermine practical significance.

Currently, statistical packages do notcompute Cohen’s d or Cohen’s f, whichmeasure effect size in standarddeviation units. However, they arerelatively simple computations.

Balkin, R. S.(2008). 10


The reporting of practical significance is veryimportant when reporting results andmandatory in many social science journals.

“For the reader to fully understand theimportance of your findings, it is almostalways necessary to include some index ofeffect size or strength of relationship in yourResults section” (APA, 2001, p. 25).

Balkin, R. S.(2008). 11

Cohen’s d

Cohen’s d is used to determine the effect sizefor the differences between two groups, suchas in a t-test or pairwise comparisons (i.e.Tukey post hoc), and is expressed in standarddeviation units.

Cohen (1988) created the followingcategories to interpret d: Small = .2 Medium = .5 Large = .8

Balkin, R. S.(2008). 12

Cohen’s d

Cohen’s d =

)1()1(21

212

2

2

21

!+!

+==

!

""nn

xxss

or

MS

or

s

XX

ww

error

error

The numerator value is the difference between two group means. The denominatoris the error term, which can be expressed in one of three ways.

Balkin, R. S.(2008). 13

Computing Cohen’s dTable 2.

Tukey post hoc analysis

Group

Comparisons Mean Difference p d

1 2 -3.00 0.0815 1.65

3 -1.00 0.8215 0.55

4 3.60 * 0.0301 1.97

2 3 2.00 0.3393 1.09

4 6.60* 0.0002 3.62

3 4 4.60* 0.0052 2.52

*p < .05

From the ANOVA example in thenotepack, the first group comparison would be computed asfollows:

65.1325.3

3=

!

The -3 came from subtractionof the means from groups 1 and 2 in the ANOVA example.

Balkin, R. S.(2008). 14

Understanding Cohen’s d

So, if d = 1.65, then the differencebetween the groups is 1.65 standarddeviation units.

This would be considered a very largeeffect size, as it is greater than .8.

Balkin, R. S.(2008). 15

Cohen’s f

Cohen’s f also expresses effect size instandard deviation units, but does sofor two or more groups.

When conducting an ANOVA, Cohen’s fcan be computed to determine thepractical significance in the differencesamong the groups.

Balkin, R. S.(2008). 16

Cohen’s f

Like the ANOVA, the Cohen’s f will identifythe magnitude of the differences among thegroups, but it will not explain differencesbetween specific groups.

To identify differences between specificgroups, a Tukey post hoc analysis followed byCohen’s d for each pairwise comparisonwould be necessary.

Balkin, R. S.(2008). 17

Cohen’s f

Cohen (1988) created the followingcategories to interpret f: Small = .10 Medium = .25 Large = .40

Balkin, R. S.(2008). 18

Computing Cohen’s f

31.130.13

)69.1381.41.801(.

325.3)4(

])1.64.2()1.67()1.69()1.66[(

)(

)( 22222

=+++

=

!+!+!+!=

!="

error

j

MSJf

µµ

So, a large effect size was found among the four

groups with an effect size of approximately 1.31

standard deviations.

Balkin, R. S.(2008). 19

Omega squared ω2 and Eta-squared η2

Practical significance is not alwaysmeasured in standard deviation unitsand may be expressed in variance units.

There are mathematical relationshipsbetween effect sizes expressed instandard deviation units and strengthsof association expressed in varianceunits.

2!

Balkin, R. S.(2008). 20


However, when conducting parametric statistics, inwhich the focus of the study is on group differences,it is best practice to express effect size in standarddeviation units as it better compliments thedescriptive data, such as means and standarddeviations.

As a rule of thumb, Cohen’s d and Cohen’s f may bemore informative for ANOVA. However, manystatistical packages provide measures of strength ofassociation, especially η2 and ω2, and so they arewidely used.

Balkin, R. S.(2008). 21


Cohen (1988) created the followingcategories to interpret strength ofassociation: Small = .02 Medium = .13 Large = .26

Balkin, R. S.(2008). 22

Eta-squared: η2

Eta-squared refers to strength of associationbetween the independent variable(s) and thedependent variable.

It indicates the amount of variance accountedfor in the dependent variable by theindependent variable(s).

If the strength of association is weak, or low,the independent variable(s) have lessmeaning/relevance to the dependent variable.

Balkin, R. S.(2008). 23

Eta-squared: η2

Similar to Cohen’s f , .683 is a verylarge effect size. The IV accounts for63% of the variance in the DV.

68.

8.167

6.142=

1==

TOT

B

SS

SS!

Balkin, R. S.(2008). 24

Omega squared: ω2

The computation of

2!

also uses terms from the

ANOVA computation:

WTOT

WB

MSSS

MSjSS

+

""=

))(1(2!

where SSTOT is the sum of SS B + SSW and j is the

number of groups.

Balkin, R. S.(2008). 25

Omega squared: ω2

61.125.171

625.104

325.38.167

)325.3)(14(6.1142 ==+

!!="

From this statistic, we can conclude that the four

groups of students account for 61% of the variance in

self-efficacy scores .

Balkin, R. S.(2008). 26

Effect size summary

Keep in mind, effect size is alwayscomputed when a statistical test isconducted.

Even if your F-test is not significant, youshould still report a Cohen’s F.However, since a TMC is not conductedfor a non-significant test, no furtheranalysis is necessary.

Effect Size - balkinresearchmethods.com · compute Cohen’s d or Cohen’s f, which measure effect size in standard deviation units. However, they are ... Omega squared ω2 and Eta-squared

Documents