1
The Campbell Collaboration www.campbellcollaboration.org
Computing Effect Sizes for Clustered Randomized Trials
Terri Pigott, C2 Methods Editor & Co-Chair Professor, Loyola University Chicago
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Computing effect sizes in clustered trials • In an experimental study, we are interested in the difference
in performance between the treatment and control group • In this case, we use the standardized mean difference, given
by
T C
p
Y Yds−
= g g
Treatment group mean
Control group mean
Pooled sample standard deviation
2
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance of the standardized mean difference
22 ( )
2( )
T C
T C T C
N N dS dN N N N+
= ++
where NT is the sample size for the treatment group, and NC is the sample size for the control group
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
1 2, ,..., TT T T
NY Y Y
TYgCYg
Overall Trt Mean Overall Cntl Mean
1 2, ,..., CC C C
NY Y Y
2TrtS 2
CntlS2pooledS
3
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
In cluster randomized trials, SMD more complex • In cluster randomized trials, we have clusters such as
schools or clinics randomized to treatment and control • We have at least two means: mean performance for each
cluster, and the overall group mean • We also have several components of variance – the within-
cluster variance, the variance between cluster means, and the total variance
• Next slide is an illustration
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
{ } { }11 1 1,... ,...,T T
T T T Tn m m n
Y Y Y YgggTrt Cluster 1 Trt Cluster mT
{ } { }11 1 1,... ,...,C C
C C C Cn m m n
Y Y Y YgggCntl Cluster 1 Cntl Cluster mC
1TY g T
TmY
g 1CY g C
CmY
g
Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster mC Mean
TYgg
••• •••
CYggOverall Trt Mean
Overall Cntl Mean
Assume equal sample size, n, within clusters
4
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The problem in cluster randomized trials • Which mean is the appropriate mean to represent the
differences between the treatment and control groups? – The cluster means? – The overall treatment and control group means averaged
across clusters? • Which variance is appropriate to represent the differences
between the treatment and control groups? – The within-cluster variance? – The variance among cluster means? – The total variance?
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Three different components of variance • The next slides outline the three components of variance that
exist in a cluster randomized trial • These components of variance correspond to the variances
that we would estimate when analyzing a cluster randomized trial using hierarchical linear models
5
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Pooled within-cluster sample variance • One component of variance is within-cluster variation • This is computed as the pooled differences between each
observation and its cluster mean
2 2
1 1 1 12
( ) ( )T Cm n m n
T T C Cij i ij i
i j i jW
Y Y Y YS
N M= = = =
− + −
=−
∑∑ ∑∑g g
M is number of clusters: M = mT + mC
N is total sample size: N = n mT + n mC = NT + NC with equal cluster sample sizes, n
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
{ } { }11 1 1,... ,...,T T
T T T Tn m m n
Y Y Y YgggTrt Cluster 1 Trt Cluster mT
{ } { }11 1 1,... ,...,C C
C C C Cn m m n
Y Y Y YgggCntl Cluster 1 Cntl Cluster mC
1TY g T
TmY
g 1CY g C
CmY
g
Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster mC Mean
TYgg
••• •••
CYggOverall Trt Mean
Overall Cntl Mean
2WS
Within-cluster variance
6
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Pooled within-group (treatment and control) sample variance of the cluster means • Another component of variation is the variance among the
cluster means within the treatment and control groups • This is computed as the pooled difference between each
cluster mean and its overall group (treatment and control) mean
2 2
2 1 1
( ) ( )
2
T Cm mT T C Ci i
i iB T C
Y Y Y YS
m m= =
− + −=
+ −
∑ ∑g gg g gg
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
{ } { }11 1 1,... ,...,T T
T T T Tn m m n
Y Y Y YgggTrt Cluster 1 Trt Cluster mT
{ } { }11 1 1,... ,...,C C
C C C Cn m m n
Y Y Y YgggCntl Cluster 1 Cntl Cluster mC
1TY g T
TmY
g 1CY g C
CmY
g
Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster mC Mean
TYgg
••• •••
CYggOverall Trt Mean
Overall Cntl Mean
2BS Variance between cluster
means
7
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Total pooled within-group (treatment and control) variance • The total variance is the variation among the individual
observations and their group (treatment and control) overall means
2 2
1 1 1 12
( ) ( )
2
T Cm n m nT T C Cij ij
i j i jT
Y Y Y YS
N= = = =
− + −
=−
∑∑ ∑∑gg gg
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
TREATMENT GROUP CONTROL GROUP
{ } { }11 1 1,... ,...,T T
T T T Tn m m n
Y Y Y YgggTrt Cluster 1 Trt Cluster mT
{ } { }11 1 1,... ,...,C C
C C C Cn m m n
Y Y Y YgggCntl Cluster 1 Cntl Cluster mC
1TY g T
TmY
g 1CY g C
CmY
g
Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster mC Mean
TYgg
••• •••
CYggOverall Trt Mean
Overall Cntl Mean
2TS Total pooled within-group variance
(treatment and control)
8
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Just a little distributional theory • As usual, we will assume that each of our observations
within the treatment and control group clusters are normally distributed about their cluster means, µT
i and µCi with
common within-cluster variance, σ2W, or
2
2
~ ( , ), 1,..., ; 1,...,
~ ( , ), 1,..., ; 1,...,
T T Tij i W
C C Cij i W
Y N i m j n
Y N i m j n
µ σ
µ σ
= =
= =
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Just a little more distributional theory • We also assume that our clusters are sampled from a
population of clusters (making them random effects) so that the cluster means have a normal sampling distribution with means µT
• and µC• and common variance, σ2
B
2
2
~ ( , ), 1,...,
~ ( , ), 1,...,
T T Ti BC C Ci B
N i mN i m
µ µ σ
µ µ σ
=
=g
g
9
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The three component s of variance are related:
2 2 2T W Bσ σ σ= +
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The intraclass correlation, ρ • This parameter summarizes the relationship among the three
variance components • We can think of ρ as the ratio of the between cluster variance
and the total variance • ρ is given by
2 2
2 2 2B B
B W T
σ σρ
σ σ σ= =
+
10
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Why is the intraclass correlation important? • Later, we will see that we need the intraclass correlation to
obtain the values of our standardized mean difference and its variance
• Also, if we know the intraclass correlation, we can obtain the value of one of the variances knowing the other two. For example:
2 2
2 2
(1 ) /
(1 )W B
W T
σ ρ σ ρ
σ ρ σ
= −
= −
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
So far, we have • Discussed the structure of the data • Seen how to compute three different sampling variances:
S2W, S2
B, and S2T
• Discussed the underlying distributions of our clustered data • Seen how the variance components in the distributions of our
data are related to one another • Next we will see how to estimate the variance components
using our sampling variances • (Yes, we are building up to the effect sizes…)
11
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator of σ2W
• As we might expect, we estimate σ2W using our estimate of
the within-cluster sampling variance, or,
2 2ˆW WSσ =
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator of σ2B
• Unfortunately, the estimate of σ2B is not as straightforward as
for σ2W
• This is due to the fact that S2B includes some of the within-
cluster variance. • The expected value for S2
B is
22 WB n
σσ +
12
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Thus, an estimator for σ2B is
22 2ˆ WB B
SSn
σ = −
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
And, given prior information, an estimate of σ2T is
2 2 21ˆT B WnS Sn
σ−⎛ ⎞= + ⎜ ⎟
⎝ ⎠
13
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Now, we are ready for computing effect sizes • There are 3 possibilities for computing an effect size in a
clustered randomized trial • These correspond to the 3 different components of variance
we have been discussing • The choice among them depends on the research design
and the information reported in the other studies in our meta-analysis
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The most common: δW • This effect size is comparable to the standardized mean
difference computed from single site designs
T C
WW
µ µδ
σ−
= g g
14
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
We estimate δW using dW
T C
WW
Y YdS−
= gg ggTrt & Cntl means
Pooled within-cluster variance
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance of dW • Variance of dW depends on the intraclass correlation, sample
sizes, and dW
{ }21 ( 1)
1 2( )
T CW
W T C
dN N nV dN N N M
ρρ
⎛ ⎞⎛ ⎞+ + −= +⎜ ⎟⎜ ⎟− −⎝ ⎠⎝ ⎠
When SB is 0 (no between-group variation among the cluster means), the variance of dW is equal to the variance for the standardized mean difference in a single-site study
15
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
A second estimator: δT • This estimator uses the total variation in the denominator • We may compute this effect size when the other studies in
our meta-analysis are also multi-site studies • The general form is:
T C
TT
µ µδ
σ−
= g g
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
There are two ways to compute δT • Method 1: dT1
– Used when the study reports S2B , the variance of the cluster
means, and S2W , the pooled within-cluster variance
• Method 2: dT2 – Used when we the study reports the intraclass correlation, and
S2T , the total variance
16
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator dT1 : When we have S2B and S2
W
1
2 2
, withˆ
1ˆ
T C
TT
T B W
Y Yd
nS Sn
σ
σ
−=
−⎛ ⎞= + ⎜ ⎟⎝ ⎠
gg gg
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of dT1 (a little messy)
{ }
[ ]
1
2 2 2212 2
(1 ( 1) )
1 ( 1) ( 1) (1 )2 ( 2) 2 ( )
T C
T T C
T
N NV d nN N
n n dn M n N M
ρ
ρ ρ
⎛ ⎞+= + − +⎜ ⎟⎝ ⎠
⎡ ⎤+ − − −+⎢ ⎥
− −⎢ ⎥⎣ ⎦
17
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator dT2 : When we have S2T and ρ
22( 1)1
2
T C
TT
Y Y ndS N
ρ⎛ ⎞− −= −⎜ ⎟ −⎝ ⎠
gg gg
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance of dT2 (also messy)
{ }
[ ]
2
2 222
(1 ( 1) )
( 2)(1 ) ( 2 ) 2( 2 ) (1 )2( 2) ( 2) 2( 1)
T C
T T C
T
N NV d nN N
N n N n N ndN N n
ρ
ρ ρ ρ ρρ
⎛ ⎞+= + − +⎜ ⎟⎝ ⎠
⎛ ⎞− − + − + − −⎜ ⎟⎜ ⎟− − − −⎝ ⎠
18
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
A third estimator based on S2B
• We use this when the treatment effect is defined at the level of the clusters, or when the other studies in our meta-analysis have been analyzed using cluster means as the unit of analysis
• Though it might not be of general interest, we can use this effect size to obtain other effect sizes of interest
• There are two estimates of this effect
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator dB1 : When we have S2B and S2
W
1
22
, withˆ
ˆ
T C
BB
WB B
Y Yd
SSn
σ
σ
−=
= −
gg gg
19
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of dB1 (a little messy)
{ }
[ ]
1
2 2212 2 2 2
1 ( 1)
1 ( 1) (1 )2( 2) 2( )
T C
B T C
B
m m nV dm m n
nd
M n N M n
ρρ
ρ ρρ ρ
⎛ ⎞+ + −= +⎜ ⎟⎝ ⎠
⎡ ⎤+ − −+⎢ ⎥
− −⎢ ⎥⎣ ⎦
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Estimator dB2 : When we have SB and ρ
21 ( 1)T C
BB
Y Y ndS n
ρρ
− + −= gg gg
20
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of dB2 (a little messy)
{ }
( )
2
22
1 ( 1)
1 ( 1)2( 2)
T C
B T C
B
m m nV dm m n
n dM n
ρρ
ρ
ρ
⎛ ⎞+ + −= +⎜ ⎟⎝ ⎠
+ −
−
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
IMPORTANT: Can compute any δ from any other effect size and ρ
1 1
1
TW B
T B W
δρδ δ
ρ ρ
δ δ ρ δ ρ
= =− −
= = −
21
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
And, can compute the variance of the effect size using the same transformations
{ } { } { }
{ } { } { }1 1
(1 )
TW B
T B W
VarVar Var
Var Var Var
δρδ δ
ρ ρ
δ δ ρ δ ρ
= =− −
= = −
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Where are we now? • We have looked at the structure of the data • We have examined the different components of variation • We have defined estimators for the different components of
variation • We have outlined the computation of three different effect
sizes, dW , dT , and dB and their associated variances
22
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
But we all know it is not that simple: Some common problems • We have unequal cluster sample sizes
– There are formulas for unequal cluster sample sizes that are much more complex
– We can assume equal cluster sizes since most studies attempt to use equal cluster sizes in the design
• We don’t know the intraclass correlation coefficient – We can estimate it from a number of sources listed in the
References – Several researchers have provided estimates from other studies
and from large-scale national samples
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Let’s look at some examples
23
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
From the analysis section
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Another clue that we have cluster means and sds reported
24
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The ICCs are also reported
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
But I could not find sample sizes within clusters so looked for another report on the intervention
25
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
So, what do we have in this example? • We have ρ for SAT Math of 0.72 • There are 10 matched-pairs of schools, with NT = 976, and
NC = 738 (conservative common n = 73) • We have the mean of the school (cluster) means and SDs for
the treatment and control group at the level of school
26
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
We have all the information needed to compute dB2
2 22
2
(7.26) (7.48) 54.33, 7.372
82.33 78.77 1 (73 1)0.727.37 73(0.72)
0.48 1.005 0.48
B B
B
S S
d
+= = =
− + −=
= =
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
To compute Var{dB2 }
{ }
( )
210 10 1 (73 1)(0.72)10*10 (73)(0.72)1 (73 1)0.72 (0.48)2(20 2) (73)(0.72)20 52.84 (52.84)(0.48) 0.201 0.013100 52.56 1892.160.214
BV d + + −⎛ ⎞= ⎜ ⎟⎝ ⎠
+ −+
−
⎛ ⎞⎛ ⎞= + = +⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠
=
27
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Say we want dW in our meta-analysis
0.720.481 1 0.72
0.48(1.60) 0.77
W Bd d ρρ
= =− −
= =
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of dW follows the same transformation
{ } { }
{ }
2
1
0.720.214(1 0.72)
0.214(2.57) 0.550.74
W B
W
Var d Var d
SD d
ρρ
⎛ ⎞= ⎜ ⎟⎜ ⎟−⎝ ⎠
⎛ ⎞= ⎜ ⎟−⎝ ⎠= =
=
28
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
If instead, we want dT
{ } { }
{ }
0.48 0.720.48(0.85) 0.41
(0.214)(0.72) 0.1540.392
T B
T W
T
d d
Var d Var d
SD d
ρ
ρ
= =
= =
=
= =
=
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Some observations on this example • The intraclass correlation is large for this measure so we
have a lot of between school variability • Thus, it is not surprising that none of the effect sizes we
calculated are significantly different from zero • In this example, we have the intraclass correlation reported,
but we may not be so lucky
29
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Another example
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
We might have individual level info in Table 2
30
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
What information do we have? • mT = 14 schools in the treatment group, mC = 10 schools in
the control group • For treatment group, sample size average is nT = 3429/14 =
244.9. For control group, sample size average is nC = 1047/10 = 104.7
• We have individual level means and standard deviations BUT • We don’t have the intraclass correlation, or other estimates
of the variance
31
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
To compute effect sizes, we need either 2 estimates of the variance, or 1 estimate and ρ
• We can only compute the estimate of S2T from our 2
estimates of the individual level sds • We will need to find an estimate for ρ from another source • From: Murray & Blitstein (2003). Methods to reduce the
impact of intraclass correlation in group-randomized trials. Evaluation Review, 27: 79-103
• The upper bound of the ICC for youth cohort studies with outcomes focused on diet is 0.0310
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Since we have individual level stats:
2 22
2
(3429 1)(0.52) (1047 1)(0.25) 0.223429 1047 2
3.40 2.16 2(104 1)0.03110.47 208 2
2.64 0.97 2.59
T
T
S
d
− + −= =
+ −
− −= −
−
= =
32
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
The variance of dT
{ }
[ ]
{ }
2
2 22
2
2
1456 1040 (1 (104 1)*0.031)1456*1040
(2496 2)(1 0.031) 104(2496 2*104)0.031 2(2496 2*104)0.031(1 0.031)2.592(2496 2) (2496 2) 2(104 1)0.031
0.007 2.59 (0.0002) 0.00830.091
T
T
V d
SD d
+⎛ ⎞= + − +⎜ ⎟⎝ ⎠
⎛ ⎞− − + − + − −⎜ ⎟⎜ ⎟− − − −⎝ ⎠
= + =
=
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
If we want dW
{ } { }
{ }
2.59 2.641 1 0.031
0.0083 0.00861 1 0.031
0.093
TW
TW
W
dd
Var dVar d
SD d
ρ
ρ
= = =− −
= = =− −
=
33
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
What about other types of effect sizes, like odds-ratios?
• We do not yet have the same research for odds ratios • What we can do is inflate the odds ratios from a clustered
trial using recommendations from the Cochrane Handbook
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Recommendations from the Cochrane Handbook • We first compute the design effect given by
1 + (nave – 1) ρ where nave is the average cluster size, and ρ is the intraclass
correlation coefficient • We divide the total sample sizes and number of events by
the design effect to obtain the corrected numbers to compute the odds-ratio
• We adjust the variance by computing the variance of the log-odds ratio using the original sample sizes and dividing by the square root of the design effect
34
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
From the Positive Action program
No smoking Smoked
Intervention 937 4.0% of 976 = 39
Control 682 7.6% of 738 = 56
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
And the intraclass correlation coefficient
35
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Adjusted values • Average cluster size: (738+976)/20 = 85.7 • Design effect: 1 + (85.7 – 1) 0.05 = 5.235
No smoking Smoked
Intervention 937/5.235=178.99 4.0% of 976 = 39/5.235 = 7.45
Control 682/5.235=130.28 7.6% of 738 = 56/5.235 = 10.7
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
Variance for the log-odds ratio • OR = (937*56)/(682*39) = 1.97 • lnOR = ln(1.97) = 0.68 • SE2(lnOR) = 1/937 + 1/56 + 1/682 + 1/39 = 0.046 • SE(lnOR) = 0.214 • Adjusted SE(lnOR) = 0.214/√5.235 = 0.094
36
Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org
References Hedges, L. V. (2007). Effect sizes in cluster-randomized
designs. Journal of Educational and Behavioral Statistics, 32, 341-370.
Hedges, L. V. & Hedberg, E. C. (2007). Intraclass correlation values for planning group randomized experiments in education. Educational Evaluation and Policy Analysis, 29, 60-87.
Cochrane Handbook: http://www.cochrane-handbook.org/
The Campbell Collaboration www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway
E-mail: [email protected]
http://www.campbellcollaboration.org