Testing treatment by covariate interactions when treatment varies within subjects

Psychological Methods1996. Vol. I , No. 4, 366-378

Copyright 19% by the American Psychological Association, Inc.1082-989 X,'96/$3.00

Testing Treatment by Covariate Interactions When TreatmentVaries Within Subjects

Charles M. Judd and Gary H. McClellandUniversity of Colorado at Boulder

Eliot R. SmithPurdue University

In contrast to the situation when an independent or treatment variable varies be-tween subjects, procedures for testing treatment by covariate interactions are notcommonly understood when the treatment varies within subjects. The purpose ofthis article is to identify analytic approaches that test such interactions. Two designscenarios are discussed, one in which the covariate is measured only a single timefor each subject and hence varies only between subjects, and the other in which thecovariate is measured at each level of the treatment variable and hence varies bothwithin and between subjects. In each case, alternative analyses are identified andtheir assumptions and relative efficiencies compared.

An issue that arises with some frequency in data

analysis in psychological research concerns the rela-

tionship between some measured variable and the de-

pendent variable and whether that relationship de-

pends on or varies across levels of a manipulated or

experimental independent variable. For instance, in a

clinical intervention study, we might randomly assign

patients to one of two conditions, either a treatment

intervention or a placebo intervention control condi-

tion. Prior to treatment, we measure a characteristic of

the patients, probably focusing on the prior course and

severity of their illness. Following the treatment, we

assess the outcome variable of symptom severity. The

primary question of interest, of course, is whether the

outcome variable is affected by the manipulated treat-

ment: Did the treatment make a difference on subse-

quent symptom severity? Additionally, however, we

may well want to know whether the relationship be-

tween the treatment and posttreatment symptom se-

verity depends on the patient's pretreatment course of

Charles M. Judd and Gary H. McClelland, Department of

Psychology, University of Colorado at Boulder; Eliot R.Smith, Department of Psychological Sciences, Purdue Uni-versity.

This work was partially supported by National Institute ofMental Health Grant R01 MH45049.

Correspondence concerning this article should be ad-dressed to Charles M. Judd, Department of Psychol-ogy, University of Colorado, Boulder, Colorado 80309.Electronic mail may be sent via the Internet to [email protected].

illness. It may be, for instance, that the treatment's

effect is greater for patients whose pretreatment

symptoms were relatively severe. Equivalently. it may

be that posttreatment symptom severity is less well

predicted by pretreatment course of illness in the case

of patients in the intervention condition than in the

case of patients in the control condition.

The pretreatment measure of illness course is typi-

cally called a covariate. The analysis that is of interest

is an analysis of covariance (ANCOVA), including

the treatment by covariate interaction (Judd & Mc-

Clelland, 1989). The two questions of interest are (a)

Is there an overall treatment main effect? and (b) Is

there a Treatment x Covariate interaction? If the in-

teraction is significant, it indicates that the covari-

ate: outcome variable relationship depends on the

treatment variable. Equivalently, it suggests that the

effect of the treatment on the outcome variable de-

pends on the level of the covariate.

The analysis is readily conducted using multiple

regression, making the standard assumption that er-

rors or residuals are independently sampled from a

single normally distributed population. Assume that

y. is the outcome variable, Z; is the covariate, and X,

is the contrast-coded (Judd & McClelland, 1989;

Rosenthal & Rosnow, 1985) treatment variable. One

estimates two least squares regression models:

7, = (30

and

Y, = p0 +

In the first equation, represents the magnitude of

366

COVARIATE INTERACTIONS 367

the treatment difference adjusting for the covariate. In

the second equation, the slope of the product term, (33,

represents the effect of the Treatment x Covariate

interaction and is interpreted as the difference in the

magnitude of the adjusted treatment difference per

unit increase in the covariate.

A number of researchers have examined the statis-

tical properties of this test of the Treatment x Covari-

ate interaction, examining in particular the conse-

quences of violating the assumption of homogeneity

of error variance in each of the two treatment groups

(Alexander & DeShon, 1994; DeShon & Alexander,

1996; Dretzke, Levin, & Serlin, 1982). The general

conclusion of this work is that violations of error ho-

mogeneity can lead to serious conclusion errors, par-

ticularly with unequal sample sizes in the two groups.

The analysis that we have just reviewed and the

literature examining its underlying assumptions have

exclusively focused on the case in which the treatment

variable is between subjects. In many instances in

psychological research, however, within-subjects de-

signs are used, and procedures for testing Covariate x

Treatment interactions in this case are not commonly

known. The purpose of this article is to identify ana-

lytic alternatives in this within-subject case and to

compare their relative efficiencies. Our work thus ex-

tends previous work on procedures for testing Covari-

ate x Treatment interactions by focusing exclusively

on designs in which the treatment variable varies

within subjects.

Although there is a literature on the use of an

ANCOVA when the treatment variable is within sub-

jects (e.g., Huitema, 1980; Khattree & Naik, 1995,

Myers, 1979; Winer, Brown, & Michels, 1991), vir-

tually all of this literature has focused simply on test-

ing treatment main effects in the presence of a covari-

ate rather than on testing Treatment x Covariate

interactions. In spite of this lack of attention, this

literature on within-subjects ANCOVA is useful in

that it makes clear that two design alternatives must

be defined when the treatment variable varies within

subjects rather than between. The two alternatives dif-

fer in whether the variation in the covariate is entirely

between subjects or whether the covariate varies

within subjects as well as between them.

To illustrate the first design alternative, consider an

experiment in cognitive psychology in which memory

for two different types of word lists is examined. Each

subject is exposed to both list types, and memory for

each is recorded. Accordingly, list type is the within-

subject manipulated variable.1 Additionally, a prior

measure of verbal ability is administered to all sub-

jects. In this case, one wants to know (a) whether

memory differs as a function of list type and (b)

whether subjects' verbal ability (the covariate) differ-

entially predicts memory for the two list types (alter-

natively and equivalently, whether the difference due

to list type depends on subjects' verbal ability). In this

case, the covariate is measured only a single time for

each subject rather than at each level of the manipu-

lated independent variable. Variation in the indepen-

dent variable is within subjects, but the covariate var-

ies only between them.

As another example of this type of design, consider

a clinical psychologist who is interested in gender

differences in marital satisfaction. Data are gathered

from a number of couples on each spouse's level of

satisfaction. Additionally, the frequency with which

each couple argues in a given time period is observed.

The clinical psychologist is interested not only in

whether the spouse's gender (the within-couple inde-

pendent variable) predicts satisfaction but also in

whether frequency of argumentation (the covariate)

predicts levels of satisfaction differently for the male

and the female spouse.

In the second alternative design, the covariate is

measured at each level of the independent variable,

and thus it varies within subjects as well as between

them. To illustrate this alternative, let us return to the

cognitive psychology study in which subjects are ex-

posed to two types of work lists, and memory for each

is the dependent variable. Now, however, study time

for each of the two lists is allowed to vary, and the

experimenter separately records the two study times

for each subject. In this study, one would like to know

(a) whether memory differs as a function of list type

and (b) whether study time differentially predicts

memory for the two list types.

As another example of this second design alterna-

tive, suppose a social psychologist is interested in

the effect of physical attractiveness on judgments of

an individual's competence and, more specifi-

cally, whether the magnitude of the attractiveness-

competence relationship depends on gender. Subjects

are given dossiers of two target individuals, one male

and one female. Included are photographs of each.

' The design would most appropriately also manipulate

order of the two lists between subjects. Although important

from an experimental design point of view, this counterbal-

ancing has no effect on the issue we are addressing.

368 JUDD, MCCLELLAND, AND SMITH

Subjects rate the attractiveness and competence of

each target individual. Gender is thus the within-

subject independent variable, and the primary ques-

tion of interest is whether attractiveness (the covari-

ate) is differentially predictive of judged competence

for the male versus the female target person.

As a final example, a researcher is interested in the

effects of alcohol on athletic performance. Subjects

come into the lab on two different days and are given

either a dose of alcohol or a placebo (the within-

subject independent variable). Twenty minutes later

they complete a strenuous performance task. Perfor-

mance on this task is known to normally be a function

of level of exertion in the previous 24 hr. Accord-

ingly, levels of exertion for the previous 24 hr (the

covariate) are measured on each of the 2 days. Not

only is the researcher interested in the alcohol versus

placebo difference on performance, but he or she is

also interested in whether levels of previous exertion

relate to performance more or less strongly following

alcohol consumption than following placebo con-

sumption.

To describe the two designs a bit more precisely, let

Yj, and Y2i be the measured dependent variables in

each of the two levels of the within-subject indepen-

dent variable. In the first design alternative, let X(

represent the single measure of the covariate. The

Covariate x Treatment interaction question is whether

the slope when Yn is regressed on X/ differs from the

slope when Y2i is regressed on X / . In the second design

alternative, the covariate is measured twice for each

subject, once for each level of the independent vari-

able, Xji and X2i. The Covariate x Treatment interac-

tion question of this second study is whether the slope

when Y,i is regressed on Xl: differs from the slope

when Y2i is regressed on X2i.

In the previous paragraphs we have intentionally

discussed these interactions as differences in slopes

rather than as differences in the "magnitude of rela-

tionships." This preference is because the raw slope

associated with a contrast-coded treatment variable

indicates the magnitude of the mean difference be-

tween the two treatments and because the slope for the

Treatment x Covariate interaction indicates changes

in that mean treatment difference per unit change in

the covariate. If we make assumptions about the

equivalence of various variances, differences in

slopes are equivalent to differences in correlations.

An extensive literature exists on testing differences

between correlations, both in the between-subjects

case and in the two within-subject cases we have iden-

tified (e.g., Meng, Rosenthal, & Rubin. 1992; Olkin,

1966; Olkin & Finn, 1990). We make use of this

literature when we add restrictive assumptions about

equality of variances. However, we develop more

general procedures that permit us to test interactions

as differences in raw slopes in the within-subject case.

Design Alternative 1:Between-Subjects Covariate

Figure 1 presents the analytic model in which we

are interested. We have two outcome measures from

the two treatment conditions, YH and Y2i. Scores on

these two measures are assumed to be dependent be-

cause of the fact that they come from the same sub-

jects. This dependence is captured in Figure I by their

common dependence on the single covariate X; and by

the fact that their residuals, e,, and s2l, are assumed to

correlate.

The test of treatment differences, commonly con-

ducted through a within-subjects analysis of variance

(ANOVA), is whether the mean of 7,, differs from the

mean of Y2r The question that is explored in this

article is whether the two slopes in the model, px v

and (31V( differ. The null hypothesis is that they do

not. Three alternative analytic strategies are pre-

sented. They differ in the assumptions they make and

in their relative efficiency. Following our presentation

of the three alternative analyses, we discuss their rela-

tive efficiency.

Analysis 1: Structural Equation Models

The first approach estimates the parameters of the

model in Figure 1 using an iterative algorithm based

on a maximum likelihood minimization criterion. A

variety of different programs can be used to accom-

plish this estimation (e.g., LISREL: Joreskog & Sor-

bom, 1993; EQS: Bentler, 1993; PROC CALLS in

SAS: SAS Institute Inc., 1990). Two models can be

Figure 1. Design Alternative 1; Between-subjects covari-

ate.


estimated, using the sample variance/covariance ma-

trix among the three variables as input. In the first, six

different parameters are estimated: the variance of Xt,

the variances of the two residuals, el; and e,2i, their

covariance, and the two slopes, $YX and $YJC Given

the six sample variances and covariances among the

three variables, X,-, Yn, and Y2i, this model is just

identified. The second estimated model imposes an

equality constraint on the two slopes. Since this sec-

ond model is overidentified, one can test the quality of

its fit to the sample data using a xfi> goodness-of-fit

test, under the assumptions of multivariate normality

and large sample size. This goodness-of-fit test is

equivalent to a test of the difference between the two

models, asking whether the fit of the sample data to

the model significantly deteriorates when the equality

constraint on the two slopes is imposed.

Analysis 2: Regression Approach

We could use ordinary linear regression to estimate

the two slopes pr x and py x, first regressing Y]t on X,,

and then regressing Y2i on X,. In the Appendix we

show that if we regressed the difference between the

two /s (Yji — Y2i) on X,-, the resulting slope would be

equal to the difference between the two estimated

slopes of interest, that is, bIY - y.f = bY* — bYx.

Accordingly, a test of whether the two slopes differ is

equivalently a test of whether the slope of X, differs

from zero when (Yu — Y2i) is the criterion variable.

The variance of this slope is also given in the Appen-

dix. The squared slope divided by its variance is dis-

tributed as an f (1, n - 2) statistic under the null hy-

pothesis. The assumptions underlying this test are that

the residuals to this criterion difference score are in-

dependently sampled from a single normally distrib-

uted population. This assumption is also assumed by

the standard within-subject ANOVA that tests wheth-

er the mean of YIt differs from the mean of Y2i.

In the one source we found that discusses Covariate

x Treatment interactions in this within-subject design

(Khattree & Naik, 1995), the ANCOVA approach that

is recommended, using PROC GLM in SAS, is

equivalent to this regression-based estimation.

Analysis 3: Test of Dependent Correlations

An extensive literature now exists on testing the

equality of "dependent" correlations. Following pro-

cedures outlined by Olkin (1966) and Olkin and Finn

(1990), one can test whether the correlation between

YH and X, (px) is equal to the correlation between Y2i

two residuals in the model of Figure 1, el; and e2/,

have equal variances, then this is equivalent to testing

the equality of the two slopes.

The Olkin and Finn (1990) procedure uses the large

sample result that the variance of the difference be-

tween two dependent correlations, pY,x ~ Piyt' can ̂

estimated as

- (

where

= - (rr,Y, ~

*4

and X, (py,x). If one makes the assumption that the

The squared difference between the two correlations,

divided by its estimated variance, is distributed as x^i,

under the null hypothesis that the true difference

equals zero.

Efficiency of Three Analysis Alternatives

The third analysis alternative makes the most re-

strictive assumptions of the three. In order to interpret

the resulting XQ> as a test of whether the two covariate

slopes differ from one another, it is necessary to as-

sume that the variances of the residuals, e^- and E2/>

are equal. Under the null hypothesis of equal slopes,

this is equivalent to assuming that the variances of the

two dependent variables, Yu and Y2i, are equal to each

other. Because the first and second analysis alterna-

tives do not make this assumption, they are useful in

a much wider variety of situations in which one is

interested in testing slope differences.

To compare the relative efficiency of the three

analysis alternatives, we met this equal variance as-

sumption of Alternative 3, assuming that the variance

of the two residuals was equal to 1.0. We also set the

variance of X,- at 1.0. Under these conditions, we ex-

amined the power of each of the three alternative tests

by varying two factors: (a) We varied the true value of

the slopes, $YiX and [J^, and the magnitude of their

difference. This factor varies the magnitude of the

effect size tested. The specific parameter values for

the slopes included in the power calculations were (0,

0.2), (0, 0.4), (0, 0.6), (0, 0.8), (0.2, 0.4), (0.2, 0.6),

(0.2, 0.8), (0.4, 0.6), (0.4, 0.8), and (0.6, 0.8). (b) We

also varied the direction and magnitude of the depen-

dence in the residuals to the dependent variables,

varying by increments of 0.2 between -0.8 and 0.8. In

within-subjects designs, one can generally assume

that the dependent variables are positively dependent


on each other. However, there are situations in which

a negative residual correlation might be expected.

Since our primary goal was to compare the relative

power of the three approaches under various effect

sizes and degrees of dependence, we chose not to vary

sample size as well. Sample size in all power calcu-

lations was set at 100.2 Power was calculated in each

case by calculating the expected value of the test sta-

tistic for each combination of effect size and degree of

dependence and then using this expected value as the

noncentrality parameter to estimate the proportion of

sample test statistics that would exceed the critical

value of the test statistic, given the null hypothesis

with an alpha set at .05. The expected values of xf]>

statistic from the structural equation analysis were

calculated using PROC CALIS in SAS. The expected

values of the test statistics in the other two cases were

derived analytically using Mathematica (Wolfram,

1991).3

Power values for the three analytic alternatives are

given in Table 1. The most striking conclusions from

these values are the following: (a) There is very little

difference in power between the three alternatives for

a given level of effect size and dependence. This is

seen most clearly from the graphed power values in

Figure 2. We have graphed only those values from

Table 1 for which one of the two slopes equals 0. The

similarity of the resulting power curves is striking, (b)

Not surprisingly, power dramatically increases as the

effect size (i.e., the true difference between the

slopes) increases, (c) There is a large monotonic in-

crease in power as the residuals to the dependent vari-

ables (and hence the dependent variables themselves)

become more positively related. In the case of the

regression-based analytic approach, the reason for this

increase can be seen by examining the variance of the

estimate provided in the Appendix. That variance de-

creases as the covariance between the residuals in-

creases.

A somewhat more subtle conclusion from these

power values concerns a slight inconsistency in re-

sults between the Olkin and Finn (1990) approach and

the other two. Holding constant the magnitude of de-

pendency, the structural equation and regression-

based approaches show identical power for each

effect size (difference between the two slopes) regard-

less of the absolute magnitude of the slopes them-

selves. For instance, given zero residual dependence,

the power of the regression-based approach for true

slopes of 0.4 and 0 is 0.80. It is also equal to 0.80

when the two slopes are 0.6 and 0.2 and when they are

0.8 and 0.4. On the other hand, as the absolute mag-

nitude of the two slopes increases, power under the

Olkin and Finn (1990) analysis shows a slight de-

crease given the same effect size, particularly at more

positive values of residual dependency. As a result,

the Olkin and Finn analysis becomes less powerful

than the other two when the values of the two slopes

are large and residual dependence is high. For in-

stance, given true slopes of 0.8 and 0.6 and residual

dependence of 0.8, the power of the structural equa-

tion modeling (SEM) test is 0.867, that of the regres-

sion-based approach is 0.879, and that of the Olkin

and Finn (1990) approach is 0.702. Additionally, of

course, the Olkin and Finn approach is more restric-

tive in that it makes the equal residual variance as-

sumption.

Design Alternative 2: Within-Subject Covariate

Figure 3 presents the analytic model for the second

design alternative. In this design, we have two out-

come variables, Yn and Y2I, one from each treatment

condition. Additionally, we have two measures of the

covariate, Xti and X2i, again one from each treatment

condition. The residuals to the two outcome measures

are assumed to be dependent, as are the two covariate

measures.

The issue of present interest is whether the rela-

tionship between the covariate and the outcome de-

pends on condition. Thus, we want to test the null

hypothesis that the two within-condition covariate-

outcome partial slopes in this model are equivalent,

that is pyiX] = 3yA.

We assume that the two crossed partial slopes, py x

and (By x are both equal to zero. That is, we assume

that the covariate from one condition is unrelated to

the outcome variable from the other, once the covari-

ate in that other condition is controlled. In terms of the

word-memory and study-time example that we used

earlier to illustrate this design alternative, we are as-

suming that study time for the first list has no effect

on memory for the second list, once second-list study

time is controlled. Although this seems a reasonable

2 Note that this sample size is on the small side for the

asymptotic assumptions underlying the structural equation

modeling analysis and the Olkin and Finn (1990) procedure,

although calculated Type I error rates, when the null hy-

pothesis of no slope difference was known to be true, were

found to equal .05 for all three tests.3 Both the SAS and Mathematica codes are available

from Charles M. Judd.


Table 1

Design 1: Power of Tests of Covuriate—Treatment Interaction

Slope

Co-variance 0.0,0.2 0.0,0.4 0.0,0.6 0.0,0.8 0.2,0.4 0.2,0.6 0.2,0.8 0.4,0.6 0.4,0.8 0.6,0.8

Analysis 1: Structural equation models

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

0.182

0.198

0.220

0.249

0.288

0.346

0.437

0.594

0.867

0.181

0.198

0.220

0.248

0.288

0.347

0.440

0.600

0.879

0.546

0.594

0.650

0.715

0.788

0.867

0.941

0.989

0.999

0.551

0.600

0.658

0.724

0.800

0.879

0.951

0.993

0.999

0.867

0.901

0.933

0.961

0.982

0.994

0.999

0.999

0.999

0.879

0.913

0.944

0.970

0.987

0.997

0.999

0.999

0.999

0.981

0.989

0.995

0.998

0.999

0.999

0.999

0.999

0.999

Analysis 2:

0.987

0.993

0.997

0.999

0.999

0.999

0.999

0.999

0.999

0.182

0.198

0.220

0.249

0.288

0.346

0.437

0.594

0.867

Regression

0.181

0.198

0.220

0.248

0.288

0.347

0.440

0.600

0.879

0.546

0.594

0.650

0.715

0.788

0.867

0.941

0.989

0.999

approach

0.551

0.600

0.658

0.724

0.800

0.879

0.951

0.993

0.999

0.867

0.901

0.933

0.961

0.982

0.994

0.999

0.999

0.999

0.879

0.913

0.944

0.970

0.987

0.997

0.999

0.999

0.999

0.182

0.198

0.220

0.249

0.288

0.346

0.437

0.594

0.867

0.181

0.198

0.220

0.248

0.288

0.347

0.440

0.600

0.879

0.546

0.594

0.650

0.715

0.788

0.867

0.941

0.989

0.999

0.551

0.600

0.658

0.724

0.800

0.879

0.951

0.993

0.999

0.182

0.198

0.220

0.249

0.288

0.346

0.437

0.594

0.867

0.181

0.198

0.220

0.248

0.288

0.347

0.440

0.600

0.879

Analysis 3: Test of dependent correlations

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

0.185

0.202

0.224

0.252

0.293

0.351

0.442

0.596

0.860

0.570

0.617

0.672

0.734

0.803

0.876

0.943

0.987

0.999

0.897

0.924

0.949

0.970

0.986

0.995

0.999

0.999

0.999

0.990

0.994

0.997

0.999

0.999

0.999

0.999

0.999

0.999

0.183

0.199

0.219

0.245

0.282

0.336

0.420

0.566

0.831

0.556

0.597

0.646

0.704

0.769

0.842

0.917

0.976

0.999

0.875

0.902

0.928

0.952

0.972

0.987

0.996

0.999

0.999

0.180

0.193

0.210

0.233

0.264

0.310

0.384

0.514

0.775

0.536

0.569

0.610

0.660

0.720

0.791

0.872

0.951

0.996

0.176

0.186

0.199

0.216

0.242

0.280

0.341

0.455

0.702

Note. Calculated for different true covariate slopes and different residual covariances.

assumption, it should be tested before proceeding to

test whether the two within-condition covariate-

outcome partial slopes are equivalent.4

As in the first design alternative, there are three

analytic approaches that we examine; one involving

structural equation modeling, a second involving or-

dinary least squares regression, and a third testing

differences between dependent correlations. We com-

pare the assumptions and efficiency of these ap-

proaches following their exposition.

Analysis 1: Structural Equation Models

The first approach estimates the parameters of the

model of Figure 2 using structural equation estimation

and then tests the goodness of fit of the model when

an equality constraint is imposed on the two slopes of

interest. Again, two models can be estimated, using

the sample variance/covariance matrix among the four

variables as input. In the first, 10 different parameters

are estimated: the variances and covariance of X,, and

4 This assumption is actually only required by the second

and third analytic alternatives that we present. Within the

structural equation modeling approach, one can estimate the

crossed slopes while forcing the within-condition slopes to

be equivalent. Nevertheless, it seems reasonable to make the

assumption throughout.


a o - O T f r - i o r - l ^ i O W o o * O ^ ; < N O r 4 ^ > o

0 O O O C J O O O O O O O O O O

1 I I 1 I I I I I I I I I I I I I

ANALYSIS 1: SEM

A SLOPES .0 .8

-•-O SLOPES .0-6

—O SLOPES .0 .4

D— SLOPES .0 .2

ANALYSIS 2: REG

ANALYSIS 3: CORR

Ptf

a£Oa.

Oc-

00 \O Tj- CJ O <N Tt •* OO

c i o o o o o c i d

COVARIANCE

Figure 2. Power of analyses for Design 1. SEM = structural equation models; REG = regression

approach; CORR = test of dependent correlations.

X-,,; the variances and covariance of the two residuals,

£,, and E2/; and Ihe four slopes, Pj"|X,: 3i-,x,. Py2.v,' and

3x,v2. Even though we assume that the two crossed

partial slopes (py,*, and Py2x,) are equal to zero in the

population, they should be estimated from the sample

data. This model is just identified. The second esti-

mated model imposes an equality constraint on the

two covariate-outcome slopes of interest, P^x, ar|d

$YX- The xf i ) goodness-of-fit statistic that results

from this overidentified model tests whether the

equality constraint on the two slopes leads to a sig-

nificantly less well fitting model, under the assump-

tions of multivariate normality and large sample size.

Analysis 2: Regression Approach

As in the first design, we could use ordinary linear

regression to estimate the four slopes in this model,

regressing each ¥t on the two covariates. If we make

the assumption that the two covariates have equal

variances, then the expected difference in the two

slopes of interest, pK|X] - $Y,x.> can be shown to equal

twice the expected slope in two different simple re-

gressions. First, if we regress the difference between

the two outcome measures (Ytl - Y2l) on the sum of

the two covariates (Xl: + X,,), the resulting expected

slope, P (K] _ y M V | + x,)< 's equal to half the difference


Figure 3. Design Alternative 2: Within-subject covariate.

between the two slopes of interest. Equivalently, if we

regress the sum of the two outcome measures (Y,, +

Y2i) on the difference between the two covariates (X,,

- X2/), the resulting expected slope, P(y] + Y^XI _ x^,

has the same value. These equivalences are shown in

the Appendix.5 Accordingly, given the assumption of

equal covariate variances, the two simple slopes from

these two regression models provide two tests of the

null hypothesis that the two slopes of interest are of

equal magnitude. Thus, we really have two tests from

the least squares regression approach, one when the Y

difference is regressed on the X sum, and one when

the Y sum is regressed on the X difference. We refer

to these as Analyses 2a and 2b in the following dis-

cussion.

Analysis 3: Tests of Dependent Correlations

The regression-based analyses rely on the assump-

tions that the variances of the covariates are equal and

that the crossed slopes are zero. If, additionally, we

assume that the variances of the two residuals (e,, and

£2;) are equal, then a test of the equality of the two

slopes of interest is equivalent to a test that the cor-

relations between Yn and X,, and between Y2i and X2I

are equal. Again, Olkin and Finn (1990) provided an

asymptotic xfi) test f°r tne equivalence of such de-

pendent correlations.

Efficiency of Analysis Alternatives

In order to compare the relative efficiency of these

analysis alternatives, we adopted the same approach

that we used in the first design in which there was

only a single covariate, making the most stringent

assumptions required by any of the alternatives. Spe-

cifically, we assumed that the variances of the two

covariates, X/r and X2l, were equivalent as were the

variances of the two residuals, E I ; and s2i. These as-

sumptions are both required if the third analysis al-

ternative, involving a test of a difference between de-

pendent correlations, is to be informative about the

difference between two dependent slopes. The first

equivalence, namely that the two covariates have

equal variances, is required by the regression-based

analyses. The structural equation approach makes no

assumptions about variance equivalencies.

The variances of both the covariates and the residu-

als were set at 1.0. Sample size was fixed at 100.

Power of each of the analysis alternatives was calcu-

lated for the same set of true slopes and the same set

of residual covariances as were used in the power

calculations for the first design. Additionally, the co-

variance between the two covariates was assumed to

always equal the covariance between the two residu-

als in the model. Thus, the same direction and mag-

nitude of dependence was assumed to be found be-

tween the two covariates and between all other

sources of variation in the two outcome measures.

The power values for the structural equation ap-

proach, the two regression approaches, and the Olkin

and Finn (1990) approach are given in Table 2. The

values are graphed in Figure 4 for cases for which one

of the two slopes equals 0.0.

What is striking about these power results is that

they are quite different across the alternative analyses.

This is particularly surprising given the consistency of

the three analytic approaches in the first design con-

sidered, for which there was only a single between-

subjects covariate. In this second design, in which the

covariate varies both between and within subjects, the

power curves vary dramatically in how they depend

on the residual and covariate covariances. For ex-

ample, the power results for the two regression-based

approaches show exactly opposite effects of these Co-

variances. With Analysis 2a, in which the difference

between the 7s is regressed on the sum of the Xs,

power dramatically increases as the residual and co-

variate covariance become more positive. With

Analysis 2b, in which the sum of the Ys is regressed

on the difference between the Xs, more negative re-

sidual and covariate covariances are associated with

s We thank a reviewer for forcing greater clarity in our

exposition here. The expectation that the two simple slopes

will equal half the difference between the two slopes of

interest holds in the population in which the variances of the

two covariates are assumed to be equal. Under this assump-

tion, it is unlikely that the two covariate variances in any

sample will be exactly equal. As a result, the two estimated

simple slopes will generally not be exactly equal to half the

difference between the two estimated slopes of interest.


Table 2

Design 2: Power of Tests of Covariate-Treatment Interaction

Slope

Covariance 0.0,0.2 0.0,0.4 0.0,0.6 0.0,0.8 0.2,0.4 0.2,0.6 0.2,0.8 0.4,0.6 0.4,0.8 0.6,0.8

Analysis 1 : Structural equation models

-0.8

-0.6

-0.4

-0.2

0.00.20.40.60.8

-0.8

-0.6

-0.4

-0.2

0.00.20.4

0.60.8

-0.8

-0.6

-0.4

-0.2

0.00.20.40.60.8

0.101

0.161

0.223

0.271

0.289

0.271

0.223

0.161

0.101

0.062

0.078

0.099

0.127

0.166

0.226

0.325

0.504

0.840

0.840

0.504

0.325

0.226

0.166

0.127

0.099

0.078

0.062

0.259

0.482

0.659

0.763

0.796

0.763

0.659

0.482

0.259

0.099

0.163

0.246

0.354

0.493

0.662

0.843

0.973

0.999

0.999

0.973

0.843

0.662

0.493

0.354

0.246

0.163

0.099

0.500

0.811

0.939

0.977

0.985

0.977

0.939

0.811

0.500

Analysis 2a:

0.158

0.296

0.461

0.642

0.812

0.936

0.991

0.999

0.999

Analysis 2b:

0.999

0.999

0.991

0.936

0.812

0.642

0.461

0.296

0.158

0.737

0.962

0.996

0.999

0.999

0.999

0.996

0.962

0.737

Regression

0.232

0.452

0.673

0.851

0.957

0.995

0.999

0.999

0.999

Regression

0.999

0.999

0.999

0.995

0.957

0.851

0.673

0.452

0.232

0.101

0.161

0.223

0.271

0.289

0.271

0.223

0.161

0.101

approach

0.062

0.076

0.095

0.121

0.158

0.213

0.305

0.475

0.812

approach

0.812

0.475

0.305

0.213

0.158

0.121

0.095

0.076

0.062

0.259

0.482

0.659

0.763

0.796

0.763

0.659

0.482

0.259

0.500

0.811

0.939

0.977

0.985

0.977

0.939

0.811

0.500

0.101

0.161

0.223

0.271

0.289

0.271

0.223

0.161

0.101

0.259

0.482

0.659

0.763

0.796

0.763

0.659

0.482

0.259

0.101

0.161

0.223

0.271

0.289

0.271

0.223

0.161

0.101

of difference on sum

0.094

0.151

0.226

0.324

0.452

0.615

0.802

0.957

0.999

0.144

0.264

0.413

0.583

0.757

0.902

0.982

0.999

0.999

0.060

0.073

0.089

0.112

0.144

0.192

0.272

0.425

0.757

0.087

0.136

0.199

0.284

0.397

0.548

0.737

0.925

0.999

0.058

0.069

0.083

0.102

0.128

0.168

0.236

0.368

0.682

of sum on difference

0.999

0.957

0.802

0.615

0.452

0.324

0.226

0.151

0.094

0.999

0.999

0.982

0.902

0.757

0.583

0.413

0.264

0.144

0.757

0.425

0.272

0.192

0.144

0.112

0.089

0.073

0.060

0.999

0.925

0.737

0.548

0.397

0.284

0.199

0.136

0.087

0.682

0.368

0.236

0.168

0.128

0.102

0.083

0.069

0.058

Analysis 3: Test of dependent correlations

-0.8

-0.6

-0.4

-0.2

0.00.20.40.60.8

0.639

0.420

0.337

0.303

0.293

0.303

0.337

0.420

0.639

0.992

0.930

0.861

0.818

0.803

0.818

0.861

0.930

0.992

0.999

0.999

0.994

0.988

0.986

0.988

0.994

0.999

0.999

0.999

0.999

0.999

0.999

0.999

0.999

0.999

0.999

0.999

0.602

0.392

0.315

0.283

0.274

0.283

0.315

0.392

0.602

0.982

0.894

0.812

0.764

0.749

0.764

0.812

0.894

0.982

0.999

0.994

0.981

0.969

0.964

0.969

0.981

0.994

0.999

0.540

0.348

0.280

0.252

0.244

0.252

0.280

0.348

0.540

0.958

0.831

0.737

0.686

0.670

0.686

0.737

0.831

0.958

0.468

0.299

0.241

0.218

0.211

0.219

0.241

0.299

0.468

Note. Calculated for different true covariate slopes and different residual and covariate covariances.

greater power. In fact, at any given value of effect size

(i.e., slope difference), the power result for Analysis

2a at covariance k equals the power result for Analysis

2b at covariance -k.

The reason for this difference in the power results

between the two regression-based approaches can

readily be seen by examining the variances of the

estimated slopes in the two cases, provided in the

Appendix. The only difference between the two vari-

ances, for Analyses 2a and 2b, concerns the signs of

the residual and covariate covariances. In Analysis 2a,

regressing the difference in the Fs on the sum of the


0.25-

1.00-

0.75-

o ' o o o d d d d d o d o 6 d d d

i i i i i i i t i I i i i i i i_ i I

ANALYSIS 1: SEM

ANALYSIS 3: CORK

&"tS~'Q"<*~O

."-o o o""

ANALYSIS 2A: REG

ANALYSIS 2B: REG

ki.oo

-0.50

-0.25

-0.75

-0.50

COVARIANCE

A— SLOPES .0.8 SLOPES .0.6 -—O—

<X SLOPES .0.4 SLOPES .0.2 D

Figure 4. Power of analyses for Design 2. SEM = structural equation models; REG = regression

approach; CORR = test of dependent correlations.

Jfs, the variance of the slope decreases as the covari-

ance increases. In Analysis 2b, exactly the opposite

occurs.

The comparison between the structural equation

analysis, Analysis 1, and the dependent correlation

analysis, Analysis 3, also suggests that the residual

and covariate covariances have rather different effects

in these two cases. Both show symmetrical effects on

power as the covariance departs from zero, either

positively or negatively. However, power decreases

for the structural equation analysis as the absolute

value of the covariance increases, whereas power in-

creases for the dependent correlation analysis as the

absolute value of the covariance increases. Note that

when the covariance of residuals and covariates is

zero, the power of these two analytic approaches is

quite similar.

Assuming that power differences were the only

consideration, one would take away from these results

three conclusions: (a) When dependencies of residuals

376 JUDD, MC-CLELLAND, AND SMITH

and covariates are highly positive, one ought to adopt

Analysis 2a, regressing the difference in the depen-

dent variables on the sum of covariates and testing

whether the resulting slope is different from zero, (b)

When dependencies of residuals and covariates are

highly negative, one ought to adopt Analysis 2b, re-

gressing the sum of the dependent variables on the

difference between the covariates. (c) When depen-

dencies in the data are near zero, both the structural

equation approach and the dependent correlation ap-

proach are more efficient than are the two regression-

based approaches. Since power decreases as the mean

value of the slopes increases with the dependent cor-

relation approach, when the two slopes are large and

the dependencies of the residuals and covariates are

near zero, the structural equation approach is slightly

more efficient.

But these power comparisons are certainly not the

only considerations in choosing among the four analy-

ses. More important are the different assumptions that

they make about the data. As we have already dis-

cussed, the two regression-based approaches assume

that the two covariate variances are equal to each

other. Only then do the slopes in the difference and

sum equations have expected values equal to the dif-

ference in the slopes that one wishes to test. Addi-

tionally, the dependent correlation approach makes

the further assumption that the residual variances to

the two Ys are equal. Again, this assumption is nec-

essary in order to know that when one finds a signifi-

cant difference between correlations, one can con-

clude that the relevant slopes differ significantly.

In all of the power calculations we have presented,

we have met the assumptions of the most restrictive of

the four analyses. That is, we have assumed that both

the covariate and residual variances are equal. Under

this constrained situation, our relative efficiency re-

sults hold. If one cannot make these assumptions, then

one needs to choose the analytic model that is most

appropriate to the data at hand rather than paying

attention to our power results. Clearly, the structural

equation approach is the most robust in terms of vari-

ance assumptions, and it does remarkably well from a

power point of view, particularly when dependencies

of residuals and covariates are close to zero. In a wide

variety of situations, it would seem to be the preferred

approach for both design alternatives.

A limitation of the efficiency results we have pre-

sented is that they are based on a relatively large

sample size (100) in comparison with those frequently

used in within-subject designs. This sample size was

chosen because of the large sample assumptions un-

derlying the inferential statistics used in the structural

equation and dependent correlation approaches. The

efficiency of these procedures has not been explored

in cases of smaller sample sizes. Note, however, that

the large sample assumptions are unnecessary in the

case of the regression-based approaches. This ap-

proach would seem to be preferable, therefore, in

many typical small-sample cases.

An additional limitation of all of the cases we have

considered is that only within-subject independent

variables having two levels have been discussed. In

fact, however, there exist relatively straightforward

generalizations of the procedures we have considered

to cases with multiple treatment levels, either having

a single covariate that varies only between subjects

(Design Alternative 1) or having as many covariates

as levels of the independent variable (Design Alter-

native 2). The generalization in the case of the struc-

tural equation approach is that additional dependent

variables are incorporated into the models and equal-

ity constrains are placed on all of the covariate-

dependent variable slopes. This results in an omnibus

X2 difference test. Rejection of the null hypothesis

implies differences in covariate-dependent variable

slopes without identifying where those differences lie.

Generalizations of the tests of dependent correlations

to this omnibus case are also available (Olkin & Finn,

1990). Generalizations of the regression-based ap-

proach are a bit more complicated, since one must

define within-subject contrasts that specify treatment

combinations whose covariate slopes are expected to

differ. One then estimates slopes for resulting within-

subject difference scores, under the assumptions

given previously for this approach.

Given the frequency with which substantive ques-

tions concern covariate-treatment interactions and the

frequency with which psychologists use within-

subjects designs, it seems to us somewhat surprising

that analytic alternatives for testing such interactions

in within-subject designs are not generally known.

This article remedies that situation by defining the

alternatives, the assumptions made by each, and their

relative efficiencies given the most restrictive as-

sumptions.

References

Alexander, R. A., & DeShon, R. P. (1994). The effect of

error variance heterogeneity on the power of tests for

regression slope difference. Psychological Bulletin, 115,

308-314.


Bentler, P. M. (1993). EQS program manual. Los Angeles,

CA: BMD Statistical Software.

DeShon, R. P., & Alexander, R. A. (1996). Alternative pro-

cedures for testing regression slope homogeneity when

group error variances are unequal. Psychological Meth-

ods, 1, 261-277.

Dretzke, B. J., Levin, J. R., & Serlin, R. C. (1982). Testing

for regression homogeneity under variance heterogeneity.

Psychological Bulletin, 91, 376-383.

Huitema, B. E. (1980). The analysis of cavariance and al-

ternatives. New York: Wiley-Interscience.

Joreskog, K. G., & Sorbom, D. (1993). L1SREL8: The

SIMPL1S command language. Chicago: Scientific Soft-

ware.

Judd, C. M., & McClelland, G. H. (1989). Data analysis: A

model comparison approach. San Diego, CA: Harcourt

Brace Jovanovich.

Khattree, R., & Naik, D. N. (1995). Applied multivariate

statistics with SAS software. Cary, NC: SAS Institute Inc.

Kenny, D. A. (1979). Correlation and causality. New York:

Wiley-Interscience.

Meng, X., Rosenthal, R., & Rubin, D. B. (1992). Compar-

ing correlated correlation coefficients. Psychological Bul-

letin, 111, 172-175.

Myers, J. L. (1979). Fundamentals of experimental design

(3rd ed.). Boston: Allyn & Bacon.

Olkin, I. (1966). Correlations revisited. In J. C. Stanley

(Ed.), Proceedings of the Symposium on Educational Re-

search: Improving experimental design of statistical

analysis (pp. 102-156). Chicago: Rand McNally.

Olkin, I., & Finn, J. (1990). Testing correlated correlations.

Psychological Bulletin, 108, 330-333.

Rosenthal, R., & Rosnow, R. L. (1985). Contrast analysis:

Focused comparisons in the analysis of variance. Cam-

bridge, England: Cambridge University Press.

SAS Institute Inc. (1990). SAS/STAT™ user's guide. Ver-

sion 6. 4th ed., (Vol. 1). Cary, NC: Author.

Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Sta-

tistical principles in experimental design (3rd ed.). New

York: McGraw-Hill.

Wolfram, S. (1991). Mathematica: A system of doing math-

ematics by computer. Reading, MA: Addison-Wesley.

Appendix

Derivation of Regression Approach Estimates

Design 1: Between-Subjects Covariate

Assume all variables in Figure 1 have expectations of

zero. The linear equations underlying the model are

and

The expected values of the two slopes are

CV(y,X)

If we regressed the difference between the two Ys (V,, - Y2i)

on Xlr the resulting slope would equal

CV[(r, - y2)X]

"<Yt-r,x V(X) '

Given the algebra of covariances (Kenny, 1979),

CV(Y,X) - CV(Y2X)

V(X)

and

IV =

The variance of estimates of this slope is equal to

V(JT) '

where V(X) is the variance of X.

Accordingly, the two covariances can be expressed aswhich reduces to

cv(jyo = B, n\(X)

378

Design 2: Within-Subject Covariate

The equations underlying the model in Figure 2 are

YH = Py,*^!/ + Py,x2X2, + elt.

JUDD, MCCLELLAND, AND SMITH

If we assume that V(X,) = V(X2), this reduces to

Pl'.-Y, ~ Pl-Jfj

We assume that the crossed effects, fiyx and Py^x,, equal

zero. Accordingly, the covariances between the Ys and Xs

can be shown to equal

and

= prACV(X,X2),

If we now regress the difference between the 7s on the sum

of the Xs, the expected slope equals

-iw-^) - V(x, + x2)

Given the algebra of covariances, this can be expressed as

cv(r,x,) + cv(y,x2)

2CV(X,X2)

and, by substituting for these covariances,

The variance of the estimate of this slope is equal to

«V(X,+X 2 )

which can be shown to equal

1/2 (priXi + (3yA)2[V(X) - CV(X,X2)]

2n[V(X) + CV(X,X2)]

where V(X) is the variance of both Xu and X2|.

When the sum of the Ys is regressed on the difference

between the Xs, the same sequence of steps can be used to

show that the resulting slope also equals (Py,Xl ~ Py,x2)/2, if

we assume that V(X,) = V(X2). Additionally, the variance

of the estimate in this case equals

Yi (P.-A + Px,x/ [V(X) + CV(X,X2)]

j) + 2CV(X,X2)'

2«[V(X)-CV(X,X2)]

Received January 22, 1996

Revision received June 5, 1996

Accepted June 11, 1996

Testing treatment by covariate interactions when treatment varies within subjects

Documents