DOCUMENT RESUME TM 004 354 Richards, James A., Jr. A ... · J. ED 104 905. AUTHOR TITLE. O. ItipTimpTipx. DOCUMENT RESUME. 95. Richards, James A., Jr. TM 004 354. A Simulation Study

J.

ED 104 905

AUTHORTITLE

O

ItipTimpTipx

DOCUMENT RESUME

95

Richards, James A., Jr.

TM 004 354

A Simulation Study of the Use, of Change Measures.to;Compare Educational Programs. Report No. 163.

Johns Hopkins Univ., Baltimore, Md. Center for the.

Stud.), of Social Organization-Df Schools.

SP WS_AGENCY National Inst.D.C.

of Educ ion (DREW), Washington,

`RE ORT'NO tS0S-R-163PU DATE .Oct 74CONTRACT-NOTE

BDRS'PRICEDESCRIPTORS

NE-C-00-3-011425p.

MF-$0.76 HC-$1.58 PLUS POTIGT*Academic Achievement; Achievement Rating; *Behavior

Chinge;/ Computers; Educational Asiessment;.

Measurement Techniques; *Program Effectiveness;*Public Schools; *Simulttion

ABSTRACTArtificial data_weri-nted to assess the-correlation--

between Several'estimites of average student change=in various,schools and the'iltrueiit imPaOt of those schools. Results indidate that-

allestimates involving pretest-posttest differences measure school

impact with'reaponable-accuracy. It is important to Measure change

oVer-_-the entire courSe af.learning, however, And-not-just over the

liter-stages of learning. le correlations between -change scores and

r-othe school_characteristic reflect with reasonable accuracy the-

relationships beiweenthoseicharacteristics and impact, but will be

large only -when tht 4pderlyp.ng relationships are substantial. Simple

gain scores-measvie_tlhe true situation About as accurately as other

change estimates, are easier to compute, and probably_are,more-meaningful to nonrese rchert.lAuthor) ;

O

,

L

A'SIMUSTION STUDY 011.'THE USE' OF CHANGE MEASURES

70-COMPARE EDUCATIONAL PROGRAMS

Contract No. NE-C-00-3-0114

Work Unit No. 2A

James M. Richards, Jr.

Report No. 183

October 1974

U.S. DEPARTMENT OF HEALTH.EDUCATION &WELFARENATIONAL INSTITUTE OF

' -EDUCATION-'THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY A$ RECEIVED FROMTHE PERSON OR ORGANIZATION ORIGIN.

.....KriZIG.LT-POINTS-OF.VIEW OR OPINIONS-_STATED DO NOT-NECESSARILY REPRE.SENT OFFICIAL NATIONAL INSTITUTE OF'EDUCATION POSITION OR POLICY.

Published by the Center for Social Organization, of Schools,-

supported-inpart as a research and development center by

C\ funds -from the United States National Instittiteof'Education,

Department of Health,. Education and Welfare.- The opinions

expressed in this publication do not necessarily reflect'

the position or policy of the National Institute of Education,_

_and no 'official endorsement by the Institute should be inferred.

The Johns. Hopkins University

Baltimore, Maryland

STAFF

John L. Holland, Director

James--M.-McPartland, Assiitint Director

'Karl Alexander

Denise C. Daiger

David L. De Vries

Joyce L. Epstein_

Ann Forthuber

Staphadie G. Freestan

Gary 1;,-Gottairedson,= _

Ellen Greenberger

Edward J:'Hitrsch

Rosemary Hollici

John H. Holiifield

JosselSon

Nancy_L. KarWett

Hazel C.

Marie eliiikurath

Daniel

1

D;-:-Mc_ComOehie

Donna H.

Edwird-McDil2

James W. Michaels__

:Mate

Susan L.

Julian C. 'Stanley

0.4,411c

O

-

:Introductory Statement

-.The Center, for Social Organization of Schools has --two primary

to develop a scientific knowledge of how schools afect

their students, and to use this knowledge to.develop better school

Objectives:

-.practices and organization.

/

The'Center works through threeiprogams to achieve its objectives.,

The Schools and'Ilaturity program ia-su yidg,the effects of school,

family, and peer group experiences on he development'of,aititudes

consistent with psychosocial maturity The objectives are to formu-

late, assess, and research importan

traditional academic acWevement.

currently concerned_with authorit

educational soils other than.

School Orlianization program is://

control structures, task structures,

reward systems, and peer gtoup processes

-program (formerly Careers!

of career development, t hag' developed

rricula)

in schools. The careers/------ --

-,,,..

bases `its work upon/4.iheOry

/_ /i

a self-iadilnisteredrVocational

guidance device and.a elf-d rected career program to promote vocational

development and to foster satisfying curricular decisions for high

. /

/

,school, college, an adule populations.

This report, repar d by the School Orianization program, examines

effectiveness of schools and edutationalmethods of assessing

programs in prom ting educational growth of students.

ii

Abstract

'

Artificial data were used to assess the correlation between

- several estimates of average student change in- various schools and

the "true" impact of those schools. Results indicate t1u,t all

estimates involving pretest - posttest differences measure school---------

impact with-reasonable accuracy. It is important to measure change

over the entire course of learning, however, and not just over the,

later stages-of learning -ThV correlations-between change scores and

other school characteristics-reflect with reasonable` accuracy the

relationships-between those characteristics and impact,-_but will be

large only-when the underlying relationships-are substantial.

himple-gain scores measure the true situation about As- accurately as

-,other_change estimates, are easier to compute, and iprobably are more

,

meaningful-to non-researchers.

A

Introduction .

A basic purpose of education is to promote _desirable -change or

growth !in the educational attainment of students. It follows that-..

SchOols cr other educational ,programs should be evaluated largely' on

their effectiveness In promoting such change. There are many theoretical

--problems in estimating_ student change _from scores cin,standard tests of.

educatio al attainment, however, and these .problems are heightened in

the typica sit tion where the students entering various_ schools differ

2 systematically (Astin and Panos, 1971; Cronbach and Furby, 1970; Hatris; _

1963; _Herriott and Muse, 1973; Klittgard and Hall, 1973; O'Connor, 1972).

It has -been difficult. to assess the practical importanCe:of these

in, m------------77St.,

thearetical-problems-becau-se7true_ chatiFge: scores are unknown o

_,, ,.. _ _. _ -

-e-

computerlongitudinal research. Recently, a coMputer procedure waS developed to. _ ,,, -

. _ _ . _

._.. . .

provide a rtif If ia- l data -in- which these true change scores are knoWn-- _-_.t

(Richards, _ Karweit,_ and Prevaet, in press) ._When such artificial" ciata--

--4. _ , -. - .

_--- were used to_compare several statistical. techniques fOr -assessing change...,...-.

in individual students (Richards, 1374), the results indicated that

individual change-is meas- ured with -reasonable adcoracy by all techniques. . .

,. .

that involve the difference between the pretest and the posttest.- In

particular, the simple difference between the pretest and the posttest

is,about as_ accurate as other change estimates, such as regre-ssed gain. ,

.

-scores, and is'much -easier to compute than other estimates. These trend's,

, .

hold even when ;students are assigned nonrandomly to schOols that differ

In -their -impadt_ on Students.

----These results strongly suggest that the theoretical probleals-of.

change measures have limited practical -significance .for measurin6

individual growth, and it is important to deteriine whether this is also

the _case for measuring school impact. Accordingly, in this study artifi,6

,tial data were used-toassesS the correlation between several estimate-s

of average student change in various schools-and the. "true" impaLt- of- A

the. same schOols. This study-is. stated in'the-cohtext of education, but

D.

the procedures for gentrating,daia and measuring change- are abstract.

Therefore, the results should generalize to many situations -where one

- wishes to compare_ithe-impAct-_-of-varying social interventions.

MethodI

Simulation- Procedure._ Because it -seems desirable for artificial

data to resemble real data as cloSely-as posSikte, the computer procedure, .

IS

was designed (Richards, et at., in press) to reproduce selected aspects..

_

t

-of the ETS- Growth Study (Hilton, _Beaton, and Bower, 1971) arid of the

\Project TALENT study of high schools in the:United States (Flanagan,

_ \ -_. _,.

tt al ._,_1 962 . In the ETS GrOlth Study- students were assessed initially

with a measure academic potential (SCAT), and a measure of educational.

attainment (STEP). -Subject to the usual attrition in longitudinal' --4

research; the educati al ttainment_of these students was reassessed.

, :--/

. -

on three Subsequent occa ons. Project/TALENT provided' intercorrelations

4/

amOng a _variety of communit school, And student chatacteristica for/

.

/

a- representative sample -of U. S high schools.

t

.

2

4

. The computer procedure generates scores for ifidiVidhal students0 -

that strive2-to reproduce the means, standard deviations, and ittersorl-.:.,.________--. ______- __

-relations-obtat rhL7d-T-1-1----TS-----G-r-citithSttidy. The student '_s score on

academic potential is generated first and used to derive that student's-

score -on initial acadeMic attainment.. Theta gain scores 'are generated

and .added _to yield subsequent attainment TrUe standard scores

are generated initially, then the appropriate amount of .random erroz

added, to each score and the' scores are transformed to the metric of the

_ETS Growth Study_abserved, scores.r:;:rhis, simulation proceduie closely.

-reproduces the ETS Growth Study ressrts (Richards, 1974)..

The simulation procedure, permiie-the.investigator to assign students

to schools either randomly or nontandonily. When tudehts_are assigned.

nonrandomly, the prograi strives to reproduce the average -correlation

between commubily per,,Capitaincome and ayerage academic -potential of

s_ tudents - estimated---from'Ptoject results -(p= .54). The ratio

of between schools variance to total variance also simulates the,Projectt

.TALENT( ratio. ------------4_.=-------_,1

The simulation -proce-dore---a-Sifumes that commUnity pek capita income -TThe 4

determines school resources, and that school resources in turn determine., ii

'school impact. A review of Project-TALENT results sugge4te'd-an average

correlation of aPprOximately .25 between commuhitY'._incOme and those

on y assumed to facilitate student 'growth, so the-. 6

simulation procedure strives. to reproduce this re ationship between

income' and resources. Community- income is dra n randomly from a -normal

.

distribution, and it is assumedthat school resources and school impadt

-.also-ere-norMally distributed.

.There is little empirical basis for estimatifig either the correla--:

motion between resources and,t4pact or_the-extent-to-OcCh schobls vary

-in impact. Therefore, the simulation procedure allows th3 investigator;

to specify both the correlatiod between resources and impact and the_

standard deviation of the impact variable. This standard deviation is

.. ,

specified in the form of a number between 0 and ly When the standard

deviation is .10, the average growth values uded in senerating scores. .

are equal to the average growth scores obtained in the ETS study for a-,

school with average' impact, and 10% higher { than the ETS averaged for

a school one standard deviation above the mean on impact. (The siiulated

* - .

data appear to meet.the assumptions for this manipulation even if the

ETS,data do not.) -.

Gain scores for individuals are generated according to the following

principle:

=t_--

Gm d

I'-whete Gt t

is-total (true).- growth, G is average (or mean) -growth (i.e.,, -._ .

the parameter estimated from:the ERS data and Gd is-a deviation frOM/ ..

this aver' in ividual differences in true growth./ The

total gain score Is, added.tO,the pretest score to ,yield the pbstteSt

score, and the posttest score-then becomes the pretest-for the nekt

growth interval. For each giowth.interval., the e-pretest is one of the

/

elements entering a multiple regression formula used to generate the

.11

-

_Gdvalues. The correlationsbetween pretest and growth become increasingly

negative'lor'successive interval(Richardsi 1974).

Ifi-generating scores, the mean growth parameters for the three'------

t

z f

. . :

-.intervals are adjusted _for-school-impat,f, and no other changes are. made.. \

Consequently, the adjusted mean growth parSmeters_frequently-will-fiot be

i

------64fial.tos,theatained average true growth scores for a given school,___

_.

A school with above average'impact. ill,have higher than average-mean .-----,,

_growth-parameter and there ore higher than average true posttest ,scbres. ------

,-These become high r -than average truelmtest scores for SubSequent

\

learning-interva s, and these higher pretest scores make an inCreasingly-

--negative contrib,tion in the computation of subsequent true growth scores.

The averages of the obtained true growth _scores for that acboOl will tend

:to be lower than the adjusted mean growth_ parameters. Similarly, the

averages of the obtained true growth scores will tend to be-higher than

the adjusted-mean growth parameters for a school with below average imPact-;

.

Table 1. presents a simplified illustration of these trends for five

hypothetical schools that-are average in every_respect except for differing-.

__, - ., - . _n__- _ -_

\.

,inimpact. Because other parameters besides pretest score_are---involved 1

i--,

A

Insert Table I lAbout Here

Pia.5...Kg_tfig.Scores-(Richards; ITT , it is conceivable_that-a school-T

with below average impact (and therefore below average adjusted mean

growth parameters) will_have_ligher average Obtained_true growth-scot-ea=

than a' school with above average impact. his ts especially true when

students are assigned_to.schoolS n nrandmmly.

12, J

A

_,-

----'-- ., - -bata _Sets--Six -1-nde-pafden--t,sets O---f simulated data were generated,_ . . .

. _ . --,

____

._ , for the present study. Iii each set stndents_were-assigned.---to-100-schools,. .

or treatments: The number of students per 'school varied randomly with

mean = 150 and,-..tandard deviation ai".15. Therefore, the total number of

tlstudents in each of these sex sets* was _approximately 15,000..

,

In three,of these sets students were asdigned randixiily to schools'-,,._-:' .. z -- .

or treatments, nd in the other three sets students - were assigned`

-nonrandomly: U der each-typeo-Uassignnient, simulated data wtite generated

for three differ nt assumptions about the relationship-between tadsiitiOl.

resources and sc iinpaa., Specifically, it was assumed that school

esources account for-5%, 20%,'or 80% of t e iVariance in school impact

(corresponding to orrelatIons of .2236, 472, or .8944) .

Finally, in all six sets the standar deviatiOn of the impact variable

-1was\ set. at .10. \At approximately this rna nit_ ude two_ simulated schools

Mr

4 one standard deviation apait.on impact -(w th N's = no)' will differ

the ..05 level when compared- with- re'spec to educational growth between\

successive occasions. \.

.. ,_ ----

_:-__Change-Measures. "X Wide variety = of chdnge measures 'have been propoded

(Cronbach and Furby, 1970), but recent results suggest _that most ,of these

measures yield essentia10 e uivalestt_re-sult-s-(--Rfah-dcord-

ingly, this study used only four measures,of change, each representing

a different approach to est mating change._ These change estiinates

included:

1. Posttest score.

2.' Posttest score adjus ed for initial academic potential. This

change estimatefris.th= difference between posttest score and ,

-6 13

4 .

predicted posttest score, using initial academic potential as

the predictor. (The giediction equation for each data` set was

/ ,

--based on the Observed relationships in that set.) Thus,--this

technique, resembles analysis of covariance with academic poten-!

tial treated as e covaiinto.

A 3

3. Raw gain. This,chan score is the simile difference between

.-

pretest score and posttest score: /`,_ ".,,

)\\ I,

4. \Raw residual gain. This estimate is the difference b. eetf'\ -f 0 \

...

posttest score and predicAed posttest score, using pretiest--___

score as, the predictor

Results

To facilitate comparison with the earlier study of_individual change5

estimates (Richards,,1974)/the first step in the data analysis was to

compute the correlationsibetween average estimatectehangescores for

Various sehools and average.tr e.change scores for the same schools. -An-

\-/

unresolved question is whether it\is better\tb coMpute change -scores for'

individual students and then avers e-within als or to computo_change

scores fromschooI-meantr(Nir, Lint, and Patton, 1969), so both procedutes

were used to estimate change in this analysis. Table 2 summarizes the__

repUlts.'

Insert Table 2 About Here

These resultn seem quite consistent.with the resUats of the eatlier4

.study of individual change estimates (Richards, 1974). Chew is estimated

7

most accurately by techniques that.involie the difference between the

retest and the posttest, and these techniques seem equally accurate,

(i.e:, raw gain is just as accurate as residual gain). For the most .

part, there is little difference between change estimates based on

,individual students and change e'stima'tes based on school means. In a

few cases estimates based on school means have a clear advantage and

.

these' estimatesestimates are also easier,to compute, so subsequentianalyses in,

this paper-involve only estimates based on school means. .

The next analysis evaluated/the accuracy of these change estimates

as measures of School impact.

impact and various change\,

table al C.);suramArizes the

groWth scores.

Table 3.summarizet the correlations between

estimates. For .comparative purposes; this

correlations between impact and average true.

Insert Table 3 About Here

These results indicate that change estimates can be quite effective

in rank ordering schonls with respect to their impact even when studentsr,

. are assiwie to schoola,honrandomly. The simple gain score's again were .

just.as accurate s the residuifgain scores and, as Cronbach and FurFy

(1970),point out. postiest_sFore measures impact adequately when students

are assigned to treatments, randomly.

The results also indicate that it is imports to measure change

,over an appropriate interval. Adjusted potttest scores, :Amp in

acores, and regressed gain scores all. rank ordered schools accurately

15

1 \

\

i

-variations in impact. Such results are more typical of what would be

,ft

obtained in a'nreal"-longitUdinal study: 'Table 4 sumnarizes the relevant

.correlations betWeen resources and change. The- tagnitu es of these

correlations clearly follow the underlying relationship between resources

and impact, but are somewhat lower. The smaller magnitude of these

when they involved change from initial status, but none of th measures'

were particularly effective in rank ordering schools when they involved

growth in the'later stage's of the learning process. This ineffective-_

ness reflected the true situation, because it is also characteristic

of the true growth scores. The ETS data resemble otheriongitudinal or

learning data in-a number of respects (Richards,, 1974), so these findings

about when to measure changeihould have considerable generalizability.

The final question examined in\this study involves the relationships

am9ng these change Obasures and the school characteristics that cause

'Insert Table 4 About Here

t.

.

correlations p rhaps is partly-the,consequence of unreliability of the

Change score, but also appears to reflect the 'imperfect correspondence-m:'

. .

.

. .

between school impact and average true change. The results again indicate

-that raw gain is about-as accurate as any other change estimate, reempha-

size the importance of measuring changeover an appIopriate interval,

and suggest that the correlation between a school characteristic and

school impact must bereasonably substantial- before ani change score/

-'will reveal the relationship.

Discussion

Theoretical_ treatments of the issues considered in this paper have

emphasized the theoretical difficulties of using change scores in gener4

and of usi'hg.simple gain scoresin particular. The results of this study,

like those-of the earlier study of individual change (Richards,

suggest that the practical importance of these theoretical difficulties

may have been exaggerated. It appears that-change estimates over-an----71

appropriate interval (e.g., the entire course of_ilearning, not just the,\

.later stages) do measure-sthool impact with reasonable accuracy: The.

correlations, between change scores and other school characteristics

reflect with'reasonable accuracy the relationships between the same thar,

acteriatica and school impact, but consequently will be large:(or11Y'

ficant") only whin the underlying relationship is fairly substaniia

'These conclusions appear relatively unaffected by random vs. nonrandom0 .

assignment of. students (although this finding could change for more severe

nonrandamness), or,by-whether change:measures involve individual scores,

or, school means.1

Insensitivityto weak relationships almost certainly .is character-,

ffisticnot just of change scores; but of all static 1sp ca -procedures that

.

might be applied to these dataand simple,gain scores appear to reflect

the true situation about as accurately as'any other estimate of change

or impact; Simple gain scores also are easier to compute than most other

-- estimates and probably are more meaningful to non -researchers. Therefore,

f."

,the-resuits of this study suggest that it often may be quite appropriate.

'It should be.emphasized that these.conclusions'apply to true longitudinal,designs and this study should not be used to justify such procedures as

Measuring impact'by educational ittainment.adjusted for a test of academic"

potential administered at the same time.%

lo

5\

0

campfire educational programs on the basis of simple pretest-pOsttest

differences.

The, discrepancy between this study and. earlier, ,theoretical treat-s

mAnts may perhaps best be resolved in terms oedegree of concern about

"Type I" errors. That is, -theoretical treatments usually seem to assume:.

that educational treatments do not differ on impact and emphasize the

possibility that use of change scores, particularly simple gain scores,

will lead to the false conclusio that they do. differ. Certainly this

cannot be ignored, especially whenthe students assigned to

-various treatments differ considerably (Astin and Panos, 1971; Cranbath.

and Furby, 1970), and certainly if is,possible,0 propose hypothetital

situations where change scores could be mikeadiag or confusing, esPeciall

one has -a tastefor_paradoxes (Lord, 1967). (This study, on the other

hand, assumed that schools do differ on impact and asked how accurately

change Azores describe these differences. The answer to this question

appears much-more favorable to change scores. Indeed, the results

suggest that\when one uses change scoreasove an inappropriate:interval

_ ..\ ,

,

in a correlational study there may be a grA0er danger ofithe false_

.

.

P . ..

conclusion that schools do not differ with respect to impact than.of th.

r1

false conclusion that schools do differ.

Cronbach and Furby (1970) correctly point out that some of the

questions to which change scores might be applied could be an ered

directly with such techniques as partial correlation. The advantages of

such techniques are that they are more direct than change scores, howver,

-hot that they are more accurate, nor that they require less statistical

11'

sophistication

who prefers to

The results of this study lend support to the investigator

use change scores for reasons of convenience or ease of

understanding.

Finally, the results of this study again illustrate the-usefulness

of simulationtechniqueo.for investigations Of longitudinal methodology. ,

\ v 0

It would'be impossible to investigate the questions considered in this

study. with "real" _longitudinal data because the investigator would have

no way of knowing either the true individual growth scores or-the true

school impact scores: At best one could compute the intercorrelations

amongdifferent estimates of change (Dyer, et al., 1969). With simulated.

data it was easy to compute the correlations between true scores and-the

different estimated scores. It would also be easy to extend the simulation_

procedures to the situation where considerable' attrition of subjects-occurs,

s

to the-situation where one has only pseudo-longitudinal data (e:g.-, test

scores for Occasions 1 'nd 2 obtained from different groups of students'

in the same school), or to different models for growth: 'Ihu-s, 'simulation

/ .

technique's offer considerable promise for refining our knowledge about

. when various proceduresfor analyzing longitudinal data are appropriate.

rt,

12 19

c References

Astin, A.W. and lianos, R.J. I The evaluation of educational prosraim.

,_ .. . /

In/R.L. Thorndike (Ed .) Educational Measurement; - Washington:

AMerican Council on Education,- 1-971.

Tronbach, L.J. andiurby, -14--.- HoWahould-We measure change--or:shouldi - R

,.__..--

'we? PSychological Bulletin) 1070, 74,6840:/

Dyer, H.S.; Linn,' R.L.; n Patton, Ma._ A compariOn_of four methods

/ . ,: s_. H

. ,

,-

of obtaining discrepancy measures -based on observed-and predicted/

school system means oh.achieyement,tests. American Educational

Research Journal, 1969,- 6 591-605.

Flanagan, J.C.; Dailey, J.V.; Shaycoft, M.F.; ;,enctGoldberg,'-I.-

Studies`of the American_High school. Project_ TALENT Monograph No. '2,-;

-Pittsburgh: University_of Pittsburgh,-1962.

Harris, C.W. (Ed.) Problems in,Measuring-Change. Madison; Wiacon4in:.

=

University of WieconsinA2ress, 1963._

_Herriott, E.H. -and Muse,,D.N., Methodologicalissues in the studp.af-

school effeots._*n F.R. Kerlinget, (Ed.) -Review of Research in

- .

'Education. 1tasca, :Illinois: __Peacock,---1973.

Hilton, T.L.; Beaton, A.E.; andBoWer; C.P. Stability and-Instability-

in Academic Growth - -A Compilation of Longitudinal Data. Princeton,

New Jereey: Educational Testing.Service,;1971..

Klittgard, R.E. and Hall, G.R. A Statistical Search for Unusually

Effective -Schools. Santa Monica; California:. Rand

F.M. A paradox in the interpretations -of -group comparisons.

Ps cholo: 304-305.

0Connor, .Extending. classical test-theory to the measurement of

4

obange. Review of Educational Research _1972, 42 73-98.

ichards, J. M., Jr. A Simulation4Study Comparing Procedures for

Assessing individual 'Educational Growth. Research Report No.-182.

.

Baltimore, Maryland: Center for Social Orgagization of. Schools,, -

i,

, -,,,,.

.. : .

The Johns Hopkins University, 1974.

Richards, J. M., Jr.; Karweit, N.;, add Prevatt,T. W. FORTRAN program

-for --Simulating iducatiOriair growth. With 'varying achoOl impact ,

Educational and Psychological Measuremene, in prer.A.

I

,Comparison of Adjusted, NeanOain Parameters With

Average

Obtained Tide"Cain.ScoresforlipothetiOalSChoOls'

School

School

Impact

1L.20

21.1-0--

31.00

4.90

5.80

trarying in Impact -

,

Adjusted 'Mean Gain 'Parameter

-,For computingTrUe:Gain

Scores in

Or

of

'Average of Obtained True Gain

Scbres,in, Interval:

,i, Occasions' Agccasibps,

cas,

Ocions'

,04asions

1to

22 ''tr.',3

,-37 to,

*A. .to,',2

;.,'

1.0392

.9852

.7812

1.0392

i,

,

.9526 ':

.9031,

.7161

.9526

".,86 6o-

'.8210

,,

.,6510''

'

',,..:8660 ,,'.

.7794'

':7389'"H

,,

.'59,'' ,.,,,.

,7794

.6928

.6568-

.520S'

.6928

Occasions

Occasions

2.to

'3

3to 4:

.9277

.,8744'

.6672

.8210

.6510

.7676

.6348

.7143

.6186

NOTE':

Gain 'scores are exiiiessed inunits of the trde s'corestandard,dViation for

OCcaSiOn 1:

Iris assumesItheseschools, are

in ever*Y;respect except

H. for differipg' in impact

Values for"Schdoi 3 'are

identical with those

'.(

estimated'from the,ETS'Growth.Study.

Change' Estimate

posttest' Score

tO

CjPostteit Score

AdjUsted'For' Initial

'Academic 'Poteniiil

Raw 'Gain

'Residual Gain

Table

Correlations ,Between,Aveage'TrueGaih-in Diff rentSchodls

and Various'4timates of Average Change

(N's = 100)

',Correlation Between

'Reedurcei and Impact",

.2236

.4472,

.8944

NOTE:

.2236.

.4472

8944

.2236'

.4472

.8944

N.2236

-8944

.Assignment of Students

Noniandoli

Random

,

Average 'True Gain in Interval:

Average True Gain in

1to

22

to

33

to

41- to 2" 2

to

3

M

72

'77

70 86

'85

77;

'70

8087

85

"'

90

90

49, .89

88

88

'190

0,0

88

,88

,

Numbers: in,columns, labelled I :show correlations betweenaverage true gain

69,

80 H

.75

75''.42

'42'

42'

36

36

42

-46

4,

37

l3.7

49'

,,e4.9

72

72,,

86

.85

80

80

76

74

87

77 0

77

'80

43,

88'

71";

82

68'

71

71,

47,

67'

73

73'.

70

73'

658

73

75'

47

47

,S4

54

50

504

78

88

:xi,

90

82,

76 90

,91',.

91

'86

86

79

AV

+

85

83

80

80,

45

45

43

43

38

.38

7382

75

77

80 79

69 70

65'

6.9

81

76

77

80 79

70

.34

and

estiMates),Ased'On,indiVidUar,studenia Numbers' in columns labelled M show

Correlations, kor'estimate6 baSed'onaChoolHmeans.: Decimals are omitted.

Foi

'this and, subsequent, tabieil'" = .14,and,r '=-\18"

er:

Art

Interval

. 3

to .4

IM

0011

-03

0011

-03-

33

47

41

.4Q

:24-38

65

.70

'61

2235 19

65 70

61

'65

70sr

tt

IS8le

Co,:reIatiOns'o", Sctiool ImpdO,i,With4Ohailie',MeeSures and Estimites

),

'Change Measured 81,.

AVerSge'True' Gain

From OCasslon'A.

Avetagetrue

Gai

n,From Judtyreyious

Occasicin

)

'Change' Estimated

By:L

Posttest' Score

110

)

ToSttest Score Adjusted

'FOr Initial Academic

1-'7-Potential

Raw Gain Frbm-TOccasion 1

4

Raw Gain From Just

PreuioUt Occasion

"Residual Gain Trotti ()Cession

,rResidual'Gain From X*:

Previous Occasion,'

Correlations etween

R#sourcesend,Impact'

.8944

2236

.4472

8944

.2236

.4472

8944

.2236

.4472

.8944 '

.2236

q4472

:8944

. .4472

.8044 .7

.2236

.4472

.,8944

.2236

.4472

,.8944

'.87

,90

.89

!87

00

.644

.71

.63

.69

.83

.80

.90

.78

:85

:77

.81

.86

.75

.53

.78

.70

.81

.57

77-

.79,

.88

80

.85

.77

.31

.39

43

.80

:40

'Change to Occasion:

,3".

42

.92

05

.04

.71

.82

.70

77.

.83

.94'

.95

.18

.21

.35

'.77

.84

.80

.80

.90

.85

.88

.90

.12

.14

.18

.85

.96.

.30 7"

21

.30

4

.86

.p2

.94

.88

.93

.94

.83

.90

.92

.86

.71

.35

..88

.82

.46

:83

75

.33

.27

.34

.37

,31

.39

.43

.22

.31

.35

.73

.84

.87

.80

.87

.83

.87

.79

.85

.89

.82

.89

.92

-.71

.85

.89

.79

.54

.26

.82

.71

.66'

.30

.78

".85

.89

.82

.89

.92

.86

.88

.78

- .25"-

.82

.61

.34

.70

.63

ti

tOrXelations, Of

'Change Measured By:

Average' True Gain

".From Occasion' 1

AVerageTrUe Gain

From JusP':PreViOds

Occasion. °

ir

4't114ilge Estimated .13V:

7r

POsttest,ScOre

60[Foit:test-Score Adjusted

For Initial, Apademic

Potential

Rati Gain From Occasion 1

pol Resources with Change

Measureii4and Estimates

(N's = 100)

Correlations Between

ResOurcesiand Impact

.2z36

12236-,

L4472'

.8944

42236

.4472

.8'944

f .2236

t:484942

.2236

.4472

.8944

2236

.4472

;8944

282934641

.4472.

.2236

.447211

'1.8944

Raw Gain"FromJuit.

Previous Occasion

'ResIdUal Gaim

,*

FromOcciSiOn,1

ReidUalrein,Ftpm Just

PrevioutOCcasicin:

Assignment of Students

Random

'Nonrandom

Change to Occasion

Charige'tobccision

23

"4

23

4

.20

',

415

.13

,.22

.21

.18

.

-.38

,.42

,42

.50

'.52

.50

.78

483"

,83

-.74

.82

,.85

o.20

,.03

-.04 '

.22

.12

-.01

.38

.39

.09',

.50

.45

..16

.78

.69

.29

..74

.72

.31

.11

1,

.06

,,08

.22

.23

.22

.23

.33-

.34

,..36

.39

.41

.53

.66

.69'

',21

:29

.33

,1..,..-

.,07

:15

.(:)Ei

.15'

;14

.11

.

28'

437

.38

7.42

,.42

445

.65

.75

.75

,.66

.,,78

.80

.21

.12

.14

.21

.20

.17

.31

.39

.39

,.48,

.147

.50

.68 -

.76

.78

..66

.78

.80

.21

,'

-406

-.05,

.21

:10

'-402

.31

.34

.07

.48

,.28"

,.22

.

.68

.54

.12

.62

;63

.24

.05_,.... ..21

..12

,.17

.15

.12

'.31

',',.40

.39 :

.41,

,.41

.44

.68..

.76

.78

.62'

478

.79.

:OS

.13

-.02,

,.17

,.04

-.02

431

.26 °

.09 ..

.41

.21

-.th

0:

'.40%.

,.23

.62,

..59

,.32

P