LEV-EL/ - DTIC · Contract No. N00014-80-C-0402 Contract Authority Identification Number NR No. 150-453 ... bi ,and ci are item parameters describing item i ~. bi. Unbiased Estimators

RR-81-50LEV-EL/UNBIASED ESTIMATORS OF ABILITY PARAMETERS,

OF THEIR VARIANCE, AND OF THEIR

PARALLEL-FORMS RELIABILITY

o0

S-Frederic M. Lord

This research was sponsored in part by thePersonnel and Training Research ProgramsPsychological Sciences DivisionOffice of Naval Research, underContract No. N00014-80-C-0402

Contract Authority Identification NumberNR No. 150-453

Frederic M. Lord, Principal Investigator

Educational Testing Service D TICPrinceton, New Jersey ELECTE

November 1981 DEC 28 1981-.

DReproduction in whole or in part is permittedfor any purpose of the United States Government.

Approved for public release; distributionunlimited.

f3p112 28060,

UNBIASED ESTIMATORS OF ABILITY PARAMETERS,

OF THEIR VARIANCE, AND OF THEIR

PARALLEL-FORMS RELIABILITY

Frederic M. Lord

This research was sponsored in part by thePersonnel and Training Research ProgramsPsychological Sciences DivisionOffice of Naval Research, underContract No. N00014-80-C-0402

Contract Authority Identification NumberNR No. 150-453

Frederic M. Lord, Principal Investigator

AceesSion For--- Educational Testing Service

ITIS ORANIDTIC TAD N Princeton, New JerseyUnannounced 1JtitloatlO . .November 1981

Dist ribut tou/_Reproduction in whole or in part is permitted

Availability Codes for any purpose of the United States Government.Avall and/orDist S peciall,

II I Approved for public release; distribution

unlimited.

- 7 I J

UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE (*%en Data Entered)

REPORT DOCUMENTATION PAGE BEOECOL NOM1. REPORT NUMBER 12. GOVT ACCESSION NO. 3. RECIPIENT-S CATALOG NUMUEER

4. TITLE (end Subtitle) 5. TYPE CF REPORT & PERIOD COVERED

Unbiased Estimators of Ability Parameters, of Technical ReportTheir Variance, and of Their Parallel-FormsReliability 6. PERFORMING G09. REPORT #V141111

Research Report 81-50

7. AUTHOR(&) S. CONTRACT ORl GRANT NUM11I[R1(s)

Frederic M. Lord N00014-80-C-0402

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASKAREA & WORK UNIT NUMBERS

Educational Testing ServicePrinceton, NJ 08541 NR 150-453

II. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT OATE

Personnel and Training Research Programs November 1981Office of Naval Research (Code 458) IS. NUMBER OF PAGES

297T4 MONITORING AGENCY NAME & ADDRESS(II different from Controllinl Office) t5. SECURITY CLASS. of this report)

S1. DECLASSIFICATION/ DOWNGRADINGSCHEDULE

16 DISTRIBUTION STATEMENT (of this Report)

Approved for publiL release; distribution unlimited.

17, DISTRIBUTION STATEMENT (of the abstract entered In Block 20, If different from Report)

IS, SUPPLEMENTARY NOTES

19. KEY WORDS (Continue on reveree oide If necesesry and Identify by block number)

Bias (statistical), Estimation, Item Response Theory, Ability, True Score,Reliability, Maximum Likelihood, Mental Test Theory, Unbiased Estimate,

Standard Error, Asymptotics

20. ABSTRACT (Continue on everse side it necessary end Identify by block number)

G4.en known item parameters, unbiased estimators are derived 1) for anexaminee's ability parameter 0 and for his proportion-correct true scoreS~2

2) for s the variance of 0 across examinees in the group tested,2

also for s , and 3) for the parallel-forms reliability of the observed

test score, the maximum likelihood estimator 0

D I JAN°73 1473 EDITION OF I NOV , G IS OBSOLETE

S N 0102- LF- 014- 6601 SECURITV I No/SEUIYCLASSIFICATION OF TNIS PAGE (R11en Pat* Mature-f)

Unbiased Estimators

Abstract

Given known item parameters, unbiased estimators are derived

1) for an examinee's ability parameter 0 and for his proportion-

correct true score ,2) for s 2 the variance of 8 across examinees

2 -omin the group tested, also for s~ and 3) for the parallel-frm

reliability of the observed test score, the maximum likelihood estimator,

6.

Unbiased Estimators

2

Unbiased Estimators of Ability Parameters, of Their Variance,

and of Their Parallel-Forms Reliability

4--- his paper is primarily concerned with determining the statistical

bias in the maximum likelihood estimate e of the examinee ability .--

parameter ' in item response theory (IRT) [Lord, 1980]; also of

certain functions of such parameters. We will deal only with uni-

dimensional tests composed of dichotomously scored items. We assume

the item response function is three-parameter logistic (2).

Available results for the sampling variance ofC are currently

limited to the case where the item parameters are known; the present

derivations are limited, to this case also. This limitation is tolerable

in situations where the item parameters are predetermined, as in item

banking and tailored testing.

In the absence of a prior distribution for 8 , it is well known

that examinees with perfect scores h~ve',8 also that examinees

who perform near or below the chance level on multiple--choice •7•rms

may be gi.-.n large negative values of ,_Q. This (correctly) suggests

that 8 is positively biased for high-ability examinees and negatively

biased'for low-ability examinees. Will a correction of 0' for bias

be helpful in such cases?

*This work was supported in part by contract N00014-80-C-0402,project designation NR 150-453 between the Office of Naval Research andEducational Testing Service. Reproduction in whole or in part is permittedfor any purpose of the United States Government.

oI,

Unbiased Estimators

3

It is also 'well known' that for any ordinary group of examinees,

2the variance ( sý ) of 0 across examinees is larger than the variance

222( s• ) of the true 0 . The ratio sý/s_ is closely related to the

classical-test-theory reliability of 6 considered as the examinee's

2 2test score. Thus it i3 not enough for us to know that s• ÷ sa as the

number n of test items becomes large; we need to know how the rela-

2 2tion of sý to sa varies as a fbnction of n . We also need a

2 2better estimate of sa than its maximum-likelihood estimator s6

These objectives can be achieved by correcting s3 for bias.

--. -•The methods used to derive formulas for correction for bias are

presented herevin detail for at least two reasons: 1) experience

with similar derivations has shown that it is easy to reach erroneous

results if details are not spelled out. 2) The general methods used

here are easily transferred to solve other problems, such as a) cor-

rection of item parameters for bias, b) obtaining higher-order approxima-

tions to the sampling variance of e

1. Statistical Bias in 0 and r

The method used here to find the bias of 8 is adapted from the

'adjusted order of magnitude' procedure detailed by Shenton and Bowman

(1977). They assume their data to be a sample from a population divided into

a denumerable number of subsets. For them, the population proportion

of observations in a given sub3et is a known function of the param-

eter e whose value they wish to estimate. Their sample estimate of

o is therefore a function of observed sample proportions in the

t

Unbiased Estimators

4

various subsets. Since our data do not readily fit this picture, we

cannot uee their final published formulas but must instead derive our

own.

Thrtughout Section 1, we deal with a single fixed examinee whose

ability e is the parameter to be estimated. All item parameters are

assumed known.

i.1 Prelimninaries

The maximum likelihood estimate G is obtained by solving the like-

lihood equation

n (Lii - P~P/~ 1 - 0(1)i'l

where u 0 or 1 is the examinee's response to item i ( i - 1,2,...,

n ), P i P 1(0) is the response function for item i Q i E 1 - Pi

PIt is the derivative of P, with respect to e , and a caret indicates

that the function is to be evaluated at ; . We deal wizh the cese

where P is the three-parameter logistic function

+- c~ + (2)i i -Ai(6-b)

1+e

where A, bi ,and ci are item parameters describing item i

~. bi

Unbiased Estimators

5

We will assume

1. 6 is a bounded variable,

2. the item parameters ai and bi are bounded,

3. ci is bounded away from 1,

(thus Pi and Q, are bounded away from 0 and 1);

4. as n becomes large, the statistical characteristics of the

test stabilize.

Rather than trying to define this last assumption formally, the reader

may substitute the more restrictive assumption usually made in mental

test theory: that a test is lengthened by adding strictly parallel forms.

With these assumpttons, the conditions of Bradley and Gart (1962)

are satisfied. It follows from their theorpms that ; is a consistent

estimator of 8 and that 'nW (0 - 0) is asymptotically normally distri-

buted with mean zero and variance lim . niP2 /Pi . The existence ofr-mn i i Ii

this limit is guaranteed by assumpt'on 4.

For compactness, we will rewrite (1) as

n .LI 1 5 rii 0 (3)

where by definition

ii

Unbiased Estimators

rli (ui - P i)P/PiQi (4)i i

Now L1 considered as a function of 6 can be expanded formally in

powers of 0 - 6 as follows:

(61-6) 2 z 2 +L1 zii ÷i(( -) r i3iS i 1 +T3

where we define

o P Qiu8 s - 1,2,... ) . (5)8, de S

This definition is Lonsistent with (3).

Let x = a - o, r = P . Rather than proving the con-Ss sirsivergence of the power series, let us use a closed form that is always valid:

1 1 3 6 3L = r 4 xr ++ x r + L x4r (6)1 1 T2 + 3 6 4 24 5()

where r 5- Max r5 and 161 < I.

1.2 Derivatives and Expectations

To proceed further, it is necessary to evaluate the r . It iski

found that

IIVIVPM_

Unbiased Estimatots

7

S(_Ai)k-I *,' uicir ki ( -A si Sk-k- (i -Q' I ) (-I + -u' '- )(7)

ri

where - is a Stirling number of the second kind (Jordan, 1947,

pp. 31-32, 168).

Define

Ys i r 5si (8)

8s -rsi - srsi8 (9)

Since Sut M Pi D we find that

y1 i 0 , (10)

*'Pi' (21

AI Pi (P )201 c i) ( c (2y3 (1 c( ) 2 P 2

Eli r li (u i- Pi)P;/P1 QiQi (13)

A c P'(u - Pi)" ii I (14)21P

Let

YS y Ysi/n c -- E e sn /(15)

s si

S. .... . ... . >'4

Unbiased Es timators

8

We will denote the Fisher information by

I i -6(dLl/de) -ny 2 P /P (16)

Setting (6) equal to zero, the likelihood equation can now be written

in terms of the y and the c asM 1 x2 + 1 x3 (.4 + +_ 4r

-1 x(Y 2 + £2) + 2 (Y3 + Y3) +6 + £4) +C -2x4T 5 (17)

We will need some information about the order of magnitude of the

terms such as those in (17). It may be seen from (7) that each cs

has the form

C I Z Ki(usi (

where Ksi does not depend on n or on u . Since P, , QI and

1 - Ci are bounded, the K., and thus cs is bounded. By assumption

(4), the bound does not depend on n . The same conclusion holds for y.

Since /n x is asymptotically normally distributed with zero mean

and finite variance, it follows that Sxr ( r r 1,2,... ) is of order

n-.r/2. A similar statement is true of / e . Thus finally Oxrct <S 8 -

(fixr2 so that 6xrt is of order n-(r+t)/2 r,t = 1,2,... ).s S

Unbiased Estimators

9

1.3 First-Order Variance of e

To clarify the procedure, let us derive from (17) the familiar

formula for the asymptotic variance of 8 . Square (17) and take

expectations to obtain

2 2 2 23 3

"i: =y 2 ++ 2 + Y2r362 + Y2 5x + ,.. (18)

If we wish to neglect terms o(n-) (of higher order than n-),

equation (18) becomes

6x 2 = 1• S2 + o(n-1) (19)2 1 2 -

By (13) and (16), because of local independence,

2 P1

e1 5-- E i (uiPi J p (u P

"n -- i £ (u- )ui -j )

n 2 1J P i J j

1 2 2 Var u

22n i

n21 P t~n i P~

I2 (20)

n£ _ __

1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Unbiased Estimators

10

Thus, finally

^' 1 n-1Var 0 + o( ) (21)

a well-known result. It is derived here to clarify the reasoning toA

be used subsequently. If 6 is substituted for 0 on the right side

of (21), the formula will still be correct to the spec'fied order of

approximation.

1.4 Statistical Bias of 8

Take the expectation of (17) to obtain

1 2 (22)

11 2 + 21 1X 2 +Y 3 ix 3 1

-1where 6 indicates an expectation in which only terms of order n are to

be retainedý Also multiply (17) by c 2 and take expectations to obtain

- CC22 2 = 1 xC2 (23)

By (9)

6 - 0 r - 1,2,... . (24)r

Trow (13) and (14)

Unbiased Estimators

11

1 A c P'2

c 1 i i i 21• 12 n 2 c, P32 ssui Pi)2

"1 2 1-c 3 2-

n i i PiQI

1 c 2 (25)=-2" :(1- ci) Pi

i

Substituting (16) and (25) into (23), we have the covariance

i A.,ci P' 2

x = -- c£(26)

Pi

Finally, substituting (16), (21), (24), and (26) into (22) and solving

for lX l , we have the bias

A Ac p'2B 9 ( - e i i-c 2L + ny3 ) (27)

i

This may be rewritten as

1 n1B 1(e) =2ZA I1(28)

where

Pi- ci a i p 2

1 - c i Q (29)

Unbiased Estimators

12

Since I is of order nl B1(6) is of order nl It may be of

interest to note that in the special case where all items are equivalent

(all P are the same), the bias simplifies to B 1(6 P/nP'

1.5 Numerical Results

A hypothetical test was designed to approximate the College

Entrance Examination Board's Scholastic Aptitude Test, Verbal Section.

This test is composed of n -90 five-choice items. Some information

about the distributions of the parameters of the 90 hypothetical item

is given in Table 1.

The standard error and bias of ; were computed from (21) and

from (27) respectively for various values of 6 The results are

shown in Table 2. It appears that the bias in e is negligible for

moderate values of e , but is sizable f or extreme values. Note that

the bias is positively correlated with e Because of guessing, zero

bias does not occur at 3 - 0 but at 0 .34 approximately.

1.6 Variance and Bias of Estimated True Score

Since the ability scale is not unique, any monotonic transformation

of e can serve as a measure of ability. Twio transformations aret

particularly useful: e0 and

=- P (W (30)-n

the proportion-correct true score (the number-right true score divided

Unbiased Estimators

13

TABLE 1

Range and Quartiles of the Item Parametersin 90-Item Hypothetical Test

ai&Al/3. 7 b c 1

Highest value 1.88 2.32 .47

Q1 1.07 1.15 .20

Median .83 .38 .15

Q3 .69 -. 41 .13

Lowest value .41 -3.94 .01

I

__i

Unbiased Estimators

14

by the number of items). One important reason for using the latter

transformation is the following.

Ordinarily, as in Table 2, we find large standard errors of

8 where 6 is extreme. Usually these large standard errors are no

more harmful to the user than are the smaller standard errors found

when e is near the level aimed at by the test. There is a reason

why this is so: If it were not, the user should have designed his test

so as to reduce those standard errors that were troublesome to him.

We see that from this point of view the size of a difference on

the 0 scale does not correspond to its importance. The discrepancy

is greatly reduced, however, if we measure ability on the C scale

instead of on the 8 scale. This is one reason, among several, wny

we are interestcd in the variance and bias of

E : ip(8)/n (31)

Although the proportion-correct true score

z u 1u1 /n (32)

is an unbiased estimator of C , z is never a fully efficient

estimator of C unless ci - 0 and ai , aj ( ij - 1,2,...,n ):

the sampling variance

nVar z r P Q (33)

n i2 1

Unbiased Estimators

15

TABLE 2

Standard Error and Statistical Bias in

e rVar e (0)

3.5 .60 .24

3.0 .43 .12

2.5 .31 .06

2.0 .23 .032

1.5 .19 .011

1.0 .19 .0032

0.5 .20 .0012

0 .22 -. 0028

-0.5 .25 -. 010

-1.0 .31 -. 025

-1.5 .41 -. 05

-2.0 .54 -. 09

-2.5 .70 -. 14

-3.0 .89 -. 22

-3.5 1.09 -. 31

.~-.--,---~~-~!

Unbiased Estimators

16

Aia not as small as the sampling variance of 4 , which we must now

derive.

By (31)

A n-~!- i PdeC .3 r

(34)

Using the 'delta' method

Var C ,- - ( E ,)2

n 0 P) Var2

By (21) and (16)

ii

Var C( 5n r Pi__ 2

(

i Pii

To find the bias of 4 we expand it In powers of x= -%:

- 2n 2n Epi +2"A- i 1 ''" (36)

Unbiased Estimators

17

where

pit d 2 P /do2

Taking expectations, and neglecting higher-order terms, we have for

the bias

(4 E + Cf()Z Var 0 (iP")i (37)1 n i 2 p

This can be rewritten as

,2I ~Aio ;2

SI •"" (l- + Z 1 +2 (38)S- i)Pi i

where C' Z •iP'/n and C" = P"/n . Let us note is passing that when

all items are equivalent (all P (6) are the same), 4 - z and its

bias (38) is zero.

1.7 Numerical Results

Table 3 shows the bias in C for the same hypothetical test con-

sidered in Section 1.5. The biases are all positive. However, they are

negligible at all except the lowest ability levels. This tends to con-

firm our choice of the 4 scale of ability rather than the 6 acale

for many purposes.

Unbiased Estimators

18

TABLE 3

Standard Error of z and of Cand Statistical Bias of

8 lVarz 'Var __

3.5 .981 .014 .014 .00045

3.0 .966 .019 .018 .00052

2.5 .937 .024 .023 .00064

2.0 .891 .031 .029 .00059

1.5 .812 .037 .035 .00021

1.0 .715 .042 .040 .00026

0.5 .608 .045 .042 .00061

0 .506 .046 .043 .00061

-0.5 .416 .047 .042 .00062

-1.0 .344 .046 .038 .00061

-1.5 .291 .045 .037 .00085

-2.0 .254 .044 .033 .0014

-2.5 .227 .042 .029 .0020

-3.0 .211 .042 .025 .0024

-3.5 .199 .041 .021 .0026

____t

Unbiased Estimatore

19

As a matter of incidental interest, for selected values of true

score Table 3 compares the standard error (35) of the maximum-likell-

hooJ estimator 4 with the standard error (33) of the unbiased estimator

z (proportion-correct score). There is little difference in accuracy

between the two estimators for 4 > .5 . At low tree-score levels, the

maximum-likelihood estimator is much better than the proportion of

cc rrect answers.

_o__ ____s2 o s2;Tt eiiiy

2. Unbiased Estimation o fa;Ts eiblt

2 2The symbols s2 and s are used for the sample variance of

8 and of C across the N examinees in the sample:

2 1 N 2 1 N 2sB N Pa N a) (39)

aan

an 2 22The maximum-likelihood estimators Of 8 ad9 are s^ and a?e

the sample variances across examinees of e and of C

2.1 Asmptotically Unbiased ;ýLIlmator of 2

Assume that t'ir examinees are a random sample of N from some

population. Denote by the population variance of 8 e Then Ne2 /(N-l)

2 2ia an unbiased estimator of 08 a Since s is unobservable, our first

task is to find a function of 0 that is an asymptotically unbiased

estimator of 2 2

0N

Unbiased Estimators

20

By the formula for the variance of a sum we have

a 0 2 a2 2+ 2a (40)e E +x 0 x ex

2where a denotes a variance across all examinees in the population

and ax is the corresponding population covariance. By a well-

known identity from the analysis of variance

2 2 + 2 (41)

x - xe 6(x18)

where • denotes an expectation across all examinees in the population.

Similarly,

c ol t ,s'xIB) (42)

Substituting (41) and (42) into (40), transposing, writing B, _ 1 (xle)

as in (28), and dropping the subscript from B for convenience, we

have

2 2 _ 2 a2 2 (43)e e-, OB exle 0B

Since by (28) B is of order n , its variance is of order n2

so aB can be neglected in (43). Since Section 1 deals with a singleB2

fixed examinee, the symbol a 2 in (43) has the same meaning as

Var e in (21):

2 1+o(a1 )oxle " o-

Unbiased Estimators

21

where I 1 1(0) is givme by (16). Since 2 -1 theXI i

effect of replacing e by e on the right in neglibible:

5c2 1n~~~+~ 1

8 xle -169)

By similar reasoning, we may replace a in (43) by a^ where B

is defined by (27) with 8 replaced by 8 • The result of these

approximations is that

2 2a8 -y - -2 - - + o(n-) . (44)

2

A useful estimator of oa can be calculated fromN

.2 N 2 2N N 1" -Ng-i N - 1 Si - N E1

(45)

a 16a

where

N N N-a;B E eB (a E ABas8BN •a-l a~a e ',/1;a)(i • a- )a-i aw a

and Ba is given by (27) with e replaced by e a , If we wish to

2estimate the sample variance of ability sB rather than the population

variance a, , we can use

"2 2 N-s 3 2s-B- (46)

a 6B N 2 a-i I(ea

* .. . .. ... .. . . ..

Unbiased Estimators

22

The second and third terms of (44)are of order n-1 , an order

of magnitude smaller than the first term but larger than the neglected

terms. The covariance of e and B is usually positive, as can be

readily seen from Table 2. Since 1(0) is necessarily positive, it

2 2appears that usually a ( , an inequality that is frequently assumed

without proof. It is not clear whether this inequality is necessarily

true.

Consider the parallel-forms reliability coefficient p'

the correlation between scores e and e' on two parallel tests.

For present purposes, two tests are parallel when for each item in one

test there is an item in the other test with the same item response

function. Let us estimate

a.;I a,

P - (47)

from a single test administration by substituting asymptotically unbiased

estimators of the numerator and of the denominator into (47).

As in (41),

coo + r (48

Unbiased Estimators

23

Because of local independence, the first term on the right vanishes.

Because of parallelism, the two expectations in the last term are

identical, so this term Is a variance. We thus have

a.2 a 2 + a 2 (49)901 S~ij) -B" B 6 + '01

From (49) and (43),

a _ xia (50)

We see that the parallel-forms reliability of 0 is

pai -- •S8 --- +o(n-1) (51)o 1 (e)

Priority in obtaining this result belongs to Sympson [Note 1].

Replacing population values on the right by the corresponding sample

statistics, we have a sample estimator of the parallel-forms reliability

coefficient of 8

N NP_ = 1 N-i 1 (52)

Be N 86 a-l 1(;

Since 8 is neither unbiased nor uncorrelated with 8 , we

should not expect the usual reliability formulas of classical test

theory to apply. A similar but not identical case is discussed in

• , -,, •,.•.•M•• • ,,,.".., .,,

Unbiased EstiWators

24

Lord and Novick (1968, Section 9.8). Thus p2 and

22a 8/a are not interchangeable definitions of reliability. Since

correlational measures are hard to interpret in the absence of

linearity and homoscedasticity, we will not now push this investigation

of reliability further.

2.3 Corresponding Results for True Score

By the same reasoning used to obtain (44) we have

2 2 2 2a* ;aZ- 2 e B14 - GBo a

- 2 __L2 -1a 2l _ + o(n- (53)

2A useful estimator of a can be calculated from

.. N 2 2N 1 N a54c{-N-1 s• N - 1 s••• - N 1T(a (54)

a-I T(e)a

To estimate 2 2 we can use

S2 N -l1 N {a,2

N I

'2 2s^.i() aN• a-i 10 (a

Unbiased Estimators

25

As in (50) - (52) we have

S(2 56)

() + o(n-l) (57)

28ý, E 1 (58)N sA a-l I(ea)

2.4 Numerical Results for True Scores

At moderate ability levels, (28) provides adequate but usually

neglible corrections for bias in e . Experience shows that at

very low ability levels, the usual test length ( n ) of 50 or 100

items is not long enough for the asymptotic results of (28) to apply.

For example, an examinee whose true e is -3 may easily obtain an

estimated ability 0 of -30 or of -= . For sufficiently long tests,

such extreme values of 0 %ould have negligible probability, but

with the usual values of n , equation (28) is totally inadequate for

correcting ; for bias at low ability levels.

This sair difficulty carries over to the unbiased estimation2

of o2 using (46). Since all ability levels are involved in (46),

the formula is useless in practice for any group that contains even

a few low-ability examinees. Fortunately, this difficulty does not

Unbiased Estimators

26

carry over to the estimation of ability on the true-score ( • )

scale.

The hypothetical SAT Verbal Test of Tables 1-3 was administered to

a typical group of 2995 hypothetical examinees. The bias in C

was estimated for each examinee and a corrected • obtained from

(51):

corrected C - B -lBI

In a few cases where the corrected • would have been below the

chance level En the corrected C was set equal to E c

The mean of the 2995 true C used. to generate the data was

.5280, the mean of the uncorrected Z was .5294, the mean of the

corrected Z was .5288. Thus the correction was in the right

direction, but not large enough. The uncorrected mean t was already

so accurate as to leave little room for improvement.

Next, (55) was used to estimate sa. The true value was

s4 .1610 , the standard deviation of • was s. - .1660 , the cor-

rected estimate from (55) was § - .1614 . The correction worked

very well here.

AAThe paralJlel-forms reliability of • was estimated from (58)

to be p^ M .9420 . We have no 'true' valuc against which this can

be compared, but the estimate seems a reasonable one. The Kuder-

Richardson formula-20 reliability of number-right scores for these

data is .9275.

a1

Unbiased Estimators

27

It should be remembered that both the formulas and the numerical

results In this report apply in situations where the item parameters

are known. These formulas may be satisfactory for situations where

the item parameters have been estimated from large groups not containing

the examinees whose ability estimates are to be corrected for bias.

These formulas will not be adequate for situations where the item

parameters and ability parameters are estimated simultaneously from

a single data set.

vi

Unbiased Estimators

28

Reference Note

1. Sympson, J. B. Estimating the reliability of adaptlve tests from

a •single test administration. Paper presented at the meeting ofthe American Educational Research Association, Boston, April 1980.

i iit[I

-gi1._

__r_ _ _

Unbiased Estimators

29

References

Bradley, R. A. & Gart, J. J. The asymptotic properties of ML estimators

when sampling from associated populations. Biometrika, 1962, 49,

205-214.

Jordan, C. Calculus of finite differences (2nd ed.). New York: Chelsea,

1947.

Lord, F. M. Applications of item response theory to practical

testing problems. Hillsdale, N.J.: Lawrence Erlbaum Associates,

1980.

Lord, F. M. & Novick, M. R. Statistical theories of mental test

scores. Reading, Mass.: Addison-4esley, 1968.

Shenton, L. R. & Bowman, K. 0. Maximum likelihood estimation i.n

small samples. Monograph No. 38. New York: Macmillan, 1977.

-2-

Technical DirectorNavy Personnel R & D I Dr. Bernard Rimland (030)San Diego, CA 92152 Navy Personnel Rt & D Center

San Diego, CA 921526 Commanding Officer

Naval Resevch Lahoratory I tr 1nold RubensteinCode 2627 Of7. Lt of Navel TechnologyWashington, DC 20390 600 N. Quincy Street

IPsychologist Arlington, VA 22217

ONi Branch Office 1 Dr. Worth Scauland, DirectorBuilding 114, Section D Research, Development, Test666 Smmer Street and EvaluationBoston, MA 02210 N-S

Naval Education and Training CommandOffice of Naval Research HASCode 437 Pensacola, FL 32508800 North Quincy StreetArlington, VA 22217 1 Dr. Robert G. Smith

Office of Chief of Naval Operations5 Personnel and Training Research OP-987H

Programs Washington, DC 20350Code 458Office of Naval Research I Dr. Alfred F. SmodeArlington, VA 22217 Training Analysis and Evaluation Group

Department of the NavyPsychologist Orlando, FL 32813ONR Branch Office1030 East Green Street 1 Dr. Richard SorensenPasadena, CA 91101 Navy Personnel R & D Center

San Diego, CA 92152SOffice of the Chief of Naval Operations

Research Development and Studies Branch I Mr. J. B. SympeonOP-115 Naval Personnel R & D CenterWashington, DC 20350 San Diego, CA 92152

LT Frank C. Petho, NSC, USN (Ph.D.) I Dr. Ronald WeitamanSelection and Training Research Division Code 54 WZHuman Performance Sciences Department Department of Administrative ServicesNaval Aerospace Medical Research Lab. U.S. Naval Postgraduate SchoolPensacola, FL 32508 Monterey, CA 93940

1 Dr. Robert Wherry562 Mallard Drive

Chalfont, PA 18914

L __ _ _ _ _ _ _

TI i -- -

DISTIIBJTION LIST

Navy

I Dr. 6d Aiken I Dr. Patrick i. MarrisonNavy Personnel R & D Center Psychology Course DirectorSan Diego, CA 94152 Leadership and Law Department (7b)

Division of Professional DevelopmentI Dr. Jack R. Borsting U.S. Naval Academy

Provost and Academic Dean Annapolis, MD 21402U.S. Naval Postgraduate SchoolMonterey, CA 93940 1 Dr. Norman J. Kerr

Chief of Naval Technical TrainingI Dr. kobert Breaux Naval Air Station Memphis (75)

Cod* N-711 Millington, TH 3d054NAVTKAEQU IPCENOrlando, FL 31613 1 Dr. William L. Naloy

Principal Civilian Advisor forI Chief of Naval Education and Education and Training

Training Liason Office Naval Training Command, Code OOAAir Force Human Resource Laboratory Pensacola, FL 32S06Flying Training DivisionWilliams Air Force Base, AZ 8I2'4 I Dr. Kneale Marshall

Scientific Advisor to DCNO(MPT)I CDR Mike Curran OpUIT

Office of Naval Research Washington, DC 203706U0 North 4uincy StreetCode 270 1 Dr. James McBrideArlington, VA 22217 Navy Personnel R & D Center

San Diego, CA 92152Or. Kichard ElaterDepartment of Administrative Sciences I Dr. William MontagueNaval Postgraduate School Navy Personnel I & D CenterMonterey, CA 9394U San Diego, CA 92152

1 or. Pat Federico I Mr. William NordbrockNavy Personnel K • U Center Instructional Program DevelopmentSan Diego, CA 91152 Building 90

NET-PDCDI Mr. Paul Foley Great Lakes NTC. IL 60088

Navy Personnel I & D Cent~erSan Diego, CA 9;152 1 Library, Code P2OIL

Navy Personnel i & U CenterDr. John Ford San Diego, CA 92152Navy Personnel K & 0 CenterSan Diego, CA 9l252

iI

4 .-- - - '

-3-

Dr. Robert Wisher 1 Dr. Hilton S. KatoCode 309 Training Technical AreaNavy Personnel R & D Center U.S. Army Research InstituteSan Diego, CA 92152 5001 Eisenhower Avenue

Alexandria, VA 22333Dr. Martin F. WiskoffNavy Personnel R & D Center 1 Dr. Harold F. O'Neil, Jr.San Diego, CA 92152 Attn: PERI-OK

Army Research InstituteMr. Ted M. 1. Yellen 5001 Eisenhower AvenueTechnical Information Office Alexandria, VA 22333Code 201Navy Personnel R & D Center 1 Dr. Robert SasmorSan Diego, CA 92152 U.S. Army Research Institute for

the Social and Behavioral Sciences5001 Eisenhower Avenue

Army Alexandria, VA 22333

I CommandantTechnical Director U.S. Army Institute of AdministrationU.S. Army Research Institute for the Attn: Dr. Sherrill

Behavioral and Social Sciences Ft. Benjamin Harrison, IN 462565001 Eisenhower AvenueAlexandria, VA 22333 1 Dr. Frederick Steinheiser

Department of the NavyDr. Myron Fischl Chief of Naval OperationsU.S. Army Rerearch Institute for the OP-113

Social and Behavioral Sciences Washington, DC 203505001 Eisenhower AvenueAlexandria, VA 22333 1 Dr. Joueph Ward

U.S. Army Research InstituteDr. Dexter Fletcher 5001 liseuhower AvenueU.S. Amy Research Institute Alexandria, VA 223335001 Eisenhower AvenueAlexandria, VA 22333

Air Force1 COL Frank Hart

Army Research Institute for theBehavioral & Social Sciences I Air Force Human Resources Laboratory

5001 Eisenhower Blvd. AIHRL/1@DAlexandria, VA 22333 Brooks Air Force Base, TX 78235

Dr. Michael Kaplan 1 U.S. Air force Office ofU.S. Army Research Institute Scientific Research5001 Eisenhower Avenue Life Sciences DirectorateAlexandria, VA 22333 Bolling Air Force Base

Washington. DC 20332

A+

i zq

-4b-

Air University Library 1 Director, Office of ManpowerAUL/LSE 76/443 UtilizationMaxwell Air Force Base, AL 36112 HQ, Marine Corps (MPU)

SIS, Building 2009Quantico, VA 22134

Dr. Earl A. AlluisiHQ, AFHRL (AFSC) I MAJ Michael L. Patrow, USMCBrooks Air Force Base, TX 78235 Headquarters, Marine Corps

Code HPI-20Mr. Raymond E. Christal Washington, DC 20380AFHRL/MOBrooks Air Force Base, TX 78235 1 Dr. A. L. Slafkosky

Scientific AdvisorDi. Genevieve Haddad Code RD-iProgram Manager HQ, U.S. Marine CorpsLife Sciences Directorate Washington, DC 20380AFOSRBolling Air Force BaseWashington, DC 20332 Coast Guard

Dr. Ross L. MorganAFIIRL/LR I Mr. Thomas A. WarmWright-Patterson AFB, OH 45433 U.S. Coast Guard Institute

P.O. Substation 18Research and Measurement Division Ozlahoma City, OK 73169Research Branch, AFHPC/HPCYPRRandolph Air Force Base, TX 78148

Other DoDDr. Malcolm RoeAFHRL/MPBrooks Air Force Base, TX 78235 1 DARPA

1400 Wilson BoulevardDr. Marty Rockway Arlington, VA 22209Technical DirectorAFHRL(OT) 12 Defense Technical Information CenterWilliams Air Force Base, AZ 58224 Cameron Station, Building 5

AtLn: TCAlexandria, VA 22314

MarinasI Dr. William Graham

Testing DirectorateDr. ". William Greenup IMECOM/MEPCT-PEducation Advisor (E031) Ft. Sheridan, IL 60037Education Center, ICDECQuantico, VA 22134

-. * " :. .. .• .. . . . . . 'T • " •; ' . . .

Military Assistant for Training I Dr. Joseph L. Youn", Directorand Personnel Technology Memory end Cognitive Processes

Office of the Under Secretary of National Science FoundationDefense for Research and Engineering Wash.ington, DC 2055U

Room 3D129, The PentagonWashington, DC 20301

Mon-GovernmentDr. Wayne SollmanOffice of the Assistant Secretary

of Defense (KRAL) I Dr. Erling R. Andersen2B209 The Pentagon Department of StatisticsWashington. DC 2030a1 Studiestraeds 6

1455 CopenhagenDENMAK

Civil GovernmentI Psychological Research Unit

Department of Defense (Army Office)Dr. Susan Chipman Campbell Park OfficesLearning and Development Canberra, ACT 26u0National Institute of Education AUSTRALIAI2UU 19th Street, NWWashington, DC 20208 1 Dr. Alan Baddeley

Medical reaerch CouncilHr. itichard HcKillip Applied Psychology Ur•.tPersonnel Rt & D Center 15 Chaucer RoadOffice of Personnel Management Cambridge CB2 2AF1900 E Street, NW ENGLANDWashington, DC 2U415

1 Dr. Isaac BeJarDr. Arthur Melmed Educational Testing ServiceNational Institute of Education Princeton, NJ Ul6i112OU 19th Street, N.W.Washington, DC 2020$ I Dr. Menucha birenbaum

School of EducationDr. Andre%. R. Nolnar Tel Aviv UniversityScience Education Development Tel Aviv, Ramat Aviv 69976

and Research ISAAELNational Science FoundationWashington, DC 205S0 1 Dr. Werner Birke

UesWPs im StreitkraeftsumtDr. Vern W. Urry Postfach 20 50 3Personnel R & D Center 0-5300 Ionn 2Office of Personnel Management WEST GERMANY1900 E Street, NWWashington, DC 20415

_

-6-

Or.r. Darrell bock I Dr. William I. CoffmnDepartment of Education Lirector, Iova Testing ProgramsUniversity of Chicago 334 Lindquist CenterChicago, IL 60037 University of Iowa

Liaison Scientists Iowa City, [A 52242Office of Naval Research I Dr. Allan M. Collinsbranch Office, London Bolt, Beranek, 4 Newman, Inc.Box 39 50 Moulton StreetFPO, NY 09510 Cambridge, MA 02136

Dr. Robert Brennan I Dr. Meredith P. CravfordAmerican College Testing Programs American Psychological AssociationP.O. Box 16b 1200 17th Street, NIowa City, IA J2240 Washington, DC 20036

Or. John B. Carrol) 1 Dr. Hans CrombagPsychometric Laboratory Education Research CenterUniversity of North Carolina University of LeydenDavis Hall U13A Boerhaavelaan 2Chapel Hill, NC 27514 2334 IN Leyden

THlE NETHERLANDSCharles Myers LibraryLivingstone House I Dr. Fritz DrasaovLivingstone Road Yale School of Organisation andStratforu ManagementLondon 915 ZLJ Yale UniversityENGLAND Box IA

Dr. Kenneth E. Clark New Haven, CT 0652UCollege of Arts and Sciences I LCOL J. C. EggenbergerUniversity of Rochester Directorate of Personnelliver Compus Station Applied ResearchRochester, NY 14627 National Defence Hq.

101 Colonel By DriveDr. Norman Cliff Uttava, KIA UB2Department of Psychology CANADAUniversity of Southern CaliforniaUniversity Park I Dr. Benjamin A. Fairbank, Jr.Los Angeles, CA 90007 McFann-Gray and Associates, lrec.

5825 CallaghanSuite 225San Antonio, TX 78226

77I

Dr. Leonard Feldt I Dr. Bert Green

Lindquist Center for Measurement Department of Psychology

University of Iowa Johns Hopkins University

Iowa City, IA 52242 Charles and 34th StreetsBaltimore, MD 212lS

I Dr. Richard L. Ferguson I Dr. Ron ambleton

The American College Testing Program School of EducationP.O. box Ibd University of Massachusetts

Iowa City, IA 52240 Amherst, NA O100

Dr. Victor Fields I Ur. Lloyd HumphreysDepartment of Psychology Department of Psychology

Montgomery College University of gllinoisRockville, MD 20d)O Champaign, IL 01820

Univ. Prof. Dr. Gerhard Fischer I Library

Psychologisches Institut der INavme/Wstern Division

Universitat Wien 27d57 Bervick Drive

Liebiggasse 5/3 Carmel. CA 93911A OIUl WienAUSTRIA I Dr. Steven Hunks

Department of Education

I Prof. Donald Fitzgerald University of Alberta

University of Now England Edmonton. Alberta

Armidale, Niow South Wales 2351 CANADA

AUSTRALIA1 Dr. Karl Hunt

Dr. Edwin A. Fleishman Department of Psychology

Advanced Research Resources Organization University of Washington

Suite VOU Seattle, WA 9d105

4330 East West HighwayWashington. DC 20014 1 Dr. Jack Hunter

2122 Coolidge Street

I Dr. John R. Frederiksen Lansing, MI 4890b

Solt, Beranek, and Newman50 Moulton Street I Dr. Huynh Muynh

Cambridge, MA 02138 College of EducationUniversity of South Carolina

Dr. Robert Glaser Columbia, SC 29204LAWCUniversity of Pittsburgh 1 Mr. Marlin KrogerJ939 O'Hara Street 1117 Via Goleta

Pittsburgh, PA 15113 Palos Verdes Estates, CA 90274'

-a1-

Dr. Michael Levine I Dr. Melvin A. Novickuepartment of tiducational Psychology 356 Lindquist Center for Measurement210 Education Building University of IowaUniversity of Illinois love City. IA 1242zChampaign, IL 61601

I Dr. Jesse Orlanakyur. Charles Levis Institute for Defense AnalysesFacultelt Social* Wetenechappen 40U Army Navy Driveaijlksuniversiteit Groningen Arlington, VA 2220OOude boteringestraat 239712GC Groningen I Dr. Wayne M. PatienceNTU•I.ANDS Americsn Council on Education

GED Testing Service, Suite IVUr. Robert Linn One Dupont Circle, NWCollege of Education Washington, DC 2003bUniversity of IllinoisUrbana, 1 bIOdU I Dr. James A. Paulson

Portland State UniversityDr. James Lumeden P.O. lox 751Uepartment of Psychol*gy Portland, ORt 97207University of Western AustraliaNedlands, Western Australia OUI I Mr. Luigi PetrulloAUSTRALIA 2431 North Idgevood Street

Arlington. VA 22207Dr. Gary MarcoEducational Testing Service I Dr. Diane H. .amsey-Kleeerinceton, NJ UdSui R-a- Research and System Design

4947 Ridgemont DriveOr. Scott Maxwell Malibu, CA 901b5Department of PsychologyUniversity of Houston I Hr. Minrat H. L. RauchHouston, TX 77U04 P II 4

lundessinisterium der VerteidigungI Or. ameuel T. Mayo Pontfach 1328

Loyola University of Chicago 0-53 Bonn 1d6U North Michigan Avenue GERMANYChicago. IL bU611

I Dr. Mark 0. ReckaseDr. Allen %tunro liducational Psychology Departmentbehavioral Technology Laboratories University of Missouri-Columbia1d45 Elena Avenue 4 dill HallFourth Floor Columbia, MO 65211Redondo leach, CA 9hJU77

,!I

Dr. Leonard L. Rosenbaum, Chairman I Dr. Kasuo ShigemasuDepartment of Psychology University of TohokuMontgomery College Department of Educational PsychologvRockville, MU ZUB50 Kawauchi, Sendai 960

JAPANDr. Ernst Z. RothkopfBell Laboratories I Dr. Edwin Shirkeyur iountailn Avenue Department of Psychology

Murray Hill, NJ 07974 University of Central FloridaDr. Lawrence Rudner Orlando, FL 32816

403 Elm Avenue I Dr. Richard SnowTakoma Park, HD 2UU12 School of EducationStanford University

Dr. J. Ryan Stanford, CA 9305Department of EducationUniversity of South Carolina 1 Dr. Robert SternbergColumbia, SC 29208 Department oi ;.ychology

Yale UniversityIProf. Fumiko jamejima Box 11A, Yale StationDepartment of Psychology New Haven, CT 06520University of TennesseeKnoxville, TN 37916 1 Dr. Albert Stevens

Bolt, Beranek, and Newman, Inc.Dr. Frank L. Schmidt 5U Moulton StreetDdepartment of Psychology Cambridge, MA 0213dBuilding GGGeorge Washington University 1 Dr. Hariharan SwaminathanWashington, DC 2UU02 Laboratory of Psychometric and

Evaluation ResearchDr. Robert J. Seidel School of EducationInstructional Technology Group University of HaasacuusettvHumRAO Amherst, MA 01003340 North Washington StreetAlexandria, VA 22314 I Dr. Kikumi Tatsuoka

Computer Based Education aesearch1 Committee on Cognitive Research Laboratoryc/o Dr. Lonnie K. Sherrod 252 Engineering Research LaboratorySocial Science Research Council University of Illinois6U5 Third Avenue Urbana, IL 61801New York, NY 10016

I Dr. David ThissenDepartment of PsychologyUniversity of KansasLawrence, KS 6b044

II

-' ' ~ t s b n - ~.~-',w,>

~ ~ ______________

-10-

Dr. Robert TsutakawaDepartment of StatisticsUniversity of MissouriColumbia, MO 6.520L

Dr. David ValeAssessment Systems

Corporation2395 University AvenueSuite 306St. Paul, MN 55114

Dr. Howard WainerEducational Testing ServicePrinceton, NJ 08541

Or. Thomas WallstenPsychometric LaboratoryDavis Hall 013AUniversity of North CarolinaChapel Hill, NC 27514

Dr. Phyllis WeaverGraduate School of EducationHarvard Universityý0U Larsen Hall, Appian WayCambridge, MA 02138

Dr. David J. WeissNb60 Elliott HallUniversity of Minnesota75 East River RoadMinneapolis, MN 55455

Dr. Susan E. WhitelyPsychology DepartmentUniversity of KansasLawrence, KS 66044

Dr. Volfgang WildgrubeStreitkraefteamtBox 20 50 03D-53OU Bonn 2WEST GER.MA