DOCUMENT RESUME ED 422 359 TM 028 911 AUTHOR Yan, Duanli; Lewis, Charles; Stocking, Martha TITLE Adaptive Testing without IRT. PUB DATE 1998-04-00 NOTE 23p.; Paper presented at the Annual Meeting of the National Council on Measurement in Education (San Diego, CA, April 12-16, 1998). PUB TYPE Reports Evaluative (142) -- Speeches/Meeting Papers (150) EDRS PRICE MF01/PC01 Plus Postage. DESCRIPTORS *Adaptive Testing; Algorithms; *Computer Assisted Testing; *Item Response Theory; Models; *Nonparametric Statistics; *Regression (Statistics); Simulation; *Test Construction ABSTRACT It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all new and currently considered computer-based tests. In addition to developing new models, researchers will need to give some attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This paper introduces an adaptive testing algorithm that balances maximum differentiation among test takers with stable estimation at each stage of testing, and compares this algorithm with a traditional one using IRT and maximum information. The adaptive testing algorithm introduced is based on the classification and regression tree approach described in L. Breiman, J. Friedman, R. Olshen, and C. Stone (1984) and J. Chambers and T. Hastie (1992). Simulation results from the regression tree approach were compared with simulation results from three parameter logistic model IRT. Simulation results show that the nonparametric tree-based approach to adaptive testing may be superior to conventional IRT-based adaptive testing in cases where the IRT assumptions are not satisfied. It clearly outperformed one-dimensional IRT when the pool was strongly two-dimensional. A technical appendix describes the algorithm. (Contains three figures and six references.) (SLD) ******************************************************************************** * Reproductions supplied by EDRS are the best that can be made * * from the original document. * ********************************************************************************
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
DOCUMENT RESUME
ED 422 359 TM 028 911
AUTHOR Yan, Duanli; Lewis, Charles; Stocking, MarthaTITLE Adaptive Testing without IRT.PUB DATE 1998-04-00NOTE 23p.; Paper presented at the Annual Meeting of the National
Council on Measurement in Education (San Diego, CA, April12-16, 1998).
PUB TYPE Reports Evaluative (142) -- Speeches/Meeting Papers (150)EDRS PRICE MF01/PC01 Plus Postage.DESCRIPTORS *Adaptive Testing; Algorithms; *Computer Assisted Testing;
*Item Response Theory; Models; *Nonparametric Statistics;*Regression (Statistics); Simulation; *Test Construction
ABSTRACTIt is unrealistic to suppose that standard item response
theory (IRT) models will be appropriate for all new and currently consideredcomputer-based tests. In addition to developing new models, researchers willneed to give some attention to the possibility of constructing and analyzingnew tests without the aid of strong models. Computerized adaptive testingcurrently relies heavily on IRT. Alternative, empirically based,nonparametric adaptive testing algorithms exist, but their properties arelittle known. This paper introduces an adaptive testing algorithm thatbalances maximum differentiation among test takers with stable estimation ateach stage of testing, and compares this algorithm with a traditional oneusing IRT and maximum information. The adaptive testing algorithm introducedis based on the classification and regression tree approach described in L.Breiman, J. Friedman, R. Olshen, and C. Stone (1984) and J. Chambers and T.Hastie (1992). Simulation results from the regression tree approach werecompared with simulation results from three parameter logistic model IRT.Simulation results show that the nonparametric tree-based approach toadaptive testing may be superior to conventional IRT-based adaptive testingin cases where the IRT assumptions are not satisfied. It clearly outperformedone-dimensional IRT when the pool was strongly two-dimensional. A technicalappendix describes the algorithm. (Contains three figures and sixreferences.) (SLD)
********************************************************************************* Reproductions supplied by EDRS are the best that can be made *
Muraki, E., & Bock, D. (1993). PARSCALE: IRT based test scoring and item analysis
for graded open-ended exercises and perfonnance tasks. [Computer program].
Chicago, IL: Scientific Software, Inc.
Schnipke, D., & Green, B. (1995). A comparison of item selection routines in linear and
adaptive tests. Journal of Educational Measurement, 32, 227-242.
Wainer, H., Lewis, C., Kaplan, B., & Braswell, J. (1991). Building algebra testlets: A
comparison of hierarchical and linear structures. Journal of Educational
Measurement, 28, 311-324.
Wainer, H., Kaplan, B., & Lewis, C. (1992). A comparison of the performance of
simulated hierarchical and linear testlets. Journal of Educational Measurement,
29, 243-251.
13
Exh
ibit
A. S
ampl
e O
utpu
t fro
m P
rogr
am to
Con
stru
ct R
egre
ssio
n T
rees
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
1 4
036
037
038
ADAPTIVE TESTING WITHOUT IRT - Calibration Sample Tree Table
NodeInfo:istage=
Stage - 0
0 inode=
0 Nsubj= 250 yval= 34.7360 Dev=28608.5760
Stage Node
Item
Parents
Deviance
Mean
00
31
-9,-9,-9,-9,-9
250
28608.5760
34.7360
999.0000
999.0000
Total Deviance at Stage
0:
28608.5760
Proportion of Variance Accounted For:
0.0000
Standard Deviation at root:
10.6974
Standard Deviation for Each Node at This Stage:
Node =
0SD =
10.6974
Stage =
1
NodeInfo:istage=
1 inode=
1 Nsubj=
71 yval=
25.3521 Dev=3716.1972
NodeInfo:istage=
1 inode=
2 Nsubj=
179 yval=
38.4581 Dev=16160.4358
In Combine:inode,jnode,t,e =
12
-11.6993
-1.5452
Inode,Jnode,tmin,emin =
12
-11.6993
-1.5452
Stage Node
Item
Parents
Deviance
Mean
11
28
0,-9,-9,-9,-9
71
3716.1972
25.3521
-11.6993
1.5452
12
27
0,-9,-9,-91-9
179
16160.4358
38.4581
999.0000
999.0000
15
1 G
Exh
ibit
A. S
ampl
e O
utpu
t fro
m P
rogr
am to
Con
stru
ct R
egre
ssio
n T
rees
(C
ontin
ued)
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
Total Deviance at Stage
1:
19876.6329
Proportion of Variance Accounted For:
0.3052
Standard Deviation at root:
10.6974
Standard Deviation for Each Node at This Stage:
Node =
1SD =
Node =
2SD =
7.2347
9.5017
Stage = 2
NodeInfo:istage=
2 inode=
NodeInfo:istage=
2 inode=
NodeInfo:istage=
2 inode=
NodeInfo:istage=
2 inode=
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
Inode,Jnode,tmin,emin =
3 Nsubj=
44 yval= 22.8409 Dev=1069.8864
4 Nsubj=
27 yval= 29.4444 Dev=1916.6667
5 Nsubj=
70 yval= 31.7857 Dev=3251.7857
6 Nsubj= 109 yval= 42.7431 Dev=7790.8073
34
-3.6375
-0.9405
35
-8.0368
-1.4907
36
-17.9650
-2.8575
45
-1.2691
-0.3012
46
-7.2206
-1.5573
56
-9.4833
-1.4190
45
-1.2691
-0.3012
Estimations during combination:
Stage Node
Item
Parents
23
24
25
1,-9,-9,-9,-9
1, 2,-9,-9,-9
2,-9,-9,-9,-9
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
In Combine:inode,jnode,t,e =
Inode,Jnode,tmin,emin =
Deviance
Mean
44
1069.8864
22.8409
97
5275.2577
31.1340
109
7790.8073
42.7431
34
-7.7947
-1.3126
35
-17.9650
-2.8575
45
-10.4748
-1.4563
34
-7.7947
-1.3126
999.0000
999.0000
-1.2691
0.3012
999.0000
999.0000
17
15
Exh
ibit
A. S
ampl
e O
utpu
t fro
m P
rogr
am to
Con
stru
ct R
egre
ssio
n T
rees
(C
ontin
ued)
078
Stage Node
Item
Parents
NDeviance
Mean
079
080
23
21,-9,-9,-9,-9
44
1069.8864
22.8409
-7.7947
1.3126
081
24
16
1, 2,-9,-9,-9
97
5275.2577
31.1340
999.0000
999.0000
082
25
33
2,-9,-9,-9,-9
109
7790.8073
42.7431
999.0000
999.0000
083
084
085
Total Deviance at Stage
2:
14135.9514
086
087
Proportion of Variance Accounted For:
0.5059
088
089
Standard Deviation at root:
10.6974
090
091
Standard Deviation for Each Node at This Stage:
092
093
Node =
3SD =
4.9311
094
Node =
4SD =
7.3746
095
Node =
5SD =
8.4543
19
1 2 3 4 5 6 7 8 9
Fig
ure
1.R
egre
ssio
n T
ree
Str
uctu
re
1416
1820
2224
-26
2830
3234
3638
4042
44
4648
5052
5456
a 0
Pre
dict
ed S
core
21
Figure 2
Comparison of Tree-based and IRT CATs in One-dimensional Application Sample(referring to true scores)
I = IRT, T = Tree
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Test Length2 2
Figure 3
Comparison of Tree-based and IRT CATs in Two-dimensional Application Sample(referring to true scores)
I = IRT, T = Tree
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Test Length2 3
U.S. DepartmentofEducationOffice of Educational Research and Improvement (OERI)
National Library of Education (NLE)Educational Resources Information Center (ERIC)
REPRODUCTION RELEASE(specific:pocumen)
I. DOCUMENT IDENTIFICATION:
ERITM028911
Title:Adaptive Testing without IRT
Author(s): Yan, Duanli, Lewis, Charles and Stocking, Martha
Corporate Source: Publication Date:
II. REPRODUCTION RELEASE:In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy,and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, ifreproduction release is granted, one of the following notices is affixed to the document.
If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottomof the page.
The sample sticker shown below will beaffixed to all Level 1 documents
1
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL HAS
BEEN GRANTED BY
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)
Level 1
Check here for Level 1 release, permitting reproductionand dissemination In microfiche or other ERIC archival
media (e.g., electronic) and paper copy.
Signhere,--)please
The sample sticker shown below will beaffixed to all Level 2A documents
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL IN
MICROFICHE, AND IN ELECTRONIC MEDIAFOR ERIC COLLECTION SUBSCRIBERS ONLY,
HAS BEEN GRANTED BY
2A
\etc\C
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)
Level 2A
Check here for Level 2A release, permitting reproductionand dissemination In microfiche and in electronic media
for ERIC archival collection subscribers only
The sample sticker shown'below will beaffixed to all Level 2B documents
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL IN
MICROFICHE ONLY HAS BEEN GRANTED BY
2B
cpTO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
Level 28
Check here for Level 2B release, permittingreproduction and dissemination In microfiche only
Documents will be processed as indicated provided reproduction quality permits.If permission to reproduce Is granted, but no box is checked, documents will be processed at Level 1.
I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this documentas indicated above. Reproductidn from the ERIC microfiche or electronic media by persons other than ERIC employees and its systemcontractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and other service agenciesto satisfy information needs of educators in response to discrete inquiries.
Signature:
7lt491./t()
7j,n1Ad:Areak-c-- eiJ)fr C.G1-041, IV- o
Printed Name/Position/Title:
13er-ne:173 -_59zp4' c:Aex>4. oft-5 c4?
Dale: V7/99iFsmfT. o 12..c.(over)
III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE):
If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, pleaseprovide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publiclyavailable, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly morestringent for documents that cannot be made available through EDRS.)
Publisher/Distributor:
Address:
Price:
IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER:
If the right to grant this reproduction i.elease is held by someone other than the addressee, please provide the appropriate name andaddress:
Name:
Address:
V. WHERE TO SEND THIS FORM:
Send this form to the following ERIC Clearinghouse:THE UNIVERSITY OF MARYLAND
ERIC CLEAMNGHOUSE ON ASSESSMENT AND EVALUATION ,1129 SHRIVER LAB, CAMPUS DRIVE
COLLEGE PARK, MD 20742-5701Attn: Acquisitions
However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document beingcontributed) to:
ERIC Processing and Reference Facility1100 West Street, rd Floor