The present study involved the ~esting of two common multiple-choice ite~ writing rules. A recent review of research revealed that muc of the advice given for writing multi?le-choice test items is based on experience and wisdom rather than on empirical research. The rules assesseG in this study include: ( 1 ) the phrasing of the stem in the form of a question versus a partial sentence; and (2) the use of the inclusive "none of the a!:)ove"option instead of a specir~c content option. Limited empirical research suggests that u:::ingthe partial sentence format and thp.inclusive "none of the above" , - ' pt i o nm a y '.eadto undesirable item and test characten.stics, while l~xtbook authors essent1ally are divided on their opinions about t va:idity of each rule. The items used in this study were from the instructor's manual for D. Myer's (1986) text entitled "Psychology." Items w~re randomly assigned to be rewritten to reflect the experimental conditions under investigation. T .... o instructors of an introductory psycho:ogy course selected 32 mUltiple-choice items for the study. The rewritten tests wer nadministered to 228 students enrolled in two sections of an introductory p ychology class. About half of the students in each section rec ived Form A and the other half received Form B, resulting in 115 Form A and 113 Form B responses. The same manipulated items were combined with 1 8 different non-manipulated items in a third section of the class to comprise Fo~ms C and D, whose administration resulted in 59 Form C and 59 Form D responses. Res lts offer no eV1dence to support the use of either type of stem and limited evidence to caution against use of the "none of the above" option, Two data tables an examples of the four it :m formats used are provided. (TJH} E D 307 298 AUTHOR TITLE PUB DATE NuTE PUB TYPE EDRS PRIcr; DESc..:RIPTORS IDENTIFIERS ABSTRACT DOCUMENT RE8UME T M 0 1 3 3).5 Crehan, Kevin; Haladyna, Tllomas M. The Validity of Two Item-WriLing Rules. 8 9 lap. Reports - Research/Technical (143) MFOl/PCOl Plus Postage. College Students; Higher Education; 1O: ult1ple cnoace Tests; P sy ch ol og y; 1 O : Te stC on st ru ct io n; Test Format; Test Validity 1O:Itemriting Rules Parallel Test Forms; Stem Analysis 1 0 : 1 0 : eproductions supplied by EDRS are the best that can be made from the origj:laldocument. 0 : 10:10:10:10:10:10:10:10:10:~10:10:10:10:10:10:10:10:10:10:1t10:10:10:10:10:1t10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10:10: 1 0 :
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
The present s tudy involved the ~est ing of two commonmult ip le -choice i te~ wri t ing ru les . A recent review of researchrevealed tha t much of the advice given for wr it ing mult i? le -choicetes t i tems is based on experience and wisdom rather than on empir icalresearch . The rules assesseG in th is study inc lude: (1) t he ph ra si ng
of the s tem in the form of a ques t ion versus a par t ial sen tence ; and(2) the use of the inc lus ive "none of the a! : )ove"opt ion ins tead of aspec ir~c content option . Limited empirica l research suggests tha tu : : : ingthe par t ial sentence format and thp. i nclus ive "none of theabove" , - 'pt ionmay ' .ead t o undes i rab le i tem and tes t charac ten .s t ics ,whi le l~xtbook authors essent1a l ly are div ided on the i r opinionsabout t va: id i ty of each ru le. The items used in th is s tudy weref rom the inst ructor 's manual for D. Myer ' s (1986) te xt e nt it le d"Psychology." Items w~re randomly ass igned to be rewri t ten to ref lectthe exper imenta l condi t ions under invest igat ion. T. . . . o ins truc tors ofan in t roductory psycho:ogy course se lected 32 mUlt iple -choice i temsfor the s tudy . The rewri t ten tes ts wer n a dm in is ter ed to 228 s tudentsenro lled in two sect ions of an in t roductory psychology class. Aboutha l f of the s tudents in each sec t ion rece ived Form A and the otherha l f rece ived Form B, resu l t ing in 115 Form A and 113 Form Bresponses. The same manipula ted i tems were combined wi th 18 di ffe ren tnon-manipula ted i tems in a th i rd sect ion of the c lass to compr iseFo~ms C and D, whose adminis tra t ion resul ted in 59 Form C and 59 FormD responses . Resul ts of fer no eV1dence to support the use of e i thertype of s tem and limi ted evidence to caut ion agains t use of the "noneof the above" opt ion, Two data tab les and examples of the four i te:mformats used are provided . (TJH}
ED 307 298
AUTHORTITLEPUB DATENuTEPUB TYPE
EDRS PRIcr ;DESc. . :RIPTORS
IDENTIFIERS
ABSTRACT
DOCUMENT RE8UME
TM 013 3).5
Crehan , Kevin ; Haladyna, Tl lomas M.The Val id i ty of Two I tem-WriLing Rules .89l ap .Rep or ts - R es ea rc h/ Te ch ni ca l (143)
MFOl/PCOl Plus Postage.Col lege Students ; Higher Education; 1O:Mult1ple cnoaceTe st s; P sy ch ol og y; 1O: Te stC on st ru ct io n; Tes t Fo rm at ;Tes t Val id it y1O:Item rit ing Rules Para l lel Test Forms; StemAnalys is
1 0 : 1 0 :eproduct ions suppl ied by EDRS are the bes t tha t can be madef rom th e o ri gj : la l do cume nt .0 :
E .p i r i ca l r e sea r ch on th i s iter vr t t i ng ru l e has b l en
th ese re f e r ences s t a t ed suppor t o r l ack o f suppor t fo r the "Don ' t
use ' non e o f the above ' a s an ~p t lon~ ru l e . Th i s vas the t en th.o s t o f t en aen t ioned ru l e , aad t h i s su r vey vas t aken as ev idence
o f the i apa r t ance o f the r u l e fo r i t em Yr i t e r~ . Hoveve r l au tho r s
ve re d iv i ded on t he i r suppor t fo r th i s ru l e , w i th 19 fo r and 1~
aga in s t . Obv ious ly soae con t rove r sy ex i s t s i n the va l i d i t y of
t he ru l e .
l i . i t e d to on ly t en s tud i e s (Boyn~ 'n , 19S0i Dudycha , ca rpen~e r ,
1973; For sy th & Spra t t , 1980; Hughes & Tr i ab l e , 1965; Muel l e r,
1915, Oos te rho f & Coa t s , 1984; Rimland , 1960; SchRe i se r &
Whi tney, 1975; Wes .an & Benne t t , 1946; Wil l i a . son & Hopk ins ,
1967). Al l o f the se s tud i e s i nvo lve J tho I t e . cha rac t e r i s t i c o f
d i f f i c u l t y , bu t on ly f ive s tud i ed I t ea d l s c r i . i na t i on and
re l i ab i l i t y , and on ly tvo va l i d i t y . In a l l i n s t an ces , t he u se o f
" none o f the above" op t ion aade i t e . s ao re d l f f i c u l t , t he aean
~f f e c t ac r03S n ine s tud i e s whe re re su l t s v e re ag9r eqab l e was
4.8\ . 11th d i sc r l a ina t i on , a vo id ing the i nc lu s i ve "none o f t he
above" op t ion . ade i t ea s s l i gh t l y ao re d i sc r l . i na t i nq , .03, whi l e
r e l i ab i l i t y v as i . p roved by a fac to r o f .04.
~ues t i on Fo r . a t V~rauB~p le t i on Fo rma t
One of th e .o s t funda . en t a l r egu l r6men t~ in HC i t em
wr i t i ng I s tha t one s t a t e s t he i t ea in a que~ t ion fo r . a t o r a
co . p l e t1on fo r . a t . On t he su r f ac e the re appea r s to be n~ r eason
- -.....a......-~.1 . Board, C. , Whitney, O. R. (1912)•. ~ effec t of se lec ted
poor Ite.- vJ: i t inC}pr,~ct ices on te at di ff ic ult y,
re l iabi l i ty , and val ic! ' , ty .ilmll;na}__of SdUCat_1o_··Ull.
t=D.JICeMot , L 125-233.
2. Baynton, H. (1~50). ~nclus lon of wnone of these- aakesapell tn~ i te .s aore di ff icul t . Idgcat lonal_6 PsychQlogicalHOllgrclCDt, ~ 431-432.
Cronbach, L. J. (1970). Reviev of on the ~heory ofachieve .ent t~s t I teaa by J .R . Bor.u th . fAYcboaotr lka , ~,509-511.
4. Our-n , Y . P. , I Goldste in , L. o . (1959). fe st d if fi cu lty ,valid i ty , and re l lahi l i ty as a function of a se l~cted.u lt ip le-ch~!ce i to. construct ion pr l rc l~lea . Idgclt lgnaland Psycholog ica l HAaBuro.ent , ~ 17~~179.
Dudycha, A. L., • carpenter, - J. 8. (1973). Bffects of i te •foraa ts on i te a d is ck ia in at io n and diff icul ty. Journa l ofAppl ied ~sycbQ1Q~ ~ 116-121.
Bbel , I . L . ( lS51) . wri t ing the te~t Ite •• In S. F.Lindquis t (Bd.) Bducatlooal "e.sure .~ (1st eel.) ( pp .185-249). Vashin9ton, DC: A•• r ican Counci l on Iducat i~n.
Feldt , L. 8. (1969)~ A t~8t of the hypothes i . tha tCronbach 's alpha or Kuder-Rlchardson coeff icient twenty isthe sa• • for test tests . 2 I J~bq . . t ' I ' a , ~ 363-373.
Forsyth , I . A. , Sprat t , I . P . (1980). Measur ing proble .solvlnq abi l i ty in .. the .. t l~ wi th .ul t lp le chol~ iteas.Yhe ~ffec t of i t•• foraa t on selec ted t te . and textcharacter ist ics. Jqgrnal o!_Jdgcat loDl l Mea'ure.ont , ' ~31-43.
,.. ,
6.
7.
8.
9. Guil ford , J . P. (1965). rDndl .~t l l s ta t i s t ic s lUfsycbology And lduCi t ion . l rd .d~r . ev York~ N.Y. , McOrav-Hil) . .
Hiladyna , T . M. ~ Dovnlny, 8. H. (1989. ) . The val id i ty ofa taxonoay of ' aul t l 'ple-cholce ita . wr i t ing rules . A»RlledMe.lgre.eDt In IdJu;:au..sm, 1..
Haiadyna, Y. H. , Downing, 8. H. (1989b) . A taxonoay of.u l t ip le-cholce I tem wri t ing rules. ApRl led MeasureMent ~ICdUCtlt 100., 1..
Hughea, H. H. p , Triable , . , a.' (1965) . f i le uae of cOIIplex·al tern .~ives In .v , l t ip le-cholce i tems. Idvcat lQoal and
13 . Nueller. D. J . ~197S).of co.plex alte~natlves
1 te . . . lCIucat lona! Iud141.
An assesa.eat of t he effect1venessl 'n aultlple choice achteV'IlMnt test
bycbo1gqical Ke .e qr e. at . . H, 135-
14. Killaan, J ., a Greene, J . (In press). !be speci f ica t ionand develo~nt uf t.st~ of achleveaent and abilities. InR. L. '"lnu (Ed.), lc2uc.tlooaL.lfeu_urellAD.t. (lrd ed.).W._sh!ngton, DC: &serlcan Council on Idncat lon.
15 . Hye r s , D . (1986)0 f ly cho lggx . "~York, •• Y •• ~~r thPubl l ihers .
23. We._h, A. G. i1S171). writing the t e a t Iteil. Irr I . L.'1' hoEmJl te (l Id .), lduc"ttoMI,tte".greaent (pp."-111).Washington, DC: Aaerlcan Coancl l on !dueation.
24. We a a a n . A. G., , Bennett, O. K. (1946). !be use of ' n o n f t
16. Mitto, A. J . (1984) . ~ook r ev i ev of Rold an~ H. l a~yna t 8 A, technolog1 for tes t - l te . vr l t lng . ~curnal of Iduc.tlonalMeasure.oot, ~ 201-204.
17. OOsterhof, A. C • .a Coats , P . K . (198.) . eoaparlsOft ofdiff icul t ies and rel lab111ty of quant i ta t ive vord proble . .in cOlIPle t lon and aultlple-cholee Ite. foraats. Appl iedbyeHlogleal MMlur_~-. L 8'7-2 '4.
18. Ri . land, B. (1960). "he'."ffects of varyln9 tiae ll.its andof ustng -right anaver not 9iven- in expert.ental for •• ofthe U. S. hvy ' l r i th .e t lc ! 'ellt . lduCtltioDll__andP8ych~c~1 NeI.uro.eDt~ ~ 5~3-539.
19. aold, G. }I~, , Haladyna, if. M. (1982). I tecbnol_o_a_y____f_Qz_t e e t - i t o . yr lt ing. Mev !or t , . 7: Acade. le Pres ••
20 • Schl ie iser, 'c. a., 'Whttney, ". It. ('1'154)• effect of tvoselected I t .a -wrl t lng practices on test difficulty,~l .cr l . lna t lon, and reliability. iOuroal of 'XQl r laen tAlUK.tiQD, ll, 30-34.
~1. Schaels.r, Cw Bv, , Whltney, D. a . t1975b) . Ybe effect ?fInCOMplete ate_ and -none of tbe above- fo1ls on test ah.J·it.a CMractez: 'atla. Paper presented at the elnnu,l lleetlnqof the .. tional Council on Measure.ent i n aducl :t tl on ,'.uhlngton, D.C.
22. 'Schrock, t'. J . , and Muel ler , D. J. (1982) . Bffects ofvtolatlnc) three aultlpl.-cholce t tea constz:uctioJ~principles. f be Journal ' of l duca \l oDa !_loaca rCb , ~ 314-311.
~f t b==c · eS .h Gp t l v " 1" t e . t conDt~vc t lo " . ~wir.l G'Iduc.tlon.l r eycbo logy . ~ 541-54'w
'Wl111a_oft, .... '·L., , RopkinlS', K. fl'. (1967). the UIIe of· noQ.-of- these~ versus hoaogeneous . l ternat ives on aul t ip le-choice tes ts ' : . • xperlaental r et la bl ll t, . a rJ v. ll dl tycOllparlsona. Jogr,.! of I dgca t l oo . l tleoaU(t:HaL .L. 53'-58.
food, R. (1917). Mult ip le choice: 1 s tate of the ar trepor t . IyllultloD in Iduc.tloDl I nt er na ti Qn al P rog re sa ,L 191-210.
fte toI1.. ,... ,••• ~•...,1. d \;1Ie her At.. loneta ...... 1.. ·_~tIIIe eel"ct_ l_t~ab.
(at) In the i r clanlc nln ... -year study, F rl ed . . n and Ro.eun foun4t l t a 1 : co.pet 1 t 1ve; h a r d - d r 1v 1ftc), ' l a p a t i - e n t , a n e t e a e l1yanqere4-1ndlvidaalB are especia l ly suscept ib le tor
.a. s ta . .ch ulcers •b . c a n C a r .
* c. bent a t tacka .d. accidents .e. non. of these
(OM) In thei r c l a s s i c nine-year .~ud7, r r iedaan and Roaeaanfound that coapet l t lve , hard-dr lvln9, i .pat lent , andeasi ly angered Indlvldu.l~ are especial ly saac@ptlb l~-to which of th.'-followll\CJ?
*
a . s toMach ulcersb . cancerc . s trokesd. ac:c tdentse . Done of t h e s e
(ca) In their c lasa lc nlne-year s tudy, Prledaan 'aDd Roseasn foundthat coapet l t lve, hard-drLvln9, i . ,a t l . . t , and eas i lyangered individual . are especial ly s u s c e p t i b l e to :
•• s toaach ulcers.b. cancer.
• c. heart a t tacks.d O . a cc ide nt ••e•• t ro ke ••
(QB) In their cla •• lc nine-yeer study, r r iedaan and 108e . . n foaDd'tha t co.petl t ive , "ha~d-dr lvin9, tapat lent , and eas i lya ng ere d I nd ivi du al s a r e e s p e c i a l l y s u s c e p t i b l e t o which o fthe fol lowing?
a . s toaacb ulcersb. cancer
* c. hear t a t tacks ·d. ' ace Id .n tse. s t roke .