7/28/2019 8. Setting and Maintaining Examination Standards
1/20
SETTING & MAINTAINING
EXAM STANDARDS
Raja C. Bandaranayake
7/28/2019 8. Setting and Maintaining Examination Standards
2/20
DEFINITIONS
Standard setting is a process of determining howmuch is good enough.
The standard or criterion level of performance is apoint on the scale of measurement at whichseparation of competence and incompetenceoccurs.
Cut-score, cut-off score or passing score representsthis standard on a given test for making decisionspertaining to the purpose for which the test wasconducted, e.g., to certify competence.
7/28/2019 8. Setting and Maintaining Examination Standards
3/20
ERROR IN MEASUREMENT
True score is a conceptual measure indicating trueextent of competence in a given subject, e.g.,Anatomy.
Observed score is the score assigned as a result oftaking a test, say in Anatomy.
The difference between true and observed scores isindicative of the amount of error in the
measurement.
The reliability of a test and the associated standarderror of measurement are estimates of theamount of error in the measurement.
7/28/2019 8. Setting and Maintaining Examination Standards
4/20
DECISION ERRORS
False positive:
passing an incompetent examinee
False negative:
failing a competent examinee
7/28/2019 8. Setting and Maintaining Examination Standards
5/20
NORM- & CRITERION-REFERENCED STANDARDS
NORM-REFERENCED
Relative
Based on peer-performance
Varies with each group
Cut-off point notrelated to competence
CRITERION-REFERENCED
Absolute
Not related to peerperformance
Standard set prior toexam
Referenced to adefinedlevel ofperformance
7/28/2019 8. Setting and Maintaining Examination Standards
6/20
METHODS OF STANDARD SETTING
1. Test-centred methodsStandards derived from hypotheticaldecisions based on test content before thetest is answered.
2. Examinee-centred methodsStandards derived from reviewing examineesperformance before deciding cut-off score.
3. Compromise methodsProvide flexibility for adjusting thestandard based on the examineesperformance on the test.
7/28/2019 8. Setting and Maintaining Examination Standards
7/20
NEDELSKY (1954) METHOD: Example
Consider N judges and n MCQ items of 1 in 5 type
Judge A identifies 2 options in item 1 as those which aminimally competent examinee should eliminate as incorrect.
MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3
Similarly, in item 2 he identifies 3 options, giving an MPLA2 =1/(5-3) = 1/2He repeats this process for each item.
The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 +
.MPLAn
Similarly, Judge Bs MPL [MPLB] is determined
The MPL for the exam (= cut-off score) is: (MPLA +
MPLB + MPLC+.MPLN) / N
7/28/2019 8. Setting and Maintaining Examination Standards
8/20
ANGOFF (1971) METHODExample
N judges consider 100 minimally competent examineestaking an MCQ exam of n items.
Judge A estimates that, of these examinees, 50 shouldanswer item 1 correctly, 20 item 2 correctly, 70 item 3
correctly, and so on to item n.
The MPL for Judge A [MPLA] = (0.5 + 0.2 + 0.7 + . xn) / n X100 = (say) A%.
Similarly, for Judges B, C, D, E, ..N, the MPLs would be B%,C%, D%, E% N%, respectively.
The MPL (cut-off score) for the exam is: (A% + B% + C% +D% + E% +....N%) / N
7/28/2019 8. Setting and Maintaining Examination Standards
9/20
EBEL (1972) METHODExample
Assume that Judge A assigns items in a 200-item MCQ test to the cells of arelevance-by-difficulty matrix, as follows.
He then estimates the percentage of items in each cell of the matrix that aminimally competent examinee should be able to answer correctly (as indicatedwithin the cell).
Each cell also includes the products of these two values.
EASY MEDIUM HARDESSENTIAL 15 x100% = 1500 25 x80% =2000 10 x60% = 600
IMPORTANT 20 x 80% = 1600 40 x60% =2400 20 x50% =1000
ACCEPTABLE 10 x 50% = 500 25 x40% = 1000 5 x10% = 50
QUESTIONABLE 10 x30% = 300 15 x20% = 300 5 x0% = 0
7/28/2019 8. Setting and Maintaining Examination Standards
10/20
EBEL (1972) METHOD - contd.Example
The MPL for Judge A [MPLA] is then:
(1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000
+ 50 + 0) / 200 = 56.25 %
Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD]..N [MPLN] are determined.
The MPL for the exam (cut-off score) is:
(MPLA+MPLB+ MPLc+ MPLD+ ..MPLN) / N
7/28/2019 8. Setting and Maintaining Examination Standards
11/20
PROPOSED EBEL MODIFICATION
EASY MEDIUM HARD
ESSENT. 6x100% = 600 12 x80% = 960 7 x50% = 350
IMPORT. 12 x80% = 960 24 x60% = 1440 19 x40% = 760
ACCEPT. 5 x60% = 300 12 x50% = 600 3 x10% = 30
MPL: =600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30
=6000/100
= 60
7/28/2019 8. Setting and Maintaining Examination Standards
12/20
Failure
Rate%
Cut-offscore(%)
10
15
20
35 40 45 50
fmin
fmax
cmin cmax
A
B
HOFSTEE METHOD
7/28/2019 8. Setting and Maintaining Examination Standards
13/20
HOFSTEE METHODExample
A plot of cut-off scores for a given examagainst resulting failure rates is given
cmin= 40%
cmax= 45%fmin = 10%fmax = 20%
A = point representing cmin,fmax
B = point representing cmax,fminLine AB intersects the curve at a cut-offpoint of 42.5%
Thus, operational cut-off score = 42.5%
7/28/2019 8. Setting and Maintaining Examination Standards
14/20
7/28/2019 8. Setting and Maintaining Examination Standards
15/20
CUT-OFF SCORE FOR 1 IN 5 MCQ[FRACS PART 1]
Probability of guessing (=1 in 5) = 20%
Total ignorance score = 20%
Maximum possible score =100%
Effective range of scores = 20% to100%
Mid-point of this range = 60%
Additional factor (as PG exam) = 5%
Nominal cut-off score (60%+5%) = 65%
7/28/2019 8. Setting and Maintaining Examination Standards
16/20
CUT-OFF SCORES:MARKER QUESTIONS
1. Comparison of exam scores
Mean score in this exam: 56.7%
Average exam mean score over last 4 years: 59.4%
Thus mean score in this exam is: 2.7% lower
Assuming this candidate group is of samestandard as in last 4 yrs, this exam is: 2.7% harder
7/28/2019 8. Setting and Maintaining Examination Standards
17/20
CUT-OFF SCORES:MARKER QUESTIONS - contd.2. Comparison of marker scores
Mean score in this exam on previously usedquestions (N=162): 62.5%
Mean score on same questions when theywere each last used: 60.5%
Thus, when compared with previous
candidates, this group of candidates, onthese items, scored (62.5-60.5)% = 2.0% higher
Thus this group of candidates is: 2.0% betterthan previous groups
7/28/2019 8. Setting and Maintaining Examination Standards
18/20
CUT-OFF SCORES:MARKER QUESTIONS contd.3. Estimating examination difficulty
Thus it is expected that their mean scorein this exam would be: 2.0% higher
But their mean score in this exam is: 2.7% lower
Thus this exam is really: 4.7% harder
7/28/2019 8. Setting and Maintaining Examination Standards
19/20
CUT-OFF SCORES:MARKER QUESTIONS contd.4. Determining cut-off score
The cut-off level for an average exam is: 65.0%
Thus the cut-off level for this exam
should be (65 4.7)% = 60.3%
Cut-off score = 60.3%
7/28/2019 8. Setting and Maintaining Examination Standards
20/20
Failure
Rate%
Cut-offscore(%)
10
15
20
55 60 65 70
HOFSTEE CURVE