Confidence-Based Assessment of Two-Alternative Format Tests AHMED A. BELAL (1) & DIALA F.AMMAR (2)

Confidence-Based Assessment of

Two-Alternative Format Tests

AHMED A. BELAL (1) & DIALA F.AMMAR (2)

(1)Computer Engineering and Informatics- Beirut ArabUniversity, Lebanon

(2)Psychology department, Lebanese American University,Lebanon

[email protected]

•Computer Aided Assessment

•Multiple Choice Exams :

No Partial Credit

Penalty for Wrong Answers

•Designing good distracters

•Multiple Select Exams

Penalty for Wrong Answers

•Multiple T/F Exams

Question may have no correct answersNo partial credit

• Individual T/F questions Weak resistance to guessing

Correct =+ XNo Answer = 0Wrong Answer= -Y

Poor discrimination

•Confidence-based Assessment

Correct 0 1 2 3 Grade 0 10 20 30

Correct 0 1 2 3 Grade -15 0 15 30 Scaled 0 15 30 45

Correct 0 1 2 3 Grade -30 -10 10 30 Scaled 0 20 40 60

+100

+10-5

+10-10

9 Different Grades

7 Different Grades

Correct 0 1 2 3 0 1 2 1 0 0 Wrong 3 2 1 0 2 1 0 0 1 0 No answer 0 0 0 0 1 1 1 2 2 3

Grade -15 0 15 30 -10 5 20 10 -5 0 Scaled 0 15 30 45 5 20 35 25 10 15

Grade -30 -10 10 30 -20 0 20 10 -10 0 Scaled 0 20 40 60 10 30 50 40 20 30

Correct +10 Wrong -5 No answer 0

Correct +10 Wrong -10No answer

0

Hard to correctly estimate one’s confidence level

Humans are more oriented to judge their uncertainty in things relative to each other

Leclereq [1,2]

Confidence Level Correct Answer Wrong Answer

0 0 0

1 +3 -1

2 +4 -2

3 +5 -5

• Student ranks questions relative to each other.

• Value of correct answer decreases as level of confidence decreases.

• Wrong answers change the confidence level.

• Use a step size and a reduction function to assign values for a correct answer.

Answer Vector

000 001 010 011 100 101 110 111

R=5 0 0 5 10 10 15 20 30

R=10 0 -10/0 0 0 10 10 20 30

R=15 0 -20/0 -5/0 -10/0 10 5 20 30

S=1

Confidence Level C1 C2 C3 C4 C5

Reduction Value 2 10 8 6 4 2

Reduction Value 5 10 5 0 -5 -10

10 Questions : answer vector 1101001100

Step size = 1R =2 Score = 10+10+0+8+0+0+4+4+0+0 = 36

R = 5 Score = 10+10+5 -5-5 = 15

Step size = 2R =2 Score = 10+10+10+8+8 = 46

R = 5 score = 10+10+10+5+5 = 40

Absolute (10-2p) (10-p) (10-5p) R=1;s=2 R=2;S=2 Answer Vector

16 24 32 10 37 34 100101100038 44 47 35 50 50 110111000025 40 50 15 56 52 101011101024 30 40 10 47 44 011010110062 64 72 40 76 72 111100111180 90 90 90 90 90 111111111062 66 73 45 80 80 101111111037 42 46 25 50 50 101111000018 32 36 20 39 38 101101000051 64 72 40 80 80 011111111030 50 60 30 66 62 111001101186 78 84 60 90 90 111011111146 42 56 20 65 60 110010111130 36 38 30 40 40 110110000058 64 67 55 70 70 111101110049 62 66 50 70 70 111011110070 58 64 40 70 70 101111110038 42 46 30 50 50 101111000016 22 36 5 45 40 010011101041 40 45 25 50 50 011111000016 24 32 5 37 34 01011010000 24 32 0 27 24 00111000001 22 26 10 29 28 0110100000

29 30 35 20 39 38 10110010004 34 37 30 39 38 1110001000

28 30 40 5 46 42 010111010054 62 66 50 68 66 111110011040 52 61 30 66 62 1110011101

Table 1

Strategy Average Standard Deviation

Absolute 37.46428571 22.57084672

10-2P 45.28571429 18.21854396

10-P 51.75 17.3901313

10-5P 29.46429 20.19989

R=1 S=2 56.14286 17.98912

R=2 S=2 54.42857 18.44597

Table 2