Multiple Choice Questions (MCQs) Ensuring the question fits the purpose Julie Vuolo, Learning and Teaching Institute [email protected] 1
Apr 01, 2015
1
Multiple Choice Questions(MCQs)
Ensuring the question fits the purpose
Julie Vuolo, Learning and Teaching Institute [email protected]
Principles for good practice assessment
Type of questions Advanced MCQs Further resources
2
This workshop aims to cover
3
Considerations when creating MCQs Pedagogical
UH Assessment for Learning Principles
Engages students with the assessment criteria
Supports personalised learning Ensures feedback leads to
improvement Focuses on student
development Stimulates dialogue Considers student and staff
effort
(UH Knowledge Exchange, 2012)
Good practice in Undergraduate Education
Encourages contact between students and lecturers
Develops reciprocity and cooperation among students
Encourages active learning Gives prompt feedback Emphasises time on task Communicates high
expectations Respects diverse talents
(Chickering and Gamson,1987)
Considerations when creating MCQsEffective vs. efficient assessment
Waste of time
Educational Effectiveness
Sta
ff/R
esou
rce
Effi
cien
cy (
or 1
/t)
Bloom’s Taxonomy, revised by Anderson Nouns to verbs and the top two reversed
Simple One correct answer True/False
Multiple Response Questions Multiple True/False Matching items Extended matching Assertion Reason
6
Types of MCQs
7
Some examples…
The main stadium for the London Olympics in 2012 is located in:
1. Great Britain2. British Isles3. England4. Scotland
8
1 2 3 4
0% 0%
100%
0%
Quintin McKellar who was previously VC at the Vet
School joined the University in?
1. December 20102. January 20113. February 20114. January 2020
1 2 3 4
17%
0%
17%
67%
MCQs
1. Are reliable2. Are easy to mark3. Are easy to
produce4. Can only be used
to test the cognitive domain
1 2 3 4
20%
0%0%
80%
11
Re-write the questions
Two possible alternatives so fair chance of getting the right answer…
Some tips: Use statements which are unequivocally true or false. Avoid lifting statements directly from assigned reading,
lecture notes or other course materials so that recall alone will not permit a correct answer.
As in multiple choice questions generally, use negatives sparingly.
Generally avoid the use of words which would signal the correct response to the test-wise student. Absolutes such as “none”, “never”, “always”, “all”, “impossible” tend to be false, while qualifiers such as “usually”, “generally”, “sometimes” “often” are likely to be true.
More usually used as a basis for more complex Assertion/Reason questions
True/False
13
Good MCQ
Well formed STEM
Clearly correct answer
Plausible distracto
rs
Components of an MCQ
Options
+/- feedback
14
Considerations Use clear, straight forward language Aim to write as a complete sentence Avoid use of unnecessary content Avoid the use of negatives. If you must use
them embolden them to make them stand out Avoid giving clues in the question e.g. an/a Avoid veering away from the
content/cognitive level that you aimed to address
Stem
15
Correct answer and Distractors Avoid repeating the same content in the options –
move it to the stem if possible Word distractors in a similar way to the correct
answer and keep to similar length Distractors should be plausible Avoid ‘all of the above’/ ‘none of the above Aim to use common misconceptions/student
errors as distractors There should be a sufficient number of distractors A correct statement but not the answer to the question is a good distractor
Options
Any questions?
Multiple Response Questions Multiple True/False Matching items Extended matching Assertion Reason
Marking gets more complex Some forms can be used in EVS but
not all E.g. Confidence marking/Negative marking
Higher level of cognitive testing
Students rate their certainty on a scale of 1-3
CBM motivates reflection about the reliability and justification for each answer, since the student gains by identifying reasons either for reservation or confidence
Maximum formative benefit requires immediate feedback, prompting reflection about uncertainties or confident errors
Certainty-Based Marking
Why use CBM? • To encourage them to really understand the
issues, not just to react immediately to a question
• To encourage students to think laterally: other pieces of knowledge may help to validate or question their answer
• To challenge the students - if they won't risk losing marks if wrong, then they don't really know the answer.
• It will encourage the students to have confidence in their knowledgeIt is more fair - a thoughtful and confident correct answer deserves more marks than a lucky hunch
• Efficient study requires that students constantly question how their ideas arise, and how reliable they are
• It encourages a reflective approach to learning
Gardner-Medwin, 2006
20
A 28 year old woman with one child has taken anti-thyroid drugs for 6 months for thyrotoxicosis. She has a friend who has been successfully treated with radioiodine. She finds she frequently forgets to take her drugs and wants to stop them to have radio-iodine treatment.
A. She should be told that because of her age radio-iodine is best avoided.B. The problems associated with radio-iodine should be discussed with her.C. Surgery as a possible alternative should be discussed with her.D. She should be advised that some form of further treatment is required.E. You should find out more about her friend’s treatment.
Options are given as different combinations of the responses,The correct answer is: - true, B, C and D and false, A and E.
Harden, R.M. & Dunn, W.G. (1981) Assessment a Work Manual, Dundee Centre of Medical Education in Brown et al 1997 from Loughborough University
Multiple true false
21
Complex MCQ (Matching)Simple MCQ
Acute intermittent porphyria is a result of a defect in the biosynthetic pathway for:
A. collagenB. corticosteroidC. fatty acidD. glucose E. haem*F. thyroxine (T4)
Matching MCQ
An otherwise healthy 33-year-old man has mild weakness and occasional episodes of steady, severe abdominal pain, but no diarrhoea. One aunt and a cousin have had similar episodes. During an episode his abdomen is distended, and bowel sounds are decreased. Neurological examination shows a mild weakness in the upper arms. These finding suggest a defect in the biosynthetic pathway for:
A. collagenB. corticosteroidC. fatty acidD. glucose E. haem*F. thyroxine (T4)
Versus
22
Extended matching
More complex with multiple questions and answers on a common theme
Basis A theme Lead in statement Multiple scenarios/questions Options
Example:
23
Extended MatchingMultiple questions and multiple answers
For each of the following patients with a myocardial infarction select the most likely complication from the list below. Each option may be used once, more than once or not at all. A) A 72 year old man is admitted to the Coronary Care Unit
following a myocardial infarction. Seven days later he complains of severe trouble breathing and soon after, he collapses. On examination he is pale. His right leg appears swollen compared to the left. The apex beat is at the 5th intercostal space, mid-clavicular line. There are no murmurs audible but there are occasional bi-basal crackles in his chest. His pulse is 128/min, blood pressure 98/55 mmHg, temperature 37.7 C and jugular venous pressure is elevated.
B) A 62 year old woman is admitted to the Coronary Care Unit following a myocardial infarction. Three days later she has become very unwell. On examination she is breathing noisily and her ankles are swollen. The apex beat is at the 6th intercostal space, anterior axillary line. There is a pansystolic murmur radiating to the axilla. Examination of the lungs reveals crackles at both bases and midzones. Her pulse is 126/min, blood pressure 105/65 mmHg, temperature 37.3 C, and respiratory rate 24/min. Her jugular venous pressure is raised etc..
24
Asystole Atrial fibrillation Cardiac tamponnade Complete heart block Deep vein thrombosis Papilliary muscle rupture Pericarditis Post myocardial infarction syndrome Pulmonary embolism Pulmonary oedema Unstable angina Ventricular fibrillation Ventricular septal rupture Ventricular tachycardia
(Medical Schools Council, 2012)
Assertion-reason questions are used to explore cause and effect and identify relationships between statements and hence test application/analysis.
When writing assertion-reason questions, keep in mind the following points: The reason should be a free standing
sentence so that it can be considered separately from the assertion.
Avoid using minor reasons. These can result in an ambiguous question.
Repeat options A-E in full for each question.
25
Assertion Reason (ARQs)
26
Example of assertion reasonA True/True
Reason is correct explanation
B True/TrueReason Is NOT a correct
explanationC True/FalseD False/TrueE False/False
Assertion High speed is a factor in car accidents
Reason Most modern cars can reach speeds in excess of 100mph
because…
On each table is an example of a type of a more complex Multiple Response Question
In your group devise a question area using principles from last task
At the end pass your question to the next table for feedback using the checklist
27
Activity
Checklist for questions
Does the question: Test the level that was
intended Have a clearly worded
stem Have only one clearly
correct answer Have a distractor that is
similar in wording and length to the correct one
Have plausible distractors
Does the question avoid: Repeating text in the
question and the answer Double negatives Use of ‘all of the above’/
‘none of the above’ Giving clues in the question
(e.g. use of an/a etc) Consider cultural bias /
incusivity
Application/Analysis and Evaluation
29
Can also use MCQs to test higher levels
"The story is told of the famous German Organic Chemist Auguste Kékulé who was struggling with the problem of how the six carbon atoms of benzene were linked together. He was getting nowhere with the problem, and one day fell asleep in front of the fireplace while he was pondering on it. He dreamt of molecules twisting and turning around like snakes. Suddenly, one of the snakes swallowed its own tail and rolled around like a hoop. Kékulé woke up with a start, and realized that his problem could be solved if the six carbon atoms of benzene were attached to each other to form a ring. Further work showed that this was entirely correct."
The above passage illustrates a particular phase of the creative process. Which one is it?1. preparation 2. incubation 3. orientation 4. illumination 5. verification 30
MCQ for testing Analysis
Question – define the creative process“The creative process is believed to take place in five stages, in the following order: ORIENTATION, when the problem must be identified and defined, PREPARATION, when all the possible information about the problem is collected, INCUBATION, when there is a period where no solution seems in sight and the person is often busy with other tasks, ILLUMINATION, when the person experiences a general idea of how to arrive at a solution to the problem, and finally VERIFICATION, when the person determines whether the solution is the right one for the problem."
How would you judge this student' s answer?1. EXCELLENT (all stages correct in the right order with clear and correct explanations) 2. GOOD (all stages correct in the right order, but the explanations are not as clear as they should be). 3. MEDIOCRE (one or two stages are missing OR the stages are in the wrong order, OR the explanations are not clear OR the explanations are irrelevant) 4. UNACCEPTABLE (more than two stages are missing AND the order is incorrect AND the explanations are not clear AND/OR they are irrelevant) 31
MCQ for testing Evaluation
Does the test overall: Test the range of content? Test:
knowledge/comprehension/application/analysis/ evaluation/synthesis as intended?
Avoid giving away answers to questions in other questions?
Have a range of a/b/c/etc correct answers Consider inclusivity
32
Checklist for tests
33
Pros and Cons of MCQsAdvantages
Banks of questions created and can be used for reuse/sharing
Reliable (if set correctly…) Marking is time efficient, no
second marking required Rapid feedback to students
and staff Enables assessment of large
range of curriculum
Disadvantages Time to compile questions Skill and scrutiny still required in the
setting process Difficulties in compiling questions
for higher order cognitive skills cannot assess innovation/creativity
Concerns re guessing Partial knowledge not recognised in
more straight forward questions If only used formatively may give
students unreal expectations in other forms of assessment
Concerns re relatively high marks
34
Conclusion
Anderson, L . W., Krathwohl D. R., Airasian, P. W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., James Raths, & Wittrock, M.C. (eds). (2000). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Allyn and Bacon
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: the classification of educational goals; Handbook I: Cognitive Domain New York, Longmans, Green, 1956
Brown, G., with Bull, J., and Pendlebury, M. (1997). Assessing Student Learning in HigherEducation. London: Routledge.
Chickering, A.W. & Gamson, Z.F. (1987). Seven Principles for good Practice in Undergraduate Education. American Association for Higher Education Bulletin, 39(7): 3-7. Accessed 2/6/12
Gardner-Medwin AR (2006) Confidence-Based Marking – towards deeper learning and better exams In : Innovative Assessment in Higher Education. Ed.: Bryan C and Clegg K. Routledge, Taylor and Francis Group, London
Fry, H., Ketteridge. S. and Marshall, S. (2005) A handbook for teaching and learning in higher education. Enhancing academic practice. 2nd edition. Routledge Falmer . Oxon.
Hornby, Win (2003) Dogs, Stars, Rolls Royces and Old Double Decker Buses: Efficiency and effectiveness in assessment. Accessed on-line 24/6/12 http://www.enhancementthemes.ac.uk/docs/publications/reflections-on-assessment-volume-i.pdf
Quinn F.M & Hughes S (2007) Quinn’s principles and practice in nurse education. 5th edition. Cengage learning. UK.
35
References
Additional sources
Constructing Written Test Questions For the Basic and Clinical Sciences (2002). 3rd Ed. National Board of Medical Examiners
Designing effective objective test questions: an introductory workshop - Computer Assisted Assessment - Loughborough University
Medical Schools Council (2012) http://www.medschools.ac.uk/MSC-AA/examplequestions/Pages/EMQs.aspx
SEEC Credit Level Descriptors for Higher Education (2010) http://www.seec.org.uk/sites/seec.org.uk/files/SEEC%20Level%20Descriptors%202010_0.pdf
UCL Home of Certainty-Based Marking http://www.ucl.ac.uk/lapt/
Writing multiple choice questions Virginia Commonwealth UniversityDesigning and Managing MCQs University of Cape Town 36
Bibliography
37
Examples
Q17. What is the scientific notation for 1,700,000?
1.7
1.7 x 106
1.7 x 10-6
17 x 106
17 x 10-6
20% 20% 20%20%20%1.1.72.1.7 x 106
3.1.7 x 10-6
4.17 x 106
5.17 x 10-6
39
Q3: You invest £200 at a nominal interest rate of 7% per annum, compounded quarterly. What is the value of your
investment after 3 years?1. £200(1 + 0.07/4)12 = £246.292. £200(1 + 0.07)3 = £245.013. £200(1 + 0.07/3)12 = £263.784. £200 + 1.073 = £201.235. (£200 x 0.073) + £200 = £200.076. £200(1 + 0.07)7 = £225.827. None of the above
Countdown
90