STUDENTS’ ASSESSMENTS IN MEDICAL EDUCATION AMAL AL OTAIBI CCP, MME
Feb 25, 2016
STUDENTS’ ASSESSMENTS
IN MEDICAL EDUCATION
AMAL AL OTAIBICCP, MME
OBJECTIVES Define assessment. Identify purpose and importance of
assessment. Enumerate different methods used in
medical education. Differentiate between formative &
summative assessment.
WHY DISCUSS ASSESSMENT?
WHAT IS ASSESSMENT?
The word ‘assess’ comes from the Latin verb ‘assidere’ meaning ‘to sit with’.
In assessment one is supposed to sit with the learner. This implies it is something we do ‘with’ and ‘for’ students and not ‘to’ students (Green, 1999).
WHAT IS ASSESSMENT?Assessment in education is the process of gathering, interpreting, recording, and using information about students’ responses to an educational task. (Harlen, Gipps, Broadfoot, Nuttal,1992)
VALUES AND ATTITUDES ABOUT ASSESSMENT1. Teachers value and believe in students.2. Sharing learning goals with the
students.3. Involving students in self-assessment.4. Providing feedback that helps students
recognize their next steps and how to take them.
5. Being confident that every student can improve.
6. Providing students with examples of what we expect from them.
TYPES OF ASSESSMENT Formative Assessment
Summative Assessment
FORMATIVE ASSESSMENT Assessment for learning Taken at varying intervals throughout a course
to provide information and feedback that will help improve the quality of student learning the quality of the course itself
Provides information on what an individual student needsTo practiceTo have re-taughtTo learn next
KEY ELEMENTS OF FORMATIVE ASSESSMENT
1. The identification by teachers & learners of learning goals, intentions or outcomes and criteria for achieving these.
2. Rich conversations between teachers & students that continually build and go deeper.
3. The provision of effective, timely feedback to enable students to advance their learning.
4. The active involvement of students in their own learning.
5. Teachers responding to identified learning needs and strengths by modifying their teaching approach(es).
Black & Wiliam, 1998
SUMMATIVE ASSESSMENT Assessment of learning Generally taken by students at the end
of a unit or semester to demonstrate the "sum" of what they have or have not learned.
Summative assessment methods are the most traditional way of evaluating student work.
"Good summative assessments--tests and other graded evaluations--must be demonstrably reliable, valid, and free of bias" (Angelo and Cross, 1993).
Formative and summative assessment are interconnected. It is widely and empirically argued that formative assessment has the greatest impact on learning and achievement.
Formative ‘… often means no more than that the assessment is carried out frequently and is planned at the same time as teaching.’ (Black and Wiliam, 1999)
‘… provides feedback which leads to students recognizing the (learning) gap and closing it … it is forward looking …’ (Harlen, 1998)
‘ … includes both feedback and self-monitoring.’ (Sadler, 1989)
‘… is used essentially to feed back into the teaching and learning process.’ (Tunstall and Gipps, 1996)
Summative‘…assessment (that) has increasingly been used to sum up learning…’(Black and Wiliam, 1999)
‘… looks at past achievements … adds procedures or tests to existing work ... involves only marking and feedback grades to student … is separated from teaching … is carried out at intervals when achievement has to be summarized and reported.’ (Harlen, 1998)
PURPOSES OF ASSESSMENT
FORMATIVE To provide a profile of what a student has learnt. To provide feedback to students so they know where to
improve their learning. To motivate students. To help students to develop their skills of self-
assessment. To improve teaching.
SUMMATIVE To evaluate a student’s strengths and weaknesses. To pass or fail a student. To grade or rank a student. To license to proceed. To license to practice. To select for future employment.
IMPORTANCE OF ASSESSMENT FOR LEARNING
[Brown & Knight] “Assessment is at the heart of the student
experience”
[Chalmers]“Assessments define what students see as
important”
[C. Van der Vleuten] - “Tests drive learning”
IMPORTANCE OF ASSESSMENT FOR LEARNING (CONT’D)
Students take cues from what is assessed rather than from what their lecturers assert is important
“If you want to change student learning then change the methods of assessment” [Chalmers]
WEAKNESSES IN ASSESSMENTS - not be representative of the student’s
capabilities- not match learning objectives of the
course- draw on too narrow a domain (e.g., only
written exams used) - be over-weighted with particular skills &
underweight others
COMMON WEAKNESSES IN ASSESSMENT SYSTEMS
Overload of students and staff. Too many assignments with the same deadline set in the
department/school Insufficient time for students to complete assignments Insufficient time for staff to mark the assignments Inadequate or superficial feedback provided to students Wide variations in assessment demands by different
courses/modules Wide variations in marking across courses/modules.
[Brown, Bull and Pendlebury]
OTHER COMMON WEAKNESSES IN ASSESSMENT SYSTEMS Wide variations in marking within a module by tutors Fuzzy or non-existent criteria. Students do not know what is expected of them Students do not know what counts as a good or bad
assignment/project. Assessment viewed by some departments as an extra
rather than a recognized use of staff time
Brown, Bull and Pendlebury
Validity
CHARACTERISTICS OF INSTRUMENTS
Reliability
Educationalimpact
Acceptability
Cost
Validity
Reliability
Educationalimpact
RELIABILITY
(the degree to which the measurement is accurate and reproducible)
VALIDITY
(WHETHER THE ASSESSMENT MEASURES WHAT IT CLAIMS TO MEASURE)
A SIMPLE MODEL OF COMPETENCE
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.
Knows
Shows how
Knows how
Does
Pro
fess
iona
l aut
hent
icity
Cognition
Behaviour
A SIMPLE MODEL OF COMPETENCE
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.
Knows
Shows how
Knows how
Does
Pro
fess
iona
l aut
hent
icity
Written, Oral orComputer based assessment
Performance or hands on assessment
CLIMBING THE PYRAMID......
Knows
Shows how
Knows how
Does
Knows Factual tests: MCQ, essay type, oral…..
Knows how (Clinical) Context based tests:MCQ, essay type, oral…..
Shows howPerformance assessment in vitro:OSCE, SP-based test…..
Does Performance assessment in vivo: Masked SPs, Video, Audits…..
ASSESSMENT METHODS Written examinations
Multiple choiceTrue/False Matching Short answer Essay
Oral
NEW ASSESSMENT METHODS Modified essay question (MEQ) Objective Structured Clinical
Examination (OSCE) Mini-Clinical Evaluation Exercise
(Mini-CEX) 360 degree (multi source feedback) Portfolio
MODIFIED ESSAY QUESTION (MEQ)
An assessment method where you have a scenario followed by sequential questions.
Strengths Good reliability. It measure students’ skills in clinical reasoning and
problem solving abilities using what they know about the topic.
Weaknesses Need resources (human) Time consuming A lot of paper work. It can’t be used alone.
WHAT IS AN OSCE
Objective Structured Clinical Examination
Harden RG and Gleeson FA Assessment of clinical competence using an objective structured clinical examination (OSCE) Medical Education,1979, Vol 13: 41-54
OSCE - OBJECTIVE
All the candidates are presented with the same test
Specific skill modalities are tested at each station History taking Clinical examination Procedures
OSCE - STRUCTURED
The marking sheet for each station is structured.
Structured interaction between examiner and student
OSCE – CLINICAL EXAMINATION
Test of performance of clinical skills candidates have to demonstrate their
skills, not just describe the theory
SKILLS – WHICH SKILLS
Clinical Diagnostic
Procedural skills
Communication skills
OSCE STATIONS TYPESI. Clinical stations involving interaction
between the student & a real or simulated patient.
II. Practical stations involving performance of technical procedures (e.g. performing CPR).
III. Static stations do not involve any physical interaction but the task may be very varied.
ESSENTIAL ELEMENTS OF OSCE STATIONS
I. A clear definition of the task to be performed by student.
II. Precise and clear instructions to students,examiners and simulated patients.
III. A rating form or marking sheet.IV. A list of requirements (e.g.
Equipments,scripts for the simulated patients.
TITLE SHEET
CANDIDATE INSTRUCTION
EXAMINER INSTRUCTION
MARKING SPECIFICATION
MARKING SHEET
REQUIREMENTS LIST
SIMULATED PATIENT INSTRUCTIONS
CLASSIFICATION
PABXRoom
Fire ExitFire Exit
Secretary
Secretary
CR
ConferenceRoom
MIT Room
Tutorial Rm.. 7
Tutorial Rm. 6
Auditorium
Students ASSEMBLYCR
CR
Kitchen
StockRoom
AliHadwer
FemaleWaiting
ElectricalRoom
Laboratory 1
Laboratory 2
CRStockRoom
Rear Entrance
Main Entrance
Library
Mr.Shakil
Con
trol
Roo
m
Computer Lab.
Tutorial Rm. 4
Station - 1
Tutorial Rm. 3
Station - 2
Tutorial Rm. 2
Station - 3
Tutorial Rm. 1
Station - 4
Station - 6
OSC
E
Exam
. Flow
PABXRoom
Rest &
Refreshm
entA
rea
Tutorial Rm. 5
Station - 7
Station - 5
Extra Rest Stationfor Round - 3
COLLEGE OF MEDICINE OSCE REGULATION
Each station is 7 minutes. 1 minute is allowed to students to move
between stations & examiners to arrange themselves.
Minimum 7 stations in each exam, maximum 9.
Training of simulated patients & examiners is obligatory.
Feedback session among the examiners & Block
Coordinator after the OSCE exam.
Every candidate does the same examinationGreater opportunity for objectivityOSCE Stations are re-useable
Advantages of OSCE
DISADVANTAGES OF OSCE
Costly in terms of resources & manpower
Time consuming to construct & administer
Appropriate space & facilities
Experience needed to administer
MINI-CLINICAL EVALUATION EXERCISE (MINI-CEX) is a rating scale developed by the
American Board of Internal Medicine (ABIM) in the 1990s to assess six core competencies of residents. These are; medical interviewing skills, physical examination skills, professionalism, clinical judgment, counseling skills, organization & efficiency.
MINI-CLINICAL EVALUATION EXERCISE (MINI-CEX) Strengths
Direct observation of candidate performance Allows global evaluation of performance Good inter-rater reliability Practical & easy to use Provide immediate feedback Authentic
Weaknesses Faculty training is needed to improve
reliability It is not possible to assess all aspects of
competencies through a single encounter Needs lot of resources
360 DEGREE (MULTI SOURCE FEEDBACK): consists of measurement tools
completed by multiple individuals in a person’s sphere of influence (ACGME, 2000). Usually, it assesses how frequently a behavior or an action is performed by a candidate using a rating scale (e.g. 1= frequently, 5= never). The observation is done by many different individuals, and generally includes the supervising physicians, peers, and nurses.
PORTFOLIO: is a collection of one’s professionals and
personal goals, achievements, and methods of achieving these goals (Amin and Khoo, 2003). It may contain items such as one’s best essay, written or research project, log books, letter of reflection and evidence of professional growth, to support individual accomplishment and progression (Friedman et al., 2001)
FACTORS INHIBITING ASSESSMENT A tendency for teachers to assess
quantity and presentation of work rather than quality of learning.
Greater attention given to marking and grading, much of it tending to lower self esteem of students, rather than providing advice for improvement.
A strong emphasis on comparing students with each other, which demoralizes the less successful learners.
THANK YOU