1 Assessment of Students Competence in Health Professionals Education Professor Hossam Hamdy University of Sharjah.

1

Assessment of Students Competence in Health Professionals Education

Professor Hossam HamdyUniversity of Sharjah

2

Professional Competence

The Habitual and Judicious use of:

• Communication

• Knowledge

• Technical Skills

• Evidence-based decision-making

• Emotions

• Values and reflection to improve the health of the individual patient and the community

3

WHAT Should Be Assessed?

ACGME Competencies- Patient care- Knowledge- Practice-based learning and

improvement - Interpersonal and communication skills- Ethics & Professionalism- System-based practice

4

Blueprinting

Test content matches objectives

• Assessment matches competencies learnt

• Assessment matches format of learning, “PBL”

5

Aim of Blueprinting

Reduce:A. Construct under-representation:

Biased samplingFew items covering a topic e.g. OSCE

B. Construct irrelevant variance:Flawed items formatInadequate sampling of student Behaviour.

6

Domain of Interest

Test Sample

Test Sample

7

Blueprinting

• Identify clinical problems that examinees should be able to handle.

• For each problem, define the clinical tasks in which the examinee is expected to be competent.

• Define level of performance / resolution

8

Blueprinting

• Prepare a blueprint to guide the selection of problems to be included in the assessment.

• Let the characteristics of the clinical problems define the tasks to be included in the assessment.

A Simple BlueprintCompetenceCategories

CVS RS … GI ….. MSS Endo/Metab

Haem/ Oncology

….

History Taking

Physical Exam

Tests and Procedures

…..

Management

Communication and Patient Education

Pre-Clerkship Phase Theme B.Sc. Exam Blueprint 2003-2004

UNITS

Inheritance,

growth &

development;

degeneration & aging

Nurture,

environment

& nutriti

on

Structure:

function;

positional

relationship

Control, regulation &

communication within the body,

Homeostasis: metabolie, circulatory, acid/base & electrolyte

Response and

adaptation

systems &

mechanisms

Injury, inflammati

on & infection;

loss, repair and substitutio

n; neoplasia

Immunological

& defense mechan

isms

Investigative

procedure &

interpretation

Intervention

prevention &

management

principles

Human behavior, ethical & medico legal

principles

1 2 3 4 5 6 7 8 9 10 11

UNIT VIII:Integrated Multi-System

UNIT VII:

Nervous System, Special Sense & Human Behavior

UNIT VI:

Integumentary & Muscular-Skeletal

UNIT V:

Hematopoietic & Immune Systems

UNIT IV:

Endocrine, Metabolism & Reproductive System

UNIT III:

GIT & Renal System

UNIT II:

Respiratory & CVS

UNIT I:

Concepts & Principles & Community Health

11

12

A Simple Model of Competence

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Pro

fess

iona

l aut

hent

icity

Written, Oral orComputer based assessment

Performance or hands on assessment

13

Validity Climbing the Pyramid . . .

Knows

Shows how

Knows how

Does

Knows Factual tests: MCQ, essay type, oral…..

Shows howPerformance assessment in vitro:OSCE, SP-based test…..

DoesPerformance assessment in vivo: Masked SPs, Video, Audits…..

Knows how(Clinical) Context based tests:MCQ, essay type, oral…..

14

Sample Good Question & ‘Cover The Options’ Rule

A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis?

A. Acute disseminated encephalomyelitisB. Guillain-Barré syndromeC. Myasthenia gravisD. PoliomyelitisE. Polymyositis

15

Sample Good Question & ‘Cover The Options’ Rule

A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis?

A. Acute disseminated encephalomyelitisB. Guillain-Barré syndromeC. Myasthenia gravisD. PoliomyelitisE. Polymyositis

16

Assessment of Doctor’s Performance

Fundamental to “Good Medical Practice”

“What doctors do in controlled representations” of practice e.g. “OSCE”

“What doctors do in real life”

Rethans et al, Med Ed 2002.

17

•Unobserved 30 – 45 minute interview and examination on a selected patient

•Candidates present only their findings to the examiner

The Traditional Long Case Examinations (LCE)

18

• It assesses the integrated interaction between the student/doctor and the real patient

• Authenticity high

• More valid than the task given in an OSCE

• Little is known about the construct validity and consequential validity “Educational Impact”

)LCE (Validity

19

• Poor intercase reliability

• Content specificity is the most crucial issue in the assessment of clinical competence

• Broad sampling across cases is essential “Multiple Biopsies”

• Logistics will be difficult

)LCE (Reliability

20

Clinical Encounter

Method

Mini – CEX “Norcini 2003” components of the encounter observed

Direct Observation Clinical Encounter Examination “DOCE”

Entire encounter observed “Hamdy 2000”

21

• Three hour examination of four 45 minutes observed long cases

• Real patient selected from a predetermined blue print of common diseases

• Check List – clinical encounter domains

• Reliability 0.8 (Cronbach’s alpha)

(Hamdy et al, Med Educ, 2003)

Direct Observation Clinical Encounter Examination

The DOCEE

22

Practical Procedural Skills

Method

Direct Observation Practical Procedure Skills “DOPS”

Whole procedure observed from start to finish

23

Multi-source Feedback (MSF) 360 Degree Assessment

Peers

Supervisors

Nurses

Secretary

Lab Technicians

Patients

24

Problems of Workplace Based Assessment of Performance

Sampling: variability in the complexity of patients problem

Judges errors:

Time: Requires adequate allocation of time and resources.

Management of Data: not easy

Validity and Reliability: variable