Slide 1

THE OSCE 30 YEARS ON

James WareKuwait

It is a pleasure to acknowledge some of the thoughts and facts freely offered me from:

• Ronald Harden, Scotland• Dale Dauphinee, Canada• Cees van der Vleuten, Holland• John Norcini, USA• David Newble, Australia• Peter Devitt, Australia• Neil Paget, Australia• Sam Leinster, England

The organisation of this brief history of the OSCE milestones is like this:

1964-1974 Pre-OSCE era, simulated patients introduced and traditional methods questioned

1975-1984 Early OSCE days

1984-1992 Research and refinements improve our understanding and the first conferences

1993-2006 National bodies introduce OSCE to test competence to practice

Barrows HS, Abrahamson S. The programmed patient: a technique for appraising student performance in clinical neurology. J Med Educ. 1964, 39:802-805

1964HS Barrows, a neurologist, with S Abrahamson, an anatomist, introduced the use of programmed patients as a teaching method and then later in assessments at the University of Southern California, reported in 1968

Barrows HS, Abrahamson S. The programmed patient: a technique for appraising student performance in clinical neurology. J Med Educ. 1964, 39:802-805

1964-68Barrows and Abrahamson introduced the use of actors to play the roles of patients – calling them programmed patients, while today we would probable name simulated or standardised patients.

1968John Evans, the first dean at McMaster, Canada, Enticed Barrows away and what was accomplished by this move was the creation of the first PBL school. He was later to recruit Tamblyn, who was another OSCE innovator.

1970s sometimeAn anecdote has it that the Irish Constabulary introduced role play for their objective structure practical examination sometime in the 1970s, but we shall hear of this later.

The source of their simulated rogues and citizens was not given!

1968After graduating at Glasgow University Ronald Harden practiced as a general physician with a special interest in endocrinology, his interest in medical education began already while in Glasgow

1969Ron Harden started creating the beginnings of the OSCE already in Glasgow and was carrying out research to investigate areas which were to raise considerable interest much later.

They reported the variation in rating the clinical competence of candidates using direct observation and videos of candidates. The wide variation (huge in the case of the videos) in determining the fate of these candidates was the challenge they clearly identified

The original OSCEs were different from what we do today and yet in several aspects what was done will resonant with some, such as feedback to the student.

Harden R et al. Two systems of marking objective examination questions. Lancet 1969, 1:40-42

Harden R et al. Examination of clinical examiners. Lancet 1969, 1:37-40

Harden R McG et al. Assessment of clinical competence using objective structured examination. BMJ, 1975, 1:447-451

1970Ron harden moves to Ninewells Hospital and becomes Head of the Division of Clinical Medical Education, University of Dundee. Later he was to set up the highly successful Centre of Medical Education, Dundee. 1975The first description of the OSCE

Harden R McG et al. Assessment of clinical competence using objective structured examination. BMJ, 1975, 1:447-451

History from patient*

Inspection of slide**

MCQ***

MCQ***

Neurological exam*

MCQ***

Inspection of an ECG**

Inspected and scored by an examiner

Inspected and scored by an examiner

* These were real patients

** were these OSPE stations?

*** MCQs or open ended questions

Check lists versus qualified marking (global ratings)Initially (Glasgow) check lists were used to rate the performance, later to be modified to allow expert judgment.

Harden R McG et al. Assessment of clinical competence using objective structured examination. BMJ, 1975, 1:447-451

STATION 1

STUDENT’S NAMEINSTRUCTIONS TO THE EXAMINERS

Yes NoThe candidateFelt the radial pulseCounted the rate with a watchElevated the limb to detect a collapsing pulseLocated the 2nd R space properlyAuscultated the neckAuscultated down the left sternal borderSat the patient up to auscultateAuscultated in full inspirationUsed diaphragm for auscultation

Harden RM, Gleeson FA. Assessment of Clinical Competence using an Objective Structured Clinical Examination (OSCE). ASME Medical Education Booklet No, 8.

Med Educ, 1979, 13:41-54

1979In January 1979, 30 years ago, Ron Harden publishes the seminal paper describing the OSCE, he also introduced the idea of the OSPE.

After 10 years of refinement Harden and Gleeson (his research fellow) are able to describe the OSCE much as we know it today. Others had begun to use the OSCE and would publish their findings

Many still remember the classical long case, unobserved, as much a test of knowledge as one of clinical competence. But interestingly what goes around . . . . . .

Cuschieri A et al. A new approach to a final examination in surgery, use of the objective structured clinical examination. Ann R Coll Surg Engl, 1979, 5:400-4005

1979Professor Alfred Cuschieri, a distinguished hepato-biliary surgeon, introduces the OSCE in the Final Surgery Examinations in Dundee and reports what happened. MCQs were used to check the knowledge base of clinical competence and check lists for Observer-Examiners

1980sAbout this time Ian Hart spent a sabbatical in Dundee with Ron Harden resulting in two landmarks, the introduction of the real OSCE to North America and also the beginnings of the Ottawa Conferences on the Assessment of Clinical Competence

1980Tamblyn assisted the first nursing OSCE in Canada, since which many health sciences have used this examination format. She was later to be heavily involved in the training of simulated patients and supporting the introduction of the OSCE in the MCCQE

1984Sam Leinster ran several OSCEs in Cardiff during the 1970s before coming to Liverpool where he ran the first OSCE in the Department of Surgery there, James Ware was one of his assistants.

It certainly was a learning experience but took me six more years to do my next one under the guidance of Ron Harden in 1990!

Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. J Med Educ. 1988, 22:325-334

1984David Newble was one of several who attended the first Cambridge Conference and left with renewed enthusiasm to set about investigating and testing new methods of measuring clinical competence, among them the OSCE. He was to provide David Swanson with much data used to analyse the psychometric properties

1985The first Ottawa Conference for the Assessment of Clinical Competence, attracting 230 participants from 19 countries who presented sixty-five papers. This was arguably the springboard for which the OSCE took off in many ways, research and the acceptance that this was a realistic way to measure the elusive test of clinical competence – but nothing is perfect.

Hart I, Harden RM, Walton HJ. Newer developments in assessing clinical competence, Proceedings of the first Ottawa Conference, 1986, Ottawa, Canada

1989Dale Dauphinee, who worked for years with the MCC reminds us that the first licensing examination using the OSCE format was held in Canada by the College of Family Physicians. Dauphinee then chaired the Future Directions Committee of the MCC to prepare the OSCE-SP MCCQE Part II examination which eventually was delivered in 1993

1980sDavid Swanson started an in depth look at the OSCE, initially with David Newble’s data. This was to provide many insights and provocative thoughts about the strengths and weaknesses of the OSCE format. He was also preparing the way for the NBME to finally introduce the OSCE twenty years later.

Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. J Med Educ. 1988, 22:325-334

1980-1990sDavid Swanson and Newble’s research with the Adelaide data showed that reliability was a problem compared to written components of Finals Examinations. They concluded that if test length was increased to six hours, criteria of reliability would be met. It was also found that careful station selection improved reliability – and sent a strong message to train OSCE examiners.

They also suggested combining OSCE scores with other sources, a message sent later by van der Vleuten (OSCE + OSPE + CBKA)

Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. J Med Educ. 1988, 22:325-334

Acknowledgement to Cees van der Vleuten for kindly allowing his assembled tabled to be used

0 1 2 3 4 5 6 7 8 9 10

Minimal Testing Time

Newble & Swanson, 1988

Conn, 1986

Van der Vleuten, 1988

Stillman et al., 1987

Colliver et al., 1989

Petrusa, 1988

Cohen et al, 1988

Norcini J, Swanson D. Factors influencing testing time requirements for simulation-based measurements: do simulations ever yield reliable scores.

Teaching and Learning in Medicine 1989; 1:85-91

1989Comparisons of different SPs playing the same role may result in significantly different examinee scores but even out if many encounters are included, while they suggested other sources are controllable.

History taking Communication

Test Length

HoursSame

SP

Different

SP

Same

SP

different

SP

1 0.34 0.33 0.59 0.56

2 0.51 0.50 0.74 0.71

4 0.67 0.67 0.85 0.83

6 0.76 0.75 0.90 0.88

8 0.81 0.80 0.92 0.91

1989Swanson and Norcini concluding their evidence that it is better to have more stations with the available examiners rather than use paired examiners.

Test Length

Hours One examiner

per station

Two examiners

per station

1 0.34 0.33

2 0.51 0.50

4 0.67 0.67

6 0.76 0.75

8 0.81 0.80

Norcini J, Swanson D. Factors influencing testing time requirements for simulation-based measurements: do simulations ever yield reliable scores. Teaching and Learning in Medicine

1989; 1:85-91

Van der Vleuten CPM et al. Training and experience of medical examiners. Med Educ. 1989, 23:290-296

1989Cees van der Vleuten with David Swanson and co-workers studied the impact of training and showed that: need and effectiveness of training was least for physician raters, more needed and effective for medical students and most needed but most effective for non-physicians (including SPs).

Van der Vleuten CPM, Swanson DB. Assessment of clinical skills with standardised patients: State of the art. Teaching and Learning in Medicine. 1990, 2:58-76

1990Cees van der Vleuten and David Swanson published an influential review describing the use of simulated patients. Many would say that SPs are heart and soul of the OSCE, while others would argue that simulation is not the real thing and so the debate goes on.

Roberts J, Norman G. Reliability and learning from the objective structured clinical examination. Med Educ. 1990, 24:219-223

1990Geoff Norman was also contributing with his data on reliability : reporting that inter-rater reliability is high, 0.80 – 0.99; as was test-retest of the same skill, ICC=0.60-0.88, but across different skills correlations were low, alpha = 0.198. This study was with Baccalaureate nursing students.

1990The UK police were not far behind introducing an OSPRE one year later in their national examinations for promotion to sergeant, inspector and detective.

Again, I wonder the source of their simulated rogues and citizens.

1993Dale Dauphinee after tireless work was able to mount the first licensing OSCE 1993 in six centres across Canada. By 1997 the MCCQE Part II exam was delivered in 17 centres with 2000 candidates on a single weekend, but of course this was a team effort, so crucial for the success of any OSCE . . . . . .

Some of the collaborators on this project, interestingly propelled by patient complaints both in Canada and the US. This resulted in a long term study: the outcome proved that scores in the communication and clinical decision making components of the OSCE licensing exam predicted complaints to medical regulatory authorities – the paper is worth a read!

Tamblyn R et al. Physician scores on a national clinical skills examination as predictors or complaints to medical regulatory authorities. JAMA. 2007, 298:993-1001

Dauphinee WD et al. Using the judgments of physician examiners in setting standards for a national multi-center high stakes OSCE. Adv Health Sci Educ Theory and Pract,

1997; 2:201-211

1997The collaborators from MCC describe their innovative Borderline Group Method for standard setting OSCE. It should be added that this method works best when there are many candidates and many standard setters – but many of us use it still in our schools.

1998The AMC Held its first IMG OSCE to license foreign medical graduates for practice in Australia in 1998. There were seven centres catering for about 700 candidates with five exam sessions each year.

The OSCE replaced a clinical exam with two VIVA stations covering diagnostic and management skills of twenty minutes each.

1998ECFMG held its first SP-OSCE. Throughout the investigations carried out by the National Boards it had been found that American medical colleges were resistant to the idea that candidates could pass a knowledge test only to be prevented from being licensed by their marks in a skills test using the OSCE.

So it was that ECFMG pioneered the way and now NBME CS is held in the same five centres as those set up by Commission.

2003John Norcini had for some time been influential for standard setting, an issue which has attracted a considerable body of literature for the OSCE. Most are based on either a Comparative Group method (NBME) or the Borderline Group Method (MCC), but regression methods have considerable merit as well, while Angoff is considered too intensive.

Norcini JJ. Standard setting on Educational tests. Med Educ 2003; 37:464-469

2000The General Medical Council introduces the SP-OSCE as part of the licensing process, PLAB, for foreign medical graduates wishing to practice in the UK

2004Emil Petrusa has been a consistent contributor to what we know about the use of simulated patients and has taken a pragmatic view, which is important about the way we set up our OSCEs. OSCEs scores shall have some predictive value about what will happen in future practice.

Petrusa ER. Taking standardised patient-based examinations to the next level. Teaching and Learning in Medicine, 2004; 16:98-110

1998Evidence for the predictive value of current assessments is relative small. If, as there seems little correlation between scores obtained when demonstrating the sequence of breast examination maneuvers with the ability to detect a lump, yet good evidence that detecting a lump in a silicon breast simulation, why do we still use the former method for assessment?

Chalabian J, Dunnington G. Do our current assessments assure competency in clinical breast evaluation skills?. Am J Surg, 1998; 175:497-502

2006National Board of Medical Examiners hold their first USMLE Step 2 CS examination under the auspices of the Educational Commission, using the same five sites as for ECFMG. The exam lasts eight hours at the centre, that sounds familiar, but there are only twelve stations, compare with 14 in Canada. Each encounter with an SP is videoed and 34-35,000 go through this process annually. The NBME use the contrasting group method to set their standards.

2009

The last word goes to Cees van der Vleuten:

OSCEs are not inherently reliable.

Neither objectivity or standardisation improve this situation.

OSCEs do not test the whole picture.

Global rating scales are superior to check lists.

Computers are efficient replacements for written stations and offer other possibilities.

The Borderline Group Method remains a cheap and valid way to set standards with.

But Maastricht use real life work based assessments to measure clinical competence today.

Thank you