Exams and assessing competence. Part A/Paper 1 changes Part A/Paper 1 changesOSPHEARCP Assessing competence in HK training – the role of exams.

Exams and assessing Exams and assessing competencecompetence

Part A/Paper 1 changesPart A/Paper 1 changes

OSPHEOSPHE

ARCPARCP

Assessing competence in HK training – Assessing competence in HK training – the role of examsthe role of exams

Part A changes -WhatPart A changes -What

Paper I - ‘Knowledge Paper‘ – No changePaper I - ‘Knowledge Paper‘ – No change

Paper II - ‘Skills Paper‘Paper II - ‘Skills Paper‘Proposed change - Proposed change - 2b to become a short 2b to become a short

answer paper – 90 minutes, five sections (10 answer paper – 90 minutes, five sections (10 marks per section) – short questions with marks per section) – short questions with short answers (numerical or no more than a short answers (numerical or no more than a paragraph) - no choice – questions will not paragraph) - no choice – questions will not be published on the websitebe published on the website

Part A – Why change?Part A – Why change?

PMETB - PMETB - direction of travel for work that must be done to ensure that the MFPH is part of an assessment programme that enables the public, the profession and the candidates/trainees to have confidence in those decisions taken on the basis of examination results Changes in response to long term Changes in response to long term aspirations of Southgate reviewaspirations of Southgate review

WhenWhen

Needs PMETB approval before final Needs PMETB approval before final confirmation (PMETB panel meeting confirmation (PMETB panel meeting January 2009)January 2009)

Commencement set for Jan 2010Commencement set for Jan 2010

The OSPHEThe OSPHE

““Objective Structured Public Health Objective Structured Public Health Examination”Examination”

Why the need? Why the need? Validity of the General VivaValidity of the General Viva

Training issues with time spent on thesisTraining issues with time spent on thesis

Issues around the limited competencies Issues around the limited competencies assessed by thesisassessed by thesis

Issues around the narrow range of assessment Issues around the narrow range of assessment available with thesis marking and vivaavailable with thesis marking and viva

OSPHE –part B MFPHOSPHE –part B MFPH

After Part A (Knowledge & Knows How)After Part A (Knowledge & Knows How)

OSPHE (Shows How) OSPHE (Shows How)

ReliableReliable

ValidValid

Will encourage participation in service Will encourage participation in service training opportunities during training as training opportunities during training as this will be the best form of preparation.this will be the best form of preparation.

MilestonesMilestones

Part A – 16m after starting Part A – 16m after starting MastersMasters

Part B – 6-9m wte after Part APart B – 6-9m wte after Part A

What is assessed in the OSPHE?What is assessed in the OSPHE?1. Presenting communication skills in a typical Public Health setting (presenting 1. Presenting communication skills in a typical Public Health setting (presenting

to a person or audience).to a person or audience).

2. Listening communication skills in a typical Public Health setting (listening 2. Listening communication skills in a typical Public Health setting (listening and ascertaining key information).and ascertaining key information).

3. Demonstrating ascertainment of key Public Health points from the material 3. Demonstrating ascertainment of key Public Health points from the material provided and using it appropriately and in relation to wider Public Health provided and using it appropriately and in relation to wider Public Health information sources.information sources.

4. Giving a balanced view/explaining appropriately key Public Health 4. Giving a balanced view/explaining appropriately key Public Health concepts in a Public Health setting.concepts in a Public Health setting.

5.5. Appropriately and sensitively handling uncertainty, the unexpected, conflict Appropriately and sensitively handling uncertainty, the unexpected, conflict and/or responding appropriately to challenging questions. and/or responding appropriately to challenging questions.

ALL the above are part of CurriculumALL the above are part of Curriculum

The OSPHEThe OSPHE

It is a test of 5 competencies – NOT It is a test of 5 competencies – NOT performance at individual stationsperformance at individual stationsIt is possible to not do very well at 2 out of It is possible to not do very well at 2 out of the 6 stations – and passthe 6 stations – and passFeedback is given as a score for each Feedback is given as a score for each competency overall - competency overall - notnot overall station overall station scores.scores.

FORMATFORMAT

12 Stations of 8 mins in each.12 Stations of 8 mins in each.6 of them are briefing/preparation time for the 6 of them are briefing/preparation time for the following assessed stationsfollowing assessed stationsGiving a short presentationGiving a short presentationExplaining something verbally to a health Explaining something verbally to a health professional , lay person , representative , professional , lay person , representative , manager etcmanager etcMedia Interview role playMedia Interview role playPhone conversation Phone conversation Small meetingSmall meeting

OSPHE - formatOSPHE - format

Candidates have 6 stations which equally Candidates have 6 stations which equally and independently tests all five and independently tests all five competenciescompetencies

Each station lasts 17 minsEach station lasts 17 mins

8 mins of briefing/preparation time 8 mins of briefing/preparation time

1 min changeover1 min changeover

8 mins assessment time8 mins assessment time

MarkingMarking

Examiner watching and marking Examiner watching and marking during examduring exam

5 grades5 grades

WeightingsWeightings

Automated scoringAutomated scoring

The OSPHEThe OSPHE

Taken around mid-point stageTaken around mid-point stage of trainingof training

We do not require candidates to be familiar We do not require candidates to be familiar with NHS procedures or any particular with NHS procedures or any particular culture.culture.

Question & Marking examples on Faculty Question & Marking examples on Faculty website www.fph.orgwebsite www.fph.org

Advice to post part A traineesAdvice to post part A trainees

““Spend as little time as possible at desk Spend as little time as possible at desk using PC”using PC”

““Spend as much time as possible doing Spend as much time as possible doing active service PH ; meetings , discussions active service PH ; meetings , discussions with other professionals , managers , lay with other professionals , managers , lay representatives of interest groups , representatives of interest groups , discussions with journalists , radio discussions with journalists , radio interviews etcinterviews etc

As the TrainerAs the Trainer

Ensure their service work/projects are in keeping Ensure their service work/projects are in keeping with lots of talking & interactionwith lots of talking & interactionEvery week before trainer-trainee meeting think Every week before trainer-trainee meeting think of something which has happened over last of something which has happened over last week suitable for a mini-OSPHEweek suitable for a mini-OSPHERole play it with your trainee – then give them Role play it with your trainee – then give them constructive feedbackconstructive feedbackTry to be able to observe your trainee in Try to be able to observe your trainee in meetings etc – and give them feedbackmeetings etc – and give them feedback

350 Euston 350 Euston RoadRoad

Waiting AreaWaiting Area

Briefing Briefing RoomRoom

Exam Exam CorridorCorridor

Prep Prep RoomRoom

Exam Exam RoomRoom

Assessment Assessment

ExamsExams

Work based assessmentsWork based assessments

Includes in-work and real-time assessments, Includes in-work and real-time assessments, directly observed procedures (DOPs), directly observed procedures (DOPs), case-based discussions (CBDs), multi-case-based discussions (CBDs), multi-source feedback (MSF)source feedback (MSF)

EvidenceEvidence

Presented through portfolioPresented through portfolio

In a variety of formats (including In a variety of formats (including supervisor reports)supervisor reports)

Assessed at ARCPAssessed at ARCP

ARCP - Annual Review of ARCP - Annual Review of Competency ProgressionCompetency Progression

Deanery processDeanery process

Requires reports from Educational Supervisor & Requires reports from Educational Supervisor & Academic Supervisor Academic Supervisor

Based on documentation only Based on documentation only – Sign-off sheetsSign-off sheets– Sample of workplace assessmentsSample of workplace assessments– Sample of log-sheetsSample of log-sheets– Other records (MSF, exam etc)Other records (MSF, exam etc)

Documents Required for the Documents Required for the Review by the ARCP PanelReview by the ARCP Panel

Educational Supervisors reports to include any Educational Supervisors reports to include any identified problems with competenciesidentified problems with competenciesCopies of all WBAs completed in that yearCopies of all WBAs completed in that yearUpdated CV – to include a sheet indicating new Updated CV – to include a sheet indicating new achievements since last review such as: achievements since last review such as: examinations, audits, presentations and posters, examinations, audits, presentations and posters, research, record of CPD for that year, details of research, record of CPD for that year, details of placements for that year.placements for that year.Personal development plan for the yearPersonal development plan for the yearResults of MSFResults of MSFPrintout of log book if relevantPrintout of log book if relevant

Educational Supervisor ReportEducational Supervisor Report

Academic Supervisor ReportAcademic Supervisor Report

Workplace Assessment Record SheetsWorkplace Assessment Record Sheets

ARCP PanelARCP Panel

Consider the evidence presented Consider the evidence presented

Consider progress against time-lineConsider progress against time-line

Decide on outcome Decide on outcome

Provide feedbackProvide feedback

(Involves lay representation)(Involves lay representation)

How the Panel WorksHow the Panel WorksConvened by the deanery - a minimum of 5 membersConvened by the deanery - a minimum of 5 members

Chaired by the chair of the STC, TPD or Associate DeanChaired by the chair of the STC, TPD or Associate Dean

Process is an assessment of documentation and submitted Process is an assessment of documentation and submitted evidenceevidence

Trainee Trainee should not normally attend the panel meetingshould not normally attend the panel meeting

Panel does not plan remedial programmePanel does not plan remedial programme

Other issues may be discussed by the panel but assessment Other issues may be discussed by the panel but assessment of evidence and judgement must be kept separateof evidence and judgement must be kept separate

ARCP OutcomesARCP Outcomes

1 – Satisfactory progress1 – Satisfactory progress

2 – Specific development needed2 – Specific development needed

3 – Inadequate progress3 – Inadequate progress

4 – Left programme 4 – Left programme

5 – Incomplete evidence5 – Incomplete evidence

6 – Completed training6 – Completed training

Outcomes from the ARCP panelOutcomes from the ARCP panel

1.1. Satisfactory progressSatisfactory progress – achieving – achieving progress and competences at expected progress and competences at expected rate.rate.

2.2. Unsatisfactory progressUnsatisfactory progress – further – further development of specific competencies development of specific competencies required but additional training time NOT required but additional training time NOT required i.e. acceptable progress overall required i.e. acceptable progress overall and not expected to delay training. TPD and and not expected to delay training. TPD and ES will make sure the trainee and employer ES will make sure the trainee and employer know what must be done.know what must be done.

Continued…...Continued…...3. 3. Inadequate progress and additional training Inadequate progress and additional training

time requiredtime required – panel identifies need for – panel identifies need for additional training time and recommends what is additional training time and recommends what is required and the level of supervision BUT the required and the level of supervision BUT the deanery determines the details depending on deanery determines the details depending on local circumstances and resources. The total local circumstances and resources. The total overall extension should be no more than 1 year. overall extension should be no more than 1 year. An absolute maximum of 2 years during the total An absolute maximum of 2 years during the total of training can only be granted by the of training can only be granted by the Postgraduate Dean. Time can be made up of Postgraduate Dean. Time can be made up of shorter periods of time divided over the course of shorter periods of time divided over the course of training.training.

Continued…...Continued…...

4. 4. Released from programmeReleased from programme with or with or without specified competencies. The panel without specified competencies. The panel can recommend a release from can recommend a release from programme if insufficient and sustained programme if insufficient and sustained lack of progress is seen despite additional lack of progress is seen despite additional training time. Any achieved competencies training time. Any achieved competencies are recorded and documented. The are recorded and documented. The trainee must give up their training number trainee must give up their training number and advice on career options should be and advice on career options should be made available.made available.

Continued…...Continued…...5.5.Incomplete evidence presented – Incomplete evidence presented – additional training time may be additional training time may be required.required. The trainee supplies incomplete The trainee supplies incomplete or insufficient information such that the or insufficient information such that the panel cannot make a statement about panel cannot make a statement about progress.progress.

6.6.Completion of trainingCompletion of training – all necessary – all necessary competencies have been achievedcompetencies have been achieved

Comments on ARCPComments on ARCP

Just startedJust started

Systematic and uniform for all specialitiesSystematic and uniform for all specialities

Moving to electronic portfolio Moving to electronic portfolio

e-portfolioe-portfolioCurrently being piloted with a group of trainees & Currently being piloted with a group of trainees & trainerstrainersPlanned to be up and running for all in January 2009 Planned to be up and running for all in January 2009 AAAllows every functionality encompassed by paper Allows every functionality encompassed by paper based portfoliobased portfolioGives interaction between educational supervisor Gives interaction between educational supervisor and traineeand traineeEvidence can be stored either confidentially or Evidence can be stored either confidentially or sharedsharedAble to support any file format (pdf excel word ppt Able to support any file format (pdf excel word ppt music files etc)music files etc)MSF incorporatedMSF incorporated

Front page for a traineeFront page for a trainee

Front page as a trainerFront page as a trainer

View portfolioView portfolio

AssessmentsAssessments

DiaryDiary

Activities e.g. presentationsActivities e.g. presentations

Add evidenceAdd evidence

Benefits e-portfolioBenefits e-portfolio

Hold record of trainee work, trainers’ Hold record of trainee work, trainers’ assessmentsassessments

Avoid time wasted on formattingAvoid time wasted on formatting

Avoid time wasted on re-typing Avoid time wasted on re-typing

Allow trainers and trainees to keep track of Allow trainers and trainees to keep track of their communicationstheir communications

Summarise progress (for trainees, trainer Summarise progress (for trainees, trainer and TPD)and TPD)

Summarise what still needs to be doneSummarise what still needs to be done

Assessing competence in HK Assessing competence in HK training – the role of examstraining – the role of exams

Part A/Part 1Part A/Part 1

Part B (OSPHE)/Part 2Part B (OSPHE)/Part 2

?Portfolio assessment ?Portfolio assessment

Possible scenariosPossible scenarios

Do nothingDo nothing

Replace Part 2 General Viva with OSPHE Replace Part 2 General Viva with OSPHE and keep thesis examination and keep thesis examination

Replace Part 2 General Viva with OSPHE Replace Part 2 General Viva with OSPHE and replace thesis examination with formal and replace thesis examination with formal and structured portfolio examinationand structured portfolio examination

Replacing general viva with Replacing general viva with OSPHEOSPHE

OSPHE structure validated, tried and OSPHE structure validated, tried and testedtested

Scenarios will need to be HK relevantScenarios will need to be HK relevant

Examiner training Examiner training

Group Work – Assessing competence Group Work – Assessing competence in HK training – the role of examsin HK training – the role of examsPart 1/A exam – is it appropriate for HK?Part 1/A exam – is it appropriate for HK?

What is the role of Part 2 general viva exam? What is being What is the role of Part 2 general viva exam? What is being assessed and is the viva exam the best way to assess it?assessed and is the viva exam the best way to assess it?

What are the advantages and disadvantages (problems) of the Part What are the advantages and disadvantages (problems) of the Part 2 general viva exam for HK?2 general viva exam for HK?

What are the benefits of replacing the general viva with the What are the benefits of replacing the general viva with the OSPHE?OSPHE?

What are the disadvantages, problems and barriers to replacing the What are the disadvantages, problems and barriers to replacing the general viva with the OSPHE?general viva with the OSPHE?

GroupworkGroupworkWhat is the role of Part 2 thesis exam? What is What is the role of Part 2 thesis exam? What is being assessed and is this exam the best way to being assessed and is this exam the best way to assess it?assess it?

What are the advantages and disadvantages What are the advantages and disadvantages (problems) of the Part 2 thesis exam for HK?(problems) of the Part 2 thesis exam for HK?

What are the benefits of replacing the thesis What are the benefits of replacing the thesis exam with a portfolio examination?exam with a portfolio examination?

What are the disadvantages, problems and What are the disadvantages, problems and barriers to replacing the thesis exam with a barriers to replacing the thesis exam with a portfolio examination?portfolio examination?

Group work – messages to take Group work – messages to take back to the FPHback to the FPH

About Part A changesAbout Part A changes

About the relevance or not of the OSPHE About the relevance or not of the OSPHE exam for HK – give reasons exam for HK – give reasons

Group sessionGroup session

If we started from scratch how would we If we started from scratch how would we design a PG training programme in PH?design a PG training programme in PH?

1.1. RecruitmentRecruitment

2.2. StructureStructure

Think about what you have now, what you Think about what you have now, what you want to keep and what you want to want to keep and what you want to change and why.change and why.

Feedback from groupsFeedback from groups

What did your ideal training programme What did your ideal training programme consist of?consist of?

From the ideas you have generated for an From the ideas you have generated for an ideal training programme what do you ideal training programme what do you think is achievable in the think is achievable in the

short term?short term?

long tem?long tem?