Audit with Feedback as a CME Tool for Radiation Oncologists:

Audit with Feedback as a CME Audit with Feedback as a CME Tool for Radiation Oncologists:Tool for Radiation Oncologists:Evaluation of efficacy, perception and Evaluation of efficacy, perception and

cost-effectivenesscost-effectiveness

Dept. Radiation Oncology,The Cancer Institute (TCI) Singapore

Tom Shakespeare MBBS, MPH, GradDipMed(ClinEpi),

FRANZCR, FAMS

BackgroundBackground

CME efficacy differs between medical specialties

little research of CME for radiation oncologists

literature review: no study evaluating RO-specific CME interventions*

*Shakespeare 2003. Ann Acad Med Singapore

BackgroundBackground

Primary CME: lectures, conferences

Secondary CME: eg audit with feedback (AWF), reminders– Secondary CME more effective than primary

CME: meta-analysis (Davis, JAMA, 1992)

– AWF effective: RCTs/meta-analysis (O’Brien et al, 2001, Cochrane Database)

TCI AWF CME programTCI AWF CME program

Commenced targeted AWF meeting for QI / CME June 2001

no other RO-specific CME program

TCI AWF CME programTCI AWF CME program

1. Fortnightly random chart audit– 2 records per consultant– independently scored using a 10-item checklist – checklist targets RO behavior-based on

RANZCR recertification instrument

Date:    

Consultant:    

Patient ID:    

Treatment site:    

     Departmental file:   Registry sheet completed

    Letter to referring doctor

    Rx intent described

    Tumour stage described

     Treatment chart:   Treatment site described

    Laterality doublet noted

    RT dose point specified

     Isodose plan:   Isodose plan signed

     Simulation film:   Patient name

    Film countersigned

Items audited in CME program checklist

TCI AWF CME programTCI AWF CME program

2. Departmental CME meetingaudit results presentedcase management discussedfeedback provided by senior peers (case and

audit)ranked audit results displayed at end

Study objectivesStudy objectives

Evaluation of educational endpoints– program perception– professional practice (behaviour and

performance)– cost-effectiveness

MethodsMethods

Chart reviewall new patients seen by ROs at 2 time points

– 2 months before program implementation (T0)– months 13-14 of the program (T1)

19-item checklistmeasured RO behaviour/performance

Targeted behaviour criteriaDepartmental notes

Registry sheet completeLetter to referring doctorTreatment intent describedTumour stage given

Treatment prescriptionTreatment site describedLaterality doublet notedRT dose point specified

Simulation/planningIsodose plan signedPatient name on filmFilm countersigned

Targeted because they are part of the CME items being audited

Behaviour criteria score

• “1” if documented, or if not applicable

•“0” if not documented, or unable to evaluate

Non-targeted behaviour criteriaDepartmental notes:

Primary tumour site documentedHistology documentedDecision for/not for radiotherapy documented

Performance criteriaIndication for treatmentTreatment intentRadiation modalityDoseFractionation scheduleField arrangement

Behaviour criteria score

• “1” if documented, or if not applicable

•“0” if not documented, or unable to evaluate

Performance criteria score

•“1” if Appropriate/Adequate or not applicable

•“0” if inappropriate/inadequate or unable to evaluate

Methods (cont)Methods (cont)19-item instrument

– 10 “targeted” behaviour items (similar to CME AWF checklist)

– 3 “non-targeted” behaviour items– 6 “performance” items

Score “1” if adequate/appropriate”reproducible*

*Shakespeare et al, 2003 Australasian Radiology

Methods (cont)Methods (cont)Comparison of

– scores for each of the 3 categories at T0 and T1

Methods (cont)Methods (cont)

Program perception/satisfaction evaluatedquestionnaire

Cost-effectivenesscost of salaries and consumablescost-per-criterion point gained

ResultsResults

113 and 118 patient cases evaluated at T0 and T1

targeted behaviour improved – 8.7 to 9.2 out of 10 (p=0.0001)

no significant change in non-targeted behaviour or performance items

ResultsResultsImprovement in

– documentation of decision for RT (non-targeted): adequacy increased 84% to 92% (p=0.08)

– letter to referring doctor: 53 to 66% (p=0.04)– description of treatment intent: 54 to 77%

(p=0.0002)– laterality noted twice: 91 to 98% (p=0.04)– isodose plan signed: 94 to 100% (p=0.006)

Change in Behavior & Performance of Radiation Oncologists before and 1 year after commencing CME program

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Behaviour / Performance Items Audited

Per

cen

tag

e P

rese

nt

at A

ud

it

2001 (pre-CME)

2002 (1 year of CME)

Behaviour item improvement: Behaviour item improvement: variation between ROsvariation between ROs

Radiation Oncologist

Criterion % Scoring “1” at T0

% Scoring “1” at T1

P value

RO 1        

  Treatment intent described 48.4% 88.5% 0.0001

  Isodose plan signed 83.9% 100.0% 0.003

RO 2        

  Decision for treatment documented

82.0% 97.1% 0.03

  Tumour stage described 91.8% 100.0% 0.09

RO 3        

  Treatment intent described 52.4% 84.4% 0.01

  Laterality doublet on treatment chart

81.0% 96.9% 0.05

Results: Program perceptionResults: Program perception

initial perception ambivalent– mean score 3.2 (out of 5)

increased significantly after evaluation of CME was presented– mean score 3.7 (p=0.0001)

Results: Cost-EffectivenessResults: Cost-Effectiveness

Cost (over 1 y): $US7,897 (91% salaries)cost-per-point gained: $15.67

– deemed cost-effective by participants and TCI

ConclusionsConclusions

Targeted audit with educational / ranked feedback effectively increased targeted behaviour of ROs

–consistent with RCTs and meta-analyses

ConclusionsConclusionsHoweverno definite impact on non-targeted

behaviour itemsno impact on high-scoring non-targeted

performance criteria– highlights need to target CME interventions– consistent with published research

ConclusionsConclusions

All ROs benefited improvements varied between radiation

oncologists– consistent with literature– importance of “Educational Needs Assessment”

prior to program design

ConclusionsConclusions

AWF positively perceived after evaluation of the intervention was

discussedneed to evaluate and report on the efficacy

of CME interventions

ConclusionsConclusionsThe future

– further research required– improve program (eg modify criteria)– integrate CME with QA/QI programs – MS Access database with planned analysis July

2004

TCI QI and CME DatabaseTCI QI and CME DatabaseBeta-version available at: www.theshakespeares.com/CME

TCI Quality Improvement and CME Programme: Start Page

Audit With Feedback Meeting

CME Tutorial Meeting

Simulation Review Meeting

Created by Tom Shakespeare and Rahul Mukherjee. For more information email [email protected]

Thank YouThank You

Thank YouThank You