7/30/2019 PBLI - Evaluation Project
1/17
Practice Based Learning
and ImprovementStephen J. Kimatian MD
Assistant Professor of Anesthesiology and PediatricsThe Penn State, Milton S. Hershey Medical Center
7/30/2019 PBLI - Evaluation Project
2/17
PRACTICE-BASED LEARNING
AND IMPROVEMENT
Residents must be able to investigate
and evaluate their patient carepractices, appraise and assimilate
scientific evidence, and improve their
patient care practices.
7/30/2019 PBLI - Evaluation Project
3/17
Residents are expected to:
analyze practice experience and perform
practice-based improvement activities using a
systematic methodology locate, appraise, and assimilate evidence from
scientific studies related to their patients health
problems
obtain and use information about their ownpopulation of patients and the larger population
from which their patients are drawn
7/30/2019 PBLI - Evaluation Project
4/17
Residents are expected to:
apply knowledge of study designs and statistical
methods to the appraisal of clinical studies and
other information on diagnostic and therapeuticeffectiveness
use information technology to manage
information, access on-line medical information;
and support their own education facilitate the learning of students and other
health care professionals
7/30/2019 PBLI - Evaluation Project
5/17
The Mandate
Develop the skills of critical self
examination and self improvement
Become life long learners
7/30/2019 PBLI - Evaluation Project
6/17
The Question
How do you evaluate yourself if you do
not know what the standard for
competence is? Must have an established standard.
How do you establish the standard for
competence?ACGME competency as applied to practice?
Practice as applied to ACGME competency?
7/30/2019 PBLI - Evaluation Project
7/17
Perspective
What is a consultant?
Knowledge Skill
Judgment
How do you evaluate?
Open Ended Easy to use but low on detail
Highly Specific Complex and time intensive
7/30/2019 PBLI - Evaluation Project
8/17
Evaluation Tools
Easy to understand
Intuitive to use
Generate useful information For self evaluation
For evaluation of competence
For focused remediation
Minimize Bias
7/30/2019 PBLI - Evaluation Project
9/17
The Evaluation Project
Goal to develop an evaluation tool that
maintains a global perspective while
allowing detailed analysis of specificareas of strength or weakness.
A tool that is simple to use
Standardized language for the discussion ofcompetency.
7/30/2019 PBLI - Evaluation Project
10/17
The Evaluation Project
Phase 1Define Consultant using
discrete statements.
Phase 2 Characterize definingstatements.
Phase 3 - Create a Database (rubrics)
that defines Competency Phase 4 - Manipulate the data base to
create evaluation tools.
7/30/2019 PBLI - Evaluation Project
11/17
The Project - Phase 1
Define Consultant in discrete statements.
Intravenous Access
Display knowledge of indications for IV access Develop plan for IV placement considering medical and surgical
factors effecting gauge, position, and timing of IV placement.
Communicate with patient throughout procedure showing
respect and compassion
Display knowledge of aseptic technique and universalprecautions
Display proper aseptic technique and universal precautions
7/30/2019 PBLI - Evaluation Project
12/17
The Project Phase 2
Characterize defining statements Subspecialty (general, peds, cardiac .)
Phase of Care (preop, op,postop,clinical,..)
Core Competency (pt care, pbli, comm..)
Educational Domain (knowledge / skill /judgment)
Training Level (PGY1, PGY2, PGY3.
consultant)
7/30/2019 PBLI - Evaluation Project
13/17
The Project Phase 2 (cont)
Intravenous Access Display knowledge of indications for IV access
General / Preop / Pt Care / Knowledge / PGY 1
Develop plan for IV placement considering medical and surgicalfactors effecting gauge, position, and timing of IV placement
General / Preop / Pt Care / Judgment / PGY 1
Communicate with patient throughout procedure showingrespect and compassion
General / Preop / Pt Care / Skill / PGY 1
Display knowledge of aseptic technique and universalprecautions
General / Preop / Pt Care / Knowledge / PGY 1
Display proper aseptic technique and universal precautions
General / Preop / Pt Care / Skill / PGY 1
7/30/2019 PBLI - Evaluation Project
14/17
The Project Phase 3
Create a Database (rubrics) that define CompetencyIV Access Cognitive / Knowledge Skills / Behaviors Attitude / Judgment
Patient Care:
Residents mustbe able to
provide patient
care that is
compassionate,
appropriate, and
effective for the
treatment of
health problems
and thepromotion of
health.
PGY1/2
-Display Knowledge ofindications for IV access
-Display knowledge of
aseptic technique and
universal precautions
PGY1/2 -Communicate with
patient showing respectand compassion
-Display proper aseptic
technique and universal
precautions
PGY1/2
-Develop plan for IVplacement considering
medical and surgical
factors effecting gauge,
position, and timing of IV
placement
CA2
- Display knowledge of
advance IV access
modalities
CA2
-Demonstrate skills needed
to establish IV access in
difficult patient- Effectively Place IV in
uncomplicated patient with
little or no assistance
CA2
CA3 CA3 CA3
7/30/2019 PBLI - Evaluation Project
15/17
The Project Phase 4
Develop Web based evaluation form
Rotating selection of 10 15 questions
Substandard rating result in automatic expansion of question
Resident performance tracked System tracks which competencies have been evaluated
Comparison of resident self evaluation vs. faculty evaluation
Deficiencies characterized to create remediation plan focused on
specific problems and with identifiable points for reevaluation
Faculty Clinical Teaching and Evaluation tracked Puts evaluations in perspective with norms
Identify Biases / Halo Effect
Part of 360 degree evaluation
7/30/2019 PBLI - Evaluation Project
16/17
Project Time Line
Phase 1 / 2
Underway for General Anesthesia
Working model by June 03 Subspecialties added by Fall 03
Phase 3 / 4
First meeting with program team 15 April
Database parameters by June 03
Phase 3 Beta testing by Sept 03
Phase 4 Beta testing by Fall 03
7/30/2019 PBLI - Evaluation Project
17/17
Questions
?