NAS Annual Program Review - acgme

Accreditation Council for Graduate Medical Education

Next Accreditation SystemII. Annual Program Review

SNS Program Director WorkshopJune 8, 2013

Pamela L. Derstine, PhD, MHPERRC Executive Director

• Annual Data Submission• Screening Annual Data• Review of Annual Data

© 2013 Accreditation Council for Graduate Medical Education (ACGME)

Topics

• Annual ADS UpdateProgram Changes – Structure and resourcesProgram Attrition – PD / core faculty / residentsScholarly Activity – Faculty and residents

• Board Pass Rate – 7 year rolling average• Clinical Experience – Case logs• Resident Survey – Common and specialty elements• Faculty Survey• Semi-Annual Resident Evaluation and FeedbackMilestones

• Omission of data© 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review

• Annual ADS Update Program Changes – Structure and

resources Participating Site information (e.g., PLA on file, distance

from primary site, rationale, rotation months) Current Block Schedule: programs must follow

instructions and use the block schedule template provided

https://www.acgme.org/ads/Content/Downloads/BlockDiagramInstructions.pdf Sponsoring institution information (e.g., DIO,

participating sites, other sponsored programs)© 2013 Accreditation Council for Graduate Medical Education (ACGME)


• Annual ADS Update Program Changes – Structure and

resources Major changes Response to current citations (if resolved on annual

review, will be removed and stored in program history) Duty Hour, Patient Safety, Learning Environment

(17 items) Overall evaluation methods (6 items)



• Annual ADS UpdateProgram Attrition – PD / core faculty / residents Program director change history since 2000 Program director CV Program faculty basic information Resident basic information Change requests



• Annual ADS Update Scholarly Activity – Faculty and residents CV used for program director Scholarly activity templates used for faculty and

residents Scholarly activity is reported for the most recently

completed academic year (12 month period, not a 5 year period) for all but the PD



Program Director CVsame information requested but entered online


Scholarly Activity Template


Faculty Scholarly Activity

Enter Pub Med ID #’s



Enter a number



Enter a number



Enter a number



Enter a number



Answer Yes or No



AnswerYes or No




Resident Scholarly Activity






• Board Pass Rate: 5 year ADS self-report



• Board Pass Rate – 7 year rolling average ABNS has provided pass rates to the ACGME

electronically for each year beginning with 2003 through 2012 for parts 1 and 2 for all programs

ABNS will provide an annual electronic update to the ACGME, beginning with the 2013 exam results

Annual ABNS reports to ACGME may preclude the need for programs to provide this information



• Clinical Experience – Case logs 2012-2013 program review

- minimum numbers for defined case categories reviewed and feedback provided (no citations based on min. numbers)

2013-2014; 2014-2015 program reviews- graduates expected to comply with minimum number requirements for all categories except critical care (DC20-28) and endovascular (DC3b)

- level-specific reports will be reviewed to monitor progress towards compliance

2015-2016 program review and beyond - graduates expected to comply with all categories



• Resident Survey – Common and specialty elements 7 survey question domains: duty hours; faculty;

evaluation; educational content; resources; patient safety; teamwork

70% response rate required Aggregated non-compliant survey responses for

each domain are reviewed; thresholds for non-compliance

Exception: programs with DHE and non-compliant responses for intimidation and service-over-education will be cited © 2013 Accreditation Council for Graduate Medical Education (ACGME)


• Resident Survey – Common and specialty elements Programs with DHE and non-compliant

responses to 88-hour question- first time non-compliance: warning- second time non-compliance: action plan required

- third time non-compliance: DHE canceled; program may not reapply for DHE



• Faculty Survey 5 question domains:

- supervision and teaching- educational content- resources- patient safety- teamwork

Intended to mirror most resident survey questions and provide opportunity to compare responses by question domain

First survey completed: spring 2013 First RRC review of faculty survey data: spring 2014 Compliance metrics not yet developed



• Milestones First milestone evaluation period: July – December

2013- Residents evaluated as usual by the program (competency-based, multiple evaluators)

First milestone reports to ACGME: Nov/Dec 2013- Collected evaluations reviewed by the CCC- CCC determines milestone level for each resident for each milestone

- Milestone reporting will be done through a link in ADS (not yet available)

Second milestone reports to ACGME: May/June 2014 First RRC review of milestone data: spring 2015



• Annual ADS Update Omission of data If any required annual ADS update information is

missing, the program will be flagged by the NAS data system

Data omission could result in an altered accreditation status




Screening Annual Data

• Thresholds for core annual data established to quickly identify programs with serious problems Aids in prioritizing Review Committee work Facilitates more timely identification of problems

and provision of feedback to programs to correct problems

Goal – ensure ongoing quality resident education


Screening for Program Changes

• Three or more changes

(e.g., participating site, resident complement, block diagram, major structural change, sponsoring institution change, GMEC report of structure change)


Screening for Program Attrition

• Three or more program attrition reports

(e.g., program director change, core faculty decrease, chair change. DIO change, CEO change, resident attrition)


Screening for Scholarly Activity

• Faculty < 100 % faculty with at least two points per year Activity in each category counts as one point

except Each peer-reviewed article and grant counts as one point

• Residents < 80% of residents with at least one point per year Activity in each category counts as one point

except Each peer-reviewed article counts as one point


Screening for Board Scores

• < 85% first time takers (for credit) pass primary exam averaged over last three years

• < 80% first time takers pass oral exam (minimum of 10 residents)


Screening for Case Logs

• Each graduating resident: below minimum number in any defined

case category reporting < 500 total cases (not limited to

defined case categories)


Screening for Surveys

• Resident Non-compliance in duty hours area Non-compliance in three or more areas

• Faculty Not yet established


Screening for Milestones

• Not yet established


Screening for Omissions

• One or more core data elements omitted

Policies and Procedures: 7/1/2013http://www.acgme-nas.org/assets/pdf/FinalMasterNASPolicyProcedures.pdf• Policy 17.61 Review of Annual Data Data RRC may use

RRC actions following annual data review

• Policy 18.00 Accreditation Actions© 2013 Accreditation Council for Graduate Medical Education (ACGME)

Policies and Procedures

Review of Annual Data• Continuous data collection/review ADS annual update Resident survey Faculty survey Milestone data Certification examination performance Case log data CLER information Hospital accreditation data Faculty and resident scholarly activity / productivityOther


Review of Annual Data• Other data (episodic) ACGME complaints (section 23.10) Verified public information Historical accreditation decisions / citations Institutional quality and safety metrics Other


Accreditation Decisions• NO proposed adverse actions • Potential Actions (if currently accredited): progress report focused site visit full site visit continued accreditation accreditation with warning (may not request

permanent complement increase or DHE)


Accreditation Decisions• Potential Actions (if currently accredited): probation (must be preceded by a site visit;

status limit of 2 years; may not request permanent complement increase or DHE)

complement reduction withdrawal of accreditation


Accreditation Decisions• Potential Actions (if currently accredited): Recognize exemplary performance; innovations Identify opportunities for program improvement Identify concerning trends Issue new citations Continue previous citations Acknowledge correction of previous citations


Accreditation Decisions• Accreditation decisions made in the past

12 months will be updated effective July 1, 2013 Short-cycled programs (1-2 year cycle

lengths) will change from continued accreditation to accreditation with warning

Accreditation with warning is NOT an adverse action; residents and applicants do NOT need to be informed


NAS Annual Program Review - acgme

Documents