Assessing Performance: New Strategies Robin Blackstone, MD, FACS, FASMBS President, ASMBS MISS 7:30am Saturday February 25
Jan 13, 2016
Assessing Performance: New Strategies
Robin Blackstone, MD, FACS, FASMBSPresident, ASMBS
MISS 7:30am Saturday February 25
Bariatric Surgical QualityOld Paradigm was a very good start……..now it is time to evolve
On February 6, 2012 the ASMBS Executive Council agreed to merge the BSCOE program with the American College of Surgeons(ACS) BCSN program.On February 10, the ACS Board of Regents agreed.
821 programs will merge April 1, 2012 based on the current volume standards.
Foundation of New Program
All current programs in either BCSN or BSCOE will be accredited in the new program under the current volume based standards
Content and Direction of the program will be through committees that are being established out of the current ASMBS and ACS bariatric committees
Joint (ACS/ASMBS/SAGES) Credentialing Recommendations will be made
A new outcomes based standard will evolve to replace the current volume based standards
The program will have a joint data registry
Robust Bariatric Quality Improvement on a National/Regional and State wide basis will be put in place
“Knowing is not enough, we must apply.
Willing is not enough, we must do.”
-Goethe
Mentor Leadership Data Registry
Quality Improveme
nt Collaborativ
es
Outcome Based
Accreditation Standards
Credentialing
Joint Task Force
Charge: To develop recommendations for facilities for credentialing of bariatric surgeons
reports.asmbs.org
“Knowing is not enough, we must apply.
Willing is not enough, we must do.”
-Goethe
Mentor Leadership Data Registry
Quality Improveme
nt Collaborativ
es
Outcome Based
Accreditation Standards
Credentialing
Community surgeons established a culture of reporting dataThe most important change that has resulted from the registry
10
Database launched in 2007; Entry of data into BOLD became a requirement for BSCOE designation in early 2008 Surgeons/surgical practices responsible for data entry
Source of initial data elements and definitions: Research Advisory Committee (RAC), Bariatric Surgery Review Committee (BSRC), SRC Research Department Staff Sought compatibility with NIH LABS data collection
Primary purpose was to monitor compliance with BSCOE requirements; Secondary purpose was for research Compliance monitoring focused only on surgical volume and
30-day serious complications (deaths, transfers, readmissions, revisions and reoperations)
Many questions asked in the database were intended for research
History of BOLDResearch Director SRC: Debbie Winegar
11
Initial IRB situation complex Informed consent required for all patients
IRB oversight required for all: Sites had option to waive oversight to central IRB or retain local IRB oversight
IRB situation simplified in 2010 with move to Copernicus Group IRB
Waiver of documentation of informed consent
Oversight of sites not required by central IRB
No specific qualifications required for data entry personnel Each program determined the appropriate skill set of
individual(s) who would enter their data
In 2010, requirement added that a BOLD Administrator be named for each practice
History of BOLD - 2
12
BSCOE participants questioned the value of data collection relative to the burden
BOLD reports not widely utilized
Felt that too many questions were research-orientedo Patient information/demographics – 13-19 questionso Preoperative visit – 38 questions o Hospital visit – 24 questionso Postoperative visit – 36 questions
A review of BOLD data elements launched in 2010 Surgeon experts proposed a significant reduction in the
number of core data elements Plan approved by BSRC and EC in 4Q2010 Improvements not executed
History of BOLD -3
13
Data self-reported by programs Potential for bias when data entered by key program personnel
with vested interest in outcomes
Heterogeneity of data entry personnel No specific qualifications (e.g., health/science background) have
been required for data entry personnel; each program has selected whomever they feel can fulfill the role
While this policy accommodates all types of program structures, it can lead to variations in data entry practices that can affect data quality
Inadequate training of data entry personnel and lack of data entry support materials Although basic data entry training has been accessible, neither a
software manual nor guidelines on how to enter specific cases have been provided to data entry personnel
Inadequate training allows room for self-interpretation of data definitions and leads to variations in data entry practices
Limitations of the Database
14
Non-standard data definitions The use of non-standard data definitions in BOLD (e.g., comorbidity
severity scale; certain complications) limits its utility for making comparisons with or for combining with data from with other registries
Lack of sensitivity of the comorbidity severity scale The severity scale captures improvement in many comorbidities
only when there is a major change in treatment regime (e.g., whether medications are required) unlike changes in standard measures (e.g., HbA1c)
Leads to potential underestimation of comorbidity resolution
Potential under-representation of complications Surgical programs have been responsible for entering all
complications in BOLD, even those managed by other health care providers
The likelihood is high that bariatric programs may not be aware of all events that have occurred thus, complications in BOLD are potentially under-represented
Limitations of the Database - 2
15
Variations in the time to data entry There has never been a required time frame with which data must be
entered into BOLD following a patient visit
Leads to the appearance of missing follow-up data at a given point in time
Frequency of on-site data auditing The current schedule for on-site data auditing (every three years as
part of site inspection) may not be sufficient to detect program-specific issues and to keep programs honest
Thoroughness of data auditing BOLD data auditing has been limited to surgeries performed during a
one-year period preceding the site inspection and has been focused on a subset of data elements: procedures, complications, readmission, reoperations and transfers
In general, ~10% of cases selected for random chart review
A 100% chart review triggered by unreported events
Limitations of the Database - 3
16
Inconsistent follow-up intervals among programs Necessitates the use of wide visit windows for the assessment of
outcomes
Incomplete follow-up data Lack of a follow-up tracking report in BOLD has been an
inconvenience for programs however incomplete follow-up is a common problem faced by all bariatric databases
Inadequate feedback to programs through BOLD reports- The BOLD reports provided to date have not shown a program’s
risk-adjusted outcomes vs national risk-adjusted benchmarks nor have they presented all program data in a useful format (e.g., comorbidity data)
- The usefulness of BOLD data for quality improvement has been limited
Limitations of the Database - 4
Potential settlement agreement will provide the data through March 31, 2012 to ASMBS for all programs (unless you specifically opt out)
All programs will receive a risk adjusted and reliability adjusted composite measure and comprehensive data report for each surgeon and program participating on June 22, 2012 at the annual meeting during the first Annual National Bariatric Quality Forum from 10am - 12 noon. All programs should make sure the bariatric coordinator and medical director of the program are there to participate in that meeting.
The data will not be migrated due to the limitations as noted in the previous slides
What will happen to the BOLD data in the transition
18
BOLD Data Dissemination Committees through Research Committee
Data Access Committee (DAC) reviewed requests for BOLD data
Through Sept 2011, 106 requests for data received, 30% for research, 17% for quality improvement
The majority of requests (58%) were received from BSCOE participants
Data Dissemination Committee (DDC) reviewed requests for publication of information derived from BOLD data analysis
Through Sept 2011, 20 abstracts/presentations and 6 manuscripts containing BOLD data were reviewed by the DDC
BOLD data provided for payor negotiations; Data on sleeve provided to CMS for NCD consideration
Dissemination of BOLD Data
19
Programs genuinely want to do the right thing in terms of data collection but they need clear information on what is expected
Data collection is expensive for everyone but is essential for quality assessment and improvement
Have a clear plan as to what the purpose of the data collection is and communicate that to program participants
Tie data collection to accreditation to emphasize its importance
Less is more - focus on fewer data points and do a better job with them
Proper training of data entry personnel is critical for obtaining high quality data
Feedback of information to programs demonstrates the value of the data collection effort and keeps them engaged
Lessons Learned
New Data Registry April 1 2012
Strong and specific data definitions
Parsimonious data set
No IRB required for CQI except as required locally
Data validation (digital oversight) ongoing through data safety monitoring board – site visited as needed to assure compliance
Collection methods – certified data entry person
MOC
Analytics and Reporting The Digital Camera Model vs. Film
“Knowing is not enough, we must apply.
Willing is not enough, we must do.”
-Goethe
Mentor Leadership Data Registry
Quality Improveme
nt Collaborativ
es
Outcome Based
Accreditation Standards
Credentialing
Volume and relationship to mortality
The mortality of bariatric surgery has declined dramatically from 2004 (o.2%) to 2009 (0.1%)
The primary driver of the decline was the wide spread adoption of laparoscopic surgery from 2004 (29.9%) to 2008 (90.2%)
The second biggest driver was the inclusion of a very low risk procedure (Adjustable Gastric Band) 2004 (1%) vs. 2008 (29%)
Not clear how much additional decrease in mortality was due to volume based accreditation
Failure to rescue patients may be of critical importance – emphasizing the importance of collaborative networks of care
Because of the current accreditation stands ASMBS does not have data on programs doing les than 125 cases per year (ACS data seems to indicate that down to 25 cases a year the outcomes are very similar, in a risk adjusted environment. Nguyen, J Am Coll Surg 2011;213:261–266
Lessons from LABS
Historic surgeon experience was not related to adverse event rate
After adjusting for patient risk, the effect of surgeon volume on outcomes for RYGP procedures in LABS showed that for each increase by 10-case per year in surgeon volume the rate of composite events improved by 10 percent.
The observed relationship between surgeon RYBP volume and CE rates was continuous, illustrating that there was no satisfactory level of annual case volume that could act as a threshold for surgeon credentialing within the BSCOE
No significant differences were observed in mortality between low and high volume surgeons.
Smith MK, Patterson E, Wahed AS, Belle SH, Bessler M, Courcoulas AP, Flum D, Halpin V, Mitchell JE, Pomp A, Pories WJ, Wolfe B. Relationship between surgeon volume and adverse outcomes after RYGP in Longitudinal Assessment of Bariatric Surgery (LABS) study. Surg Obes Rel Dis 2010; 6:118-125.
The Role of Risk Adjustment
Important because surgeons feel that if their outcomes are poor it is because they are operating on sicker patients
Challenges in Risk adjustment there is no one factor that has emerged to adjust risk Risk adjustment has changed with time and the data set The most significant predictor of risk in any report has
been the procedure itself The adverse event curve is “J” shaped In order to use data to do risk adjustment the data
definitions have to be crystal clear and the number of patients in follow up has to be high – pulmonary hypertension example
The challenge of Risk Adjustment
Reliability AdjustmentWhen data is unreliable
The “Shrinkage” Debate
• In traditional statistics, the observed rate is thought to best represent the truth (n outcomes/k trials)
• Bayesian statistics considers observed data in the context of prior information
• Empirical Bayes derives prior information from the data– e.g., the “best” guess is a rate
somewhere between the observed rate and the overall rate
• Accomplished using hierarchical modeling
Traditional ApproachShrink to the average mortality
0%
20%
Mo
rtal
ity
rate
(%
)
Mortality rates forhigh-risk surgery 10%
Mo
rtal
ity
rate
(%
)
15%
5%
Overall mean mortality rate
Adjusted for reliability
Observed mortality rates
Box size is proportional to the hospital caseload
Composite Measure Approach: Shrink to the mortality for volume group
0%
20%M
ort
alit
y ra
te (
%)
Mortality rates forhigh-risk surgery 10%
Mo
rtal
ity
rate
(%
)
15%
5%
Observed mortality rates
Low volume
Medium volume
High volume
Mortality rates
Composite mortality
Develop accreditation standards based on outcomes measuresDevelopment of a Bariatric Surgery Composite Measure
Promise of composite outcome measures
• Global indicators of performance—combine multiple domains of quality (structure, process, outcome) into a single quality score• Empirically weight input measures• Filter out statistical noise• Extract as much quality “signal” as possible
from existing data• Designed to optimally describe and forecast
hospital outcomes
Composite Measures are the best predictors of quality to use for accreditation
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Hospital volume Serious complications Composite measure
3-star2-star1-star
Hospital rankings (2008-09)
Risk-adjusted serious complications
(2010)
3.3
2.72.4
3.4 3.43.0
3.2
4.0
4.6
Hospital volumeSerious
complicationsCompositemeasure
Odds Ratio (95% CI), 1-star vs. 3-star 0.85 (0.43-1.68) 1.56 (0.84-2.91) 1.99 (1.14 -3.47)
% VariationExplained 0% 28% 89%
MBSC Data
Volume still counts
Only counts as much as the data dictates it should
With the new program we will get data from the first case
If Volume is found to matter it can be added back based on the data
Key components: Data/structure and process set up at beginning Somewhere between 25-40 cases you will be able to
generate a composite outcomes score
Prediction of serious complication after RYGB using hospital volume from
2008-2009 (BOLD)
1 2 3 4 50.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
5.2 4.9 4.9 5.24.8
Hospital Volume Quintile (08-09)
Ris
k A
dju
sted
Seri
ou
s C
om
plica
tion
in
20
10
Surgeon volume
1 2 3 4 50.0%
2.0%
4.0%
6.0%
8.0%
10.0%
4.9%4.3% 4.6%
5.2%
4.0%
Surgeon Volume Quintile
Ris
k a
dju
sted
seri
ou
s co
m-
plica
tion
BOLD data through 2011
Prediction of serious complication after RYGB using composite measure
from 2008-2009 (BOLD)
1 2 3 4 50.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
2.1
3.4
5.25.8
9.1
Composite Measure Quintile (08-09)
Ris
k A
dju
ste
d S
eri
ou
s C
om
pli
-ca
tio
n i
n 2
01
0
Risk Adjusted/Reliability Adjusted
Broader composite measures?
Global composite measure
Quality of lifeFunctional outcomes
Perioperative safety
Long-term effectiveness
Patient-centered results
MorbidityReoperation
Weight lossResolution of comorbid diseases
reports.asmbs.org
“Knowing is not enough, we must apply.
Willing is not enough, we must do.”
-Goethe
Mentor Leadership Data
Registry
Quality Improvement Collaborative
s
Outcome Based
Accreditation Standards
Credentialing
Goal of the MBSQIP
Establish national/regional and state collaboratives to improve care Decrease morbidity by 50% over
five years Decrease readmissions,
reoperations Improve VALUE of metabolic
surgery by increasing safety, improving efficacy and decreasing cost of care
Share best practices
Collaboratives for Quality ImprovementWhy are they different?
Strategies for improving outcomes
Steer patients to the best hospitals
Improve care everywhere
Accreditation Programs
Pay for Performance
Public Reporting
Michigan Bariatric Surgery CollaborativeExample of the Quality Improvement Process that will be established
The Michigan program in regional collaborative improvement
Partnership between BCBSM, Michigan hospitals, and clinician scientists Pilot test with PCI in1998, broad implementation 2005-
6
$30 million annual investment from BCBSM
12 collaborative quality improvement programs PCI /PVI, Cardiac, NSQIP, bariatrics, breast cancer,
cardiac CT, trauma, joint replacement, and medical admissions
50+ hospitals 200,000+ pts / year
Collaborative quality improvement
Basic idea: Physicians/hospitals collaborate with and learn from each other in improving outcomes
Robust data registry with “digital” capability
Empirical and non-empirical identification of best practices Leveraging “natural experiments” associated with variation in
practice across hospitals and physicians Non-empirical learning
Performance feedback, collaborative meetings, site visits, etc.
Empirical identification and dissemination of best practices Process-outcomes linkage, leveraging “natural experiments” associated
with variation in practice Guideline implementation and evaluation
Health Affairs, April, 2011
Michigan Bariatric Surgery Collaborative (MBSC)
29 hospitals
65 surgeons
7,000 pts / yr
Trends in VTE Guideline Adherence
2006 2007 2008 2009 2010 20110%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Intervention
*Based on random site audit of 1,148 charts
Trends in VTE Rates
QI intervention
Will collaborative quality improvement reduce costs, or just save lives?
Inferior vena cava (IVC) Filters
Aim to prevent fatal pulmonary embolism after surgery
Used commonly in bariatric surgery (10% in Michigan)
Effectiveness unclear
Total BCBSM payments with gastric bypass (2006)
$32,008
$45,559
Variation in the use of IVC filters before gastric bypass
Low use hospitals
High use hospitals
Complications in gastric bypass patients with and without IVC filters
OR=1.3 (0.5-3.2)
OR=1.4 (0.9-2.2)
OR=2.5 (1.0-6.3)
Birkmeyer NJO et al., Ann Surg, July 2010
Over half of deaths and permanent disability directly attributable to the filter itself
Trends in the Use of IVC Filters
Q1 20
07
Q3 20
07
Q1 20
08
Q3 20
08
Q1 20
09
Q3 20
09
Q1 20
10
Q3 20
10
Q1 20
11
Q3 20
110
2
4
6
8
10
Time Period
Perc
en
t
IVC filter intervention
Data first presented at Collaborative Meeting
Net savings to BCBSM
$2.6 million / yr.
The Payor CommunityImprove Access to Care by giving VALUE (COST/QUALITY) back to payors
Quality Events at ASMBS Annual Meeting
Sunday, June 17 ASMBS Presents Quality Improvement Workshop, Nuts and Bolts
(free to all registered attendees)
Monday, June 18 National Bariatric Surgery Collaborative; The Next Level of
Excellence
Wednesday, June 19 ASMBS Town Hall Meeting; New Direction for ASMBS
Thursday, June 20 Quality in Bariatric Surgery Mason Lecture, John Birkmeyer, MD
Friday, June 21 ASMBS National Collaborative Process Improvement Initiative
Thank you