Performance Measures and CQI – Essential Public Health Tools Jonathan E. Fielding, M.D., M.P.H., M.B.A Director of Public Health and Health Officer L.A.

Performance Measures and CQI – Essential Public Health Tools

Jonathan E. Fielding, M.D., M.P.H., M.B.ADirector of Public Health and Health Officer

L.A. County Department of Public Health

“Using Performance and Outcome to Improve Treatment” Summit

Los Angeles, CA – March 20, 2008

Los Angeles County – Background

• 4,300 square miles• 89 incorporated cities and 2 islands• 9.9 million residents (more than 42 States)• 46% Latino, 32% White, 13% Asian/Pacific

Islander, 10% African American, 0.3% American Indian

• Over 100 different languages spoken by significant size populations

• 15% living in poverty (14% of families & 24% <18)

• 22% of adults & 8% of children have no health insurance

Los Angeles CountyDepartment of Public Health

• Vision– Healthy People in Healthy Communities

• Mission: – To protect health, prevent disease, and promote

health and well-being

• Public Health stats:– Annual budget of over $750 million– 4,000 employees– Created as a separate department in 2006

The Core Functions &Essential Services of Public Health

Three Major Public Health Responsibilities

• Health Protection– Emergency Preparedness and Response e.g. biological, radiological

terrorism; emerging infections –flu, SARS etc.– Assuring conditions to protect health (e.g., apartment and food

facility inspections, environmental health programs)– Licensure of hospitals/long term care facilities

• Disease and Injury Prevention and Control – HIV/AIDS Prevention– Acute Communicable Disease Control – TB and STD Control Programs – Injury and Violence Prevention

• Health Promotion– Maternal, Adolescent & Child Health– Chronic Disease Control –Tobacco Control, Physical Activity &

Nutrition Programs & Obesity Work Group)– Alcohol and Drug Programs– Promote policies that support our mission

Los Angeles County Public Health Department Programs

• Acute Communicable Disease Control • AIDS Programs • Alcohol & Drug Program• Children's Medical Services • Children's Planning Council • Communicable Disease Control and Prevention • Data Collection & Analysis Unit • DPH Communications Office • Environmental Health • Epidemiology Unit • Health Assessment Unit • Health Education Administration • HIV Epidemiology Unit • Immunization Program • Injury & Violence Prevention Program • Lead Program • Maternal, Child, and Adolescent Health • Nutrition Program • Office of Planning, Evaluation and Development • Oral Health Program • Organizational Development & Training • Physical Activity Program

• Policies for Livable, Active Communities and Environments (PLACE)

• Education in Medicine • Laboratory • Nursing • Policy • Quality Assurance and Professional Standards • Sexually Transmitted Disease Program • SPA'S in Your Community • SPA1 - Antelope Valley Area Health Office • SPA2 - San Fernando Area Health Office • SPA3 - San Gabriel Area Health Office • SPA4 - Metro Area Health Office • SPA5 - West Area Health Office • SPA6 - South Area Health Office • SPA7 - East Area Health Office • SPA8 - South Bay - Long Beach Area Health

Office • Tobacco Program • Toxics Epidemiology • Tuberculosis Control Program • Veterinary Public Health • Women's Health Office

Why We Try to Continuously Improve Quality of Services Offered

• There are demands for accountability for usage of County funds– From the Board of Supervisors and CEO– From County residents/taxpayers– From consumers

• Want the best results for our limited funding– Need to look at return on investment (ROI), both in

terms of $ and in terms of health– Need to ensure that we are using evidence-based

best practices

Continuous Quality Improvement Process

The PDCA (or PDSA) Cycle was originally conceived by Walter Shewhart in 1930's, and later adopted by W. Edwards Deming. The model provides a framework for the improvement of a process or system.

UltimateProcessStructure

OutcomesIntermediate

PH ProgramPublic Health Measures Initiative

LAC DPHPublic Health

Report Card, Goals and Strategic Plan

Key Indicators ofPublic Health

LA CountyPerformance Counts!

Population IndicatorsPerformance Measures

IndicatorsOperational Measures

DPH Framework for Development ofQuality Indicators

Objective Area 3: Resource Utilization

9. Percent of programs that have had one or more staff complete leadership training q

Percent of Program Directors who have completed leadership training

10. Percent of programs whose employees’ Performance Evaluations were completed on-time r

11. Percent of employees who have completed “Core Functions of Public Health” training s

Percent of programs that have had one or more staff complete “Core Functions of Public Health” training

12. Percent of employees who participated in at least one emergency preparedness training, drill, or exercise during 2004.- All Employees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- Employees in targeted PH programs . . . . . . . . . . . . . . . . . . . . .- Physicians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- Nurses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- Epidemiologists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- Others . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

See Notes on reverse.

Report Card2006-2007

2005

-06

Res

ult

2004

Ben

chm

ark

X% 100%

X% 100%

X% 80%

X% 80%

100%X%

100%X%

100%X%

100%X%

100%X%

100%N/A

90%N/A

95%X%

95%X%

X% 100%

X% 90%

75%N/D

25%X%

100%X%

N/A – Not applicable as this is a new measure for 2004.N/D – No data available for 2003.

Lon

g-T

erm

Goa

l

100%

100%

100%

100%

100%

100%

100%

100%

100%

90%

100%

100%

100%

100%

100%

100%

100%

100%

75%N/A 100%

X% 95%

X% 75%

95%

90%

N/A

N/A

N/A

N/A

N/A

N/A

25%

N/A

50%

50%

50%

50%

X%

X%

X%

X%

X%

X%

2006

-07

RE

SUL

T

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

X%

Objective Area 1: Use of Data and Evidence to Improve Quality

1. Percent of programs that use population-based data to guide their planning and monitoring activities.a

2. Percent of programs that have approved:- Mission and Vision statements b

- Population goals and indicators c

- Performance goals and performance measures d

3. Evidence-based interventions: - Percent of program directors/management staff that have participated in training in evidence-based Public Health e

- Percent of programs with documentation of a systematic review of literature and prioritized effective interventions f

- Percent of programs with documentation that current interventions and practices are based upon the best available evidence g

4. Proportion of targetedh Public Health units that are connected to the VCMR (electronic disease-reporting) i

Proportion of targeted Public Health units that are currently using data from the VCMR j

Objective Area 2: Communication, Planning, and Technology

5. Percent of CORE programs that have completed Phase I and Phase II of the SPA improvement process k

6. Effective collaboration within Public Health:- Percent of programs that have developed an action plan l

- Percent with significant progress in implementation m

7. Percent of programs that have a website accessible to the public through www.lapublichealth.org n

Percent of programs whose directors have verified that their website is current o

8. Percent of programs that have:- Completed “Service During Tough Times” training- Developed all “Service Scripts” to assure consistency of information p

2005

-06

Res

ult

2004

Ben

chm

ark

Lon

g-T

erm

Goa

l

2006

-07

RE

SUL

T

This is an example of our performance-based report card that we use to evaluate our programs in DPH. We also participate in the County’s “Performance Counts” program.

Steps in the Linkage of Program Goals with Performance Measures

• Identify Overall Program Goals (in Public Health these are Population Goals)

• Review of the Evidence Base to Identify Effective Strategies

• Identify Key Roles and Performance Goals • Develop Performance Measures that measure

the Quality of the work that you do

Example: STD Clinical Services Population Goal: Reduce incidence of selected communicable diseases

(including Sexually Transmitted Diseases).

Effective, Evidence-Based Strategies: (STD Treatment Guidelines)1. Follow-up of reported cases and contacts to assure treatment- Screening, diagnosis, and treatment of targeted high risk groups- Education of index case and contacts to lower rates of repeat and new infections- Timely, accurate, and complete reporting - Population-specific STD prevention and intervention programs- Targeted outreach programs and events Program Roles:Investigate, assure treatment, and prevent transmission of communicable diseases by: 1. providing clinical care in PH clinics2. case management3. coordination with and provision of consultation to the private medical sector4. contact investigation

Performance Measures:1. Percent of assigned GC cases interviewed for contacts 2. Percent of of identified contacts to GC that were treated3. Percent of assigned chlamydia cases interviewed for contacts4. Percent of identified contacts to chlamydia that were treated5. Average number of patients seen per clinic session (Clinic Efficiency Measure)6. Rate of broken appoints (Clinic Efficiency Measure)

Example: Immunization Program Population Goal: To reduce morbidity and mortality from vaccine-preventable diseases by

improving immunization levels.

Effective, Evidence-Based Strategies: (from the Community Guide)1. Client reminder/recall reminder2. Multi-component interventions with education3. Reducing out-of-pocket costs4. Multi-component interventions for expanding access5. Provider-based: Provider reminder/recall6. Provider-based: Assessment and feedback for providers7. Provider-based: Standing orders for adults

Program Roles:1. Conduct surveillance2. Assess immunization coverage levels3. Support and monitor providers of immunization services through contracts, professional education and

training, technical assistance, supplying vaccine and assisting with vaccine management and registry deployment

4. Support bio-terrorism preparedness in Los Angeles County. 5. Collaborate with stakeholders to improve influenza and pneumonia immunization rates within vulnerable

populations (or those 65 years and older).

Performance Measures:1. Percent of children under age 6 years who participate in fully operational population-based registries2. Percent of providers receiving vaccine from LA County Public Health who meet 90% or more of the 18

Standards of Pediatric Immunization as measured by an annual Quality Assurance review3. Number of healthcare providers who receive training on immunization recommendations and VPDs

Design Principles for Measurable Objectives

QUANTITY QUALITY

What difference did the program

make?OUTPUT

How Much Service Did We

Deliver?

How Well Did We Deliver Service?

How Much Effect or

Change Did We Produce?

What Quality of Effect or

Change Did We Produce?

INPUT

Focus on Quality of Outcomes More Than Quantity if Possible

Design Principles for Measurable Objectives Example: Provider Education for Immunization Services

QUANTITY QUALITY

What difference did the program

make?OUTPUT

Number of providers who

received training.

Percent of providers whose

knowledge increased by

20% or more on pre/post-exams

Number of appropriate

immunizations ordered by

trained providers.

Rate of Vaccine-Preventable

Diseases among patients seen by

trained providers.

INPUT

Importance of Performance Measures

• Measures are treatment improvement tools

• Two types of performance measures:• Process measures• Outcome measures

• Tracking program performance is critical

• Current priority to complete design and implementation of performance measures

Four Key Performance/Outcome Measures

1. Engagement — the individual has attended at least one session during the first 30 days and is an ongoing active participant. This measure is considered an indicator of treatment and participant fit, but also program efforts and participant commitments.

2. Retention — identifies 90 days of continuous program participation; the point in time where treatment begin to influence the participants attitude and behavior.

Four Key Performance/Outcome Measures

3. Exit Interview — is a scheduled interaction, preferably face-to-face, between program staff and the participant at the completion of his/her individualized program treatment plan. A key part of this exit interview is the completion of the Los Angeles County Participant Reporting discharge questionnaire.

4. Positive Compliance — refers to participants who completed treatment or left before completing treatment with satisfactory progress. While “completed treatment” is the most successful conclusion, “left before completion with satisfactory progress” is also perceived as a positive outcome.

Los Angeles County Residential Programs

• Program Number and Size – 75 residential programs located throughout Los Angeles County.

Programs range from 12 participant admissions per year to 500-1,500 admissions per year.

• Programs by Gender – 29 (39%) residential programs provide treatment services for men and

women– 27 (36%) residential programs provide treatment services for women

only– 19 (36%) residential programs provide treatment services for men only

• Programs by Race/Ethnicity – 2 (3%) residential programs were predominantly White (> 70%)– 9 (12%) residential programs were predominantly Black/African

American (> 70%)– 6 (8%) residential programs were predominantly Latino/Hispanic

(> 70%)

Residential Programs by Performance Measures: Engagement

N = 75

Mean = 70.6

Median = 71.0

Std. Dev. = 15.4

Skewness = -.823

Kurtosis = 1.846

Minimum = 13.4

Maximum = 100.0

Residential Programs by Performance Measures: Retention

N = 75

Mean = 43.7

Median = 44.4

Std. Dev. = 16.1

Skewness = -.473

Kurtosis = -.250

Minimum = 1.7

Maximum = 77.8

N = 75

Mean = 57.7

Median = 60.0

Std. Dev. = 21.1

Skewness = -.380

Kurtosis = -.213

Minimum = 11.1

Maximum = 99.1

Residential Programs by Performance Measures: Positive Compliance

Residential Programs by Performance Measures: Exit Interview

N = 75

Mean = 81.0

Median = 84.2

Std. Dev. = 17.0

Skewness = -1.587

Kurtosis = 3.105

Minimum = 17.5

Maximum = 100.0

Positive Compliance

Engagement Retention

Exit Interviews

Residential Programs Performance Measures - Fiscal Year 2006-07

Performance Measures

Mean Median Std. Dev. Range

Engagement 71 71% 15% 13-100%

Retention 44 44% 16% 2-78%

Exit Interviews

81 84% 17% 18-100%

Positive Compliance

58 60% 21% 11-99%

Residential Programs Performance Levels

Performance Measures

Actual

Median 2005-06

Actual

Median 2006-07

Estimated

Median 2007-08

Projected

Median 2008-09

Engagement 67% 71% 75% 77%

Retention 42% 44% 47% 49%

Exit Interviews

72% 84% 87% 89%

Positive Compliance

56% 60% 65% 67%

Individual Residential Programs and Performance Measures

76

61

9499

53

2118 18

0

10

20

30

40

50

60

70

80

90

100

Engagement Retention ExitInterviews

PositiveCompliance

Pe

rce

nt

Program A

Program B

Current and Future Efforts to Improve Performance Measures & Implement CQI

• Design and Implement Approved Performance Measures and Standards

• Continued Implementation of Evidence – Based Practices

• Design and Implementation of Performance-Based Contracting

Challenges Facing Substance Abuse Providers and ADPA

• Changing trends in substance abuse– New usage patterns– Increase in dual diagnosis rates?– We need to adapt accordingly

• Financial pressures– Recent budget cuts at state level– Shrinking pool of available NCC (net county cost)

funding– Cuts at federal level

• Need to improve our skills in CQI– Lack of experience and expertise– Lack of common terminology