Program Level Performance Measurement: HMIS and Measures that Matter Tom Albanese, Abt Associates Inc. September 23, 2008.

Program Level Performance Measurement:

HMIS and Measures that Matter

Tom Albanese, Abt Associates Inc.

September 23, 2008

2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development2

Learning Objectives

1. Learn the fundamentals of performance measurement.

2. Learn how to develop performance measures and use HMIS data to calculate relevant program outcome measures.

3. Learn how to interpret performance measures and apply them to improve (future) program operations and services.


Why HUD is Interested in Performance Measurement

Accountable to Congress, OMB, the White House, and taxpayers

HUD’s overall objective: to “end chronic homelessness and to move homeless families and individuals to permanent housing”

At least 71.5% of formerly homeless individuals will remain housed in HUD permanent housing projects for at least 6 months

At least 63.5% of homeless persons will move from HUD transitional housing into permanent housing

At least 19% of persons exiting HUD homeless assistance projects will be employed

CoC’s accountable for achieving & documenting results

Performance Measurement


Performance Measurement Fundamentals


What is Performance Measurement?

Performance measurement is a process that systematically evaluates whether your efforts are making an

impact on the clients you are serving.


Multiple Levels of Performance Measurement

Local Service Provider

1. Program Level APR or Other Performance

Measurement Tool

CoC

2. CoC/System LevelCoC Application or Other Performance Measurement Tool

3. State Level State-Specific Performance

Measurement Tool

4. National Level PART Reviews


Purpose of Program Measurement

• Are actions achieving intended goals?– What is the purpose of your program? Goals?– Does the program work?– If yes, what makes it work?– If no, what component doesn’t work and how can

it be fixed to make it work?

• Note that program goals may only relate to certain types of clients or components of the program (e.g., different goals for severely disabled persons)


Performance Measurement Building Blocks

• Inputs include resources dedicated to, or consumed by, the program—e.g., money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies.

• Activities are what the program does with the inputs to fulfill its mission, such as providing shelter, feeding the homeless, or providing job training.

• Outputs are the direct products of program activities. They usually are presented in terms of the volume of work accomplished—e.g., number of participants served and the number of service engagements.

• Outcomes are benefits or changes among clients during or after participating in program activities. Outcomes may relate to change in client knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.


Group Exercise 1:Outcome, Output, or Neither?

1. 1 new HMIS program administrator will be hired.

2. 150 clients will receive prevention services.

3. 90% of persons who exit the program will move to permanent housing.

4. 75% of program staff will be trained in crisis management techniques.

5. 40% of clients will receive one or more mainstream public benefits.

Examples Answers

Input

Output

Outcome

Output

Outcome


Achieving Your Outcomes

• Short-term outcomes: What change will the client experience within a month of his/her involvement in the program? How will you measure this?

• Intermediate outcomes: What change will the client experience within a year of being involved in the program? How will you measure this?

• Long-term outcomes (i.e., Impact): What is the long-term (e.g., 3-year) impact of the program on clients? Has it been sustained? How will it be measured?

Achieving your outcomes can be a progression. Monitoring this progression requires data that are

collected at different intervals:


Example: Single Adult Shelter System’s Impact Over Time

75% of clients will have a plan to obtain and maintain stable housing.

50% of clients who exit will obtain stable housing.

85% of clients who obtain stable housing will not return to shelter within 12 months (43% of all exits).

400 single adults will receive shelter and access to services.

350 clients will receive individualized housing information and referral assistance.

180 clients will receive weekly check-up calls and problem solving assistance, as needed, after exiting to stable housing.

Short-term

Long-term

Outputs Outcomes


Framework for Converting Program Goals into Outcomes

Who is the base for measuring

results?

Step 1

What do you hope to achieve

with this population?

Step 2

Within the base, how many persons achieved it?

Step 3

Who is the base population for measuring results?

Within the base, how many persons achieved it?

Step 4

Outcome (%)


Converting Program Goals into Outcomes: Example

Program Goal: Exiting clients into “stable” housing

Persons who exited during the past year

(N= 40)

Step 1

Obtain stable housing

Step 2

10 persons obtained stable housing

Step 3

40

10

Step 4

25% obtainedstable housing at exit


Look Out for Ambiguous Concepts

For example, what do we mean by…

Developing and measuring performance outcomes often invites ambiguous concepts into the process.

• …obtaining stable housing?

• …obtaining employment?• …increasing income?• …accessing services?• …becoming more self-sufficient?

Which data elements and responses will count?


Key Underlying Assumptions

• Staff are collecting the required information at entry and exit for each client.

• Staff are recording this information into the HMIS in a timely fashion (e.g., 4th business day of every month for prior month).

• The target population is clearly defined.

• The program goals are clearly defined for the target population.

Outcomes can be calculated consistently for all clients if, and only if:


Developing Performance Measures: Key Elements

• For each measure (output or outcome):a) Purpose: identifies the reason for the measure

b) Application: identifies the programs and/or system for which a measure applies

c) Definition: definition for the measure

d) Goal-Setting Methodology: used by program or system to set goals around the measure

e) Reporting Methodology: detailed description of how measure is calculated

• Use HMIS data elements• Develop and test before implementation


Example: Successful Housing Outcome

a) Purpose: Indicates program’s success in ending homelessness by linking households to appropriate next step housing which includes transitional and permanent housing. A higher number and rate are considered positive.

b) Programs: All

c) Definition: the number of distinct households that exit (i.e., latest exit for households with multiple stays during report period) to successful housing as defined in Table X and the percentage this represents of total distinct households exited during the report period.

d) Goal-setting methodology: Meet or exceed CoC goal or prior performance. Multiply the percentage goal by the projected number of exited households (i.e., latest exit for households with multiple stays during the report period).

e) Reporting methodology: Calculate the total number of distinct household exits and the total number of destinations that are considered successful housing outcomes. Divide the number of successful housing outcomes by the number of total exits during the report period.


Considerations When Developing Program Measurement Methodology

Specify all parts of query/calculation:• Programs

– Across a program type (e.g., all outreach, all PSH programs)– All programs– Programs that serve a specific subpopulation

• Clients (de-duplicated across programs)– All Exiters– All participants– All currently enrolled– Those who stayed > 6 months

• Data Elements– Same HMIS elements as program measures– Need rules to reconcile data across programs if client was enrolled in more

than one program

• Timeframe– Define date range– Specify date field (entry date, exit date, placement date, assessment date,

etc.)


Wherever Possible Use HMIS to Define Your Concepts

• Emergency shelter • Apartment or house that you own

Based on the Destination HMIS data element, we can define stable housing (narrowly) using the following response categories:

• Transitional housing

• Permanent housing

• Substance abuse facility or detox center

• Hospital (non-psychiatric)

• Jail, prison, or juvenile detention center

• Room, apartment, or house that you rent

• Don’t Know

• Staying/living with family

• Staying/living with friends

• Hotel or motel voucher paid for without ES voucher

• Place not meant for human habitation

• Other

• Foster care home or group home

• Refused


Group Exercise 2

Employment Program

The goals of the program are to help unemployed clients obtain employment and help employed clients get “better” jobs. During the past year, the program served 6 (unduplicated) persons:

Client ID

Entry Date Exit Date Employment Entry

Employment Exit

1 1/31/07 9/15/07 Unemployed Employed

2 3/15/07 6/28/07 Unemployed Unemployed

3 7/11/07 -- Unemployed --

4 7/7/07 9/18/07 Employed Same Employment

5 8/2/06 5/12/07 Employed Higher Paying Job



Define the Base Population for Each Goal

Client ID

Entry Date Exit Date Employment Entry

Employment Exit


2 3/15/07 6/28/07 Unemployed Unemployed

3 7/11/07 -- Unemployed --

4 7/7/07 9/18/07 Employed Same Employment

5 8/2/06 5/12/07 Employed Higher Paying Job


Goal 1: Achieve employment at exit

Goal 2: Improve employment income at exitIs everyone part of the target population? e.g., Do you expect to calculate an outcome for everyone?


Calculate the Outcome for Goal 1

All unemployed persons at entry who

exited (N = 3)

Step 1

Achieve employment

Step 2

2 persons achieved employment

Step 3

3

2

Step 4

67% achieved employment

Program Goal 1: Obtain Employment at Exit


Calculate the Outcome for Goal 2

Persons who were employed at entry and exited (N = 2)

Step 1

Improve employment

income

Step 2

1 person increased earnings

Step 3

2

1

Step 4

50% improved income

Program Goal 2: Improve Employment Income at Exit


Exercise 2 and the Performance Measurement Process

Inputs

Money: $250,000Staff: 4 FTEs

1 Facility

ActivitiesJob Training ClassesInterview Assistance

Job Placement Services

Outputs6 enrolled in

weekly services6 employment

assessmentsReferred to av. 4

jobs each

Outcomes67% achieved empl.

50% improved income


Using Outcomes to Inform Future Program Operations

Step 1

Review outcomes

with program managers

Develop action steps

and timelines

Step 2

Implement action steps

Step 3

Regular monitoring

Step 4


Step 1: Reviewing Outcomes with Managers

• Program director and managers should review outcomes collaboratively to understand what the outcomes are suggesting.

• Break down the outcomes to understand the underlying forces:

What’s Going On?

− What are we doing right? What activities contributed to our ability to meet/exceed our benchmarks?

− Where do we need to improve? What activities fell short of producing the desired outcomes?

− What else might be contributing to our outcomes? How can we influence or mitigate these external forces to further our positive outcomes?


Step 2: Developing Action Steps and Timelines

Reinforcing the Good and Adjusting the Bad

• Outcomes that were achieved/exceeded: Continue to support the activities that led to positive performance.

• Outcomes that were not achieved: Allocate inputs differently to support different types/levels of activities.

• Set target dates for reviewing all outcomes—e.g., 3-month intervals.

• Collaborate with other service providers to “control” the external impacts on the program.


Step 3: Implementing the Action Steps

Getting Buy-In Through Information Sharing

• You can’t implement what you don’t understand: program directors, managers and front-line staff must understand the reasons for making changes in program operations.

• Information sharing promotes the idea that “we are all in this together.”

• Information sharing is fluid: program directors, managers and front-line staff can learn from one another; it’s not a one-way (top-down) process.


Step 4: Regular Monitoring

It’s Easier to Adjust Program Operations Incrementally than Wholesale

• Monitor progress by generating performance outcomes at different periods of time—e.g., 3-month intervals.

• Adjust approach as needed, but usually incrementally.

• Important to acknowledge that clients’ needs may shift, and thus program goals and approach may also need to shift.


Comparing Program Results

• Compare results between programs, sites, teams, etc. to see which are working best and which are working least well– With limited dollars, fund those that are most

effective– Use results to identify best practices and practices

that need TA– Use results from multiple programs/sites/teams to

help set a program/agency/community expectation or standard of performance


Apples to Apples: Risk Adjustment

• Problem: Comparing program results can encourage programs to “cream” to ensure strong results

• Solution: Risk adjustment allows you to account for differences in client populations when comparing results across or within programsResults can be adjusted on the basis of…– Client characteristics, such as demographics, family

size, disability– Client history, such as past eviction, criminal

background, – Client functionality or level of engagement/commitment

to change


Risk Adjustment Requires Expertise

• To adjust for client differences, programs need to collect consistent data on clients to use during analysis of program results– Agree on these standards beforehand

• Develop an analysis plan for how you intend to adjust for client differences– Engage a researcher to help develop the plan

• Even if results aren’t formally adjusted, acknowledge that different programs may have different outcome expectations based on differences in clients targeted and/or served


Case Study: Columbus, Ohio Performance Measures

Prevention Outreach Emergency Shelter

Resource Specialists

Rapid

Re-Housing

Permanent Supportive

Housing

Number Served Successful Housing Outcomes Average Length of Stay Average Length of Participation

Recidivism Successful Income Outcomes

(Tier II shelter)

Direct Client Assistance Utilization

Occupancy (Tier II

shelter)

Housing Stability Housing Retention


Some notes of caution…

• There is more to performance measurement than conducting the analysis– Educate, train, obtain buy-in

• Be careful about how data is interpreted and used1. Jump in, but don’t be careless in how results are

used2. Look at the results within the context of all the

outputs, interim measures and impact measures to validate the interpretation that’s being made

3. Vet the results before publicly releasing anything4. Appropriately caveat the limitations of the data and

analysis


Additional Resources

• What Gets Measured Gets Done: A Toolkit on Performance Measurement for Ending Homelessness, National Alliance to End Homelessness, www.endhomelessness.org

• The Community Partnership for the Prevention of Homelessness (Washington, DC) www.community-partnership.org

• Community Shelter Board (Columbus, OH) www.csb.org

http://www.endhomelessness.org/

http://www.community-partnership.org/

http://www.csb.org/

Questions & Discussion

Program Level Performance Measurement: HMIS and Measures that Matter Tom Albanese, Abt Associates Inc. September 23, 2008.

Documents

department of housing

program work

program goals

hmis data

performance measures

hud transitional housing

hud permanent housing

job training