1 Outcome Measurement: Assessing Clients’ Perspectives of the Impact of Legal Aid Services in Their Lives Legal Aid Services of Oklahoma, Inc. Gary W.

1

Outcome Measurement:

Assessing Clients’ Perspectives of the Impact of

Legal Aid Services in Their Lives

Legal Aid Services of Oklahoma, Inc.Gary W. Dart, Esq. and Denise Caudill, DrPH

2

Outcome Measurement Is …

The regular, systematic tracking of the

extent to which

program participants

experience the benefits or changes intended. United Way, 2000

3

What are Outcomes?• Benefits or changes for program

participants during or after their involvement with a program

United Way, 2000

Examples of Outcomes:• Change in knowledge (understand infant care needs)• Change in skills (increased reading level)• Change in attitude (greater self-esteem)• Change in circumstances (foster care to family)• Change in behavior (drop-outs attend school)• Change in status (unemployed to employed)

Michael Quinn Patton, Utilization Focused Evaluation, 1997

4

Benefits Of OM

% of UW agency executives agree or strongly agree that Outcome Measurement helps them:

• Focus staff on shared goals (88%)• Communicate results to stakeholders (88%)• Clarify program purpose (86%)• Identify effective practices (86%)• Compete for resources (83%)• Enhance record keeping (80%)• Improve service delivery (76%)

5

Yogi Berra

You’ve got to be careful if you don’t know where you’re going ‘cause you might not get there.

6

I. Get Ready1. Assemble and orient an OM work group

Director of Litigation, Managing Attorneys of largest two law offices in state,

Resource Development Specialist; Managing Attorney of the Hotline,

Information System Specialist, Community Ed and Pro Se Coordinator

Support Staff

2. Decide which program(s) to start withWhose outcomes to measure -- all or some clients?

7

II. Choose the Outcomes You Want to Measure

1. Gather ideas for what the program’s outcomes are from a variety of sources

2. Select the outcomes that are important to measure

3. Construct a logic model

8

The Logic Model

9

Outcomes from Whose Perspective?

Client?

Attorney?

Both?

10

Financial resources from grants, contracts and donations to provide staff, operational expenses and equipment. Non-monetary resources such as; donated time from volunteers, donated facilities and partnerships from collaborative efforts. Senior citizens; low-income individuals and families; nonprofit groups and others in Oklahoma who apply and / or become clients of LASO

Legal counseling, brief services, self-help materials, legal clinics, community education and involvement, referrals to community partners and full representation.

# of Full Representation services provided

# of Referrals to community partners

# of Brief services provided

# of Legal Counseling services provided

OUTPUTS

Initial OUTCOMES

IntermediateOUTCOMES

Long-term OUTCOMES

How the System Works: Clients gain knowledge of legal system

How the Clients Fit Into the Legal System: Clients gain knowledge of their legal rights and responsibilities

Clients Learn What To Do or Not Do to Solve Their Legal Problems: Work with an advocate, help themselves through pro se, seek help thru non-legal means (social agencies) or not act at all.

Clients have increased security in achieving and protecting their basic needs, such as food, shelter, income, health care, personal safety, and family relationships.

Clients obtain access to justice system, civil liberties, equal protection and intended benefit of law.

Clients feel empowered to make informed decisions (to act or not to act)

Clients act to obtain resolution.

# of Community Education & Involvement Activities

INPUTS

Legal Aid Services Of Oklahoma - Logic Model

ACTIVITIES

11

III. Specify Indicators

1. Specify one or more indicators for each outcome

2. Decide what factors could influence participant outcomes

12

LEGAL AID SERVICES OF OKLAHOMA OUTCOME MEASUREMENT FRAMEWORK

INITIAL Outcome1

Indicator Questions

How the System Works: Clients gain knowledge of legal system

# & % of clients who understand the legal system

# & % of clients who understand whether or not their problem can be solved by legal system

As a result of services received from Legal Aid do you understand the legal system better?

As a result of services received from Legal Aid do you understand whether or not your problem can be solved by the legal system?

13

INITIAL Outcome2

Indicator Questions

How the Clients Fit Into the Legal System: Clients gain knowledge of their legal rights and responsibilities

# & % of clients who understand rights and responsibilities of client and adverse party

As a result of services received from Legal Aid do you understand your rights?

As a result of services received from Legal Aid do you understand your responsibilities, or what you’re supposed to do?

14

INITIAL Outcome3

Indicator Questions

Clients Learn What To Do or Not Do to Solve Their Legal Problems: Work with an advocate, help themselves through pro se, seek help thru non-legal means (social agencies) or not act at all.

# & % of clients who understand options and merits of those options

As a result of services received from Legal Aid do you understand your options?

As a result of services received from Legal Aid do you understand the pro’s and con’s of those options?

15

INTERMEDIATE Outcomes

Indicator Questions

1. Clients feel empowered to make an informed decision (to act or not to act)

# & % of clients with increased confidence, hope, self-determination

Are you feeling better about the reason for your contact with Legal Aid?

Are you more confident that you will be able to handle problems like this in the future

Did the help you received from Legal Aid reduce your worry and stress?

2. Clients act to obtain legal resolution

# & % of clients who acted or did not act consistent with legal options

Did Legal Aid help you to solve your problem?

Did Legal Aid help you keep the problem from getting worse?

16

LONG-TERM Outcomes Indicator Questions

1. Clients have increased security in achieving and protecting their basic needs, such as food, shelter, income, health care, personal safety, and family relationships.

# & % of clients who feel they have increased stability

As a result of the help from Legal Aid, are you (and/or your family) better off today?

2. Clients obtain access to justice system, civil liberties, equal protection and intended benefit of law.

# & % of clients who feel they obtained access to justice system, civil liberties, equal protection and intended benefit of law.

Do you think your side of the legal problem was heard? Do you think that receiving help from Legal Aid made a difference in whether or not you had access to the legal system?

17

Influencing Factors

LASO level of service -- full or partial service:• Received or didn’t receive materials• Priority of case• Wanted a lawyer to represent them but didn’t get one

Respondents’ mental health: including learned helplessness, locus of control, self confidence, self esteem

Those things that prevent people from being able to do what needs to be done, e.g. transport, child care, money, rural / urban

LASO phone system

Respondents’ desire to please during interview and their understanding of questions

Respondents’ economic, educational and literacy levels

Selection bias introduced by telephone communication, more stable clients instead of transient, moving

18

IV. Prepare to Collect Data on Your Indicators

1. Identify data sources for indicators

2. Design data collection methods

3. Pretest data collection instruments and procedures

19

How to Identify Cases and Clients for OM?

LSC Case Definitions

and Closure

Categories

Tools, Transactions and Representation

20

New Definitions of Service Categories

• TOOLS: Giving a client advice, information, self-help (pro se) materials or forms that the client needs to understand possible options and/or to address their legal problems on their own.

• TRANSACTIONS: Serving a client with non-advocacy legal work that resolves the client’s problem with the drafting of documents, such as; wills, deeds, contracts, DPOA, incorporation, application for tax benefits, advanced medical directives/DNR.

• REPRESENTATION: Advocacy on behalf of a client with or without litigation, such as negotiation with a landlord, appearing in court or administrative hearing.

21

Data Collection Method: Telephone Interviews

Tools 1 (Initial survey) All tools cases closed during previous week

Tools 2 (Intermediate survey) All tools cases closed 5 weeks back

TransactionsAll transactions cases closed 5 weeks back

RepresentationAll representation cases closed 5 weeks back

22

Outcome Measurement Call Sheet OM Study # __________

Service Provided Tools Transactions Representation

Level of Service Fully Served Partially Served

Problem Code: _____Case Closing Code: _____Advocate Code: _____Case Number: _____

Survey Date: ______________

Interviewed client Refused Client not in; answered by other Busy No answer Machine answered Disconnected Other __________________

Interviewer: ________________

Last Name ____________________ First Name _____________________

Phone Number(s) ___________________ ________________________

Call Instructions:

23

Field testing Process

In the Oklahoma City field test, 78 calls were made and 24 interviews were completed, a response rate of 31%.

In the Tulsa Hotline office, 26 interviews were attempted and completed.

OKC Field test # %

Interview 24 .31

Refused 4 .05

No answer 9 .12

Other answered 10 .13

Busy signal 1 .01

Answering machine

18 .23

Disconnected 12 .15

78

.49

24

• Simplify and improve wording of questions• Improve opening statement• Improve format of questionnaires, placement of

questions and response devices• Reinforce procedures, i.e., callers should not

leave messages on answering machines• Add more information to call sheet options• Process for providing feedback to attorneys if

clients indicate need for more information• Identify need to attempt to reach people by

phone in the evenings and/or week-ends

Field testing Revisions

25

• Significant improvements in the questionnaire• LASO staff “buy-in” to the OM process. In one

office, each attorney was asked to interview 10 cases. The staff were surprised by:– the challenge of reaching people during

working hours– some of the more negative responses

Field testing Lessons

Staff gained new perspectives in relating to clients. Since the field testing experience, their supervisor has seen positive changes in attitude toward clients among staff.

26

Field testing Results

• In general, 55% to 65% of respondents in OKC answered most questions positively; Tulsa Hotline respondents ranged between 60% to 75% positive

• Room for improvement in services primarily related to response time and difficulties with the phone system

More Results

from OKC Test

27

As a result of the help from Legal Aid, are you (and/or your family) better off today?

YES NO

Not Sure Somewhat

YES• “2000% better”• “Very much”• “ We have benefits”• “Got help paying family bills”•“Power of attorney”

NO• “Didn’t do anything”

NOT SURE• “Didn’t get a response”

SOMEWHAT• “Problem still up in the air”

OKC: N = 8

28

Did Legal Aid help you keep the problem from getting worse?

YES NO Somewhat

YES• “Got me out of a bad situation, on with my life”• “Got benefits needed to live”• “She was there with me during the hearing”

NO• “Didn’t call” • “Didn’t help or seem to know how”

SOMEWHAT• “It’s going to stay the same”

OKC: N = 7

29

Do you feel the help you received from Legal Aid gave you

access to the justice system?

YES NO Not Sure

YES• “Got me into court with a lawyer”• “He knew the route to take”• “I couldn’t have done it myself”

NO• “I didn’t get to talk with anyone” • “I don’t think they understood”

NOT SURE “Another lawyer done better than they do”

OKC: N =8

30

V. Try Out the Outcome Measurement System

1. Develop a trial strategy

2. Prepare the data collectors

3. Track and collect outcome data

4. Monitor the outcome measurement process

31

VI. Analyze and Report Findings

1. Enter the data and check for errors

2. Tabulate the data

3. Analyze data by key characteristics

32

VII. Improve Outcome Measurement System

1. Review trial-run experience, make necessary adjustments

2. Replicate in all offices statewide.

33

VIII. Use The FindingsEXTERNAL• Retain & increase funding• Enhance public image• Promote program to partners & clientsINTERNAL• Inform staff & board• Improve programs• Guide budgeting & resource allocation• Identify training needsBOTH • Document real differences

program is making in people’s lives

34

Outcome Measurement Training• The Tulsa Area United Way provided Capacity Building

funds for Outcome Measurement Training services to a select group of its member agencies in 2003.

• Legal Aid Services of Oklahoma, Inc. (LASO) was one of 10 agencies to participate in the program. A consultant worked with the LASO team for a period of 110 hours during the year.

• The United Way of Central Oklahoma has committed to provide funding to continue this process with the consultant for an additional 75 hours through 2004

• The United Way logic model approach described in the book, Measuring Program Outcomes: A Practical Approach, is the one being used by LASO.

35

For more information, contact:

Legal Aid Services of Oklahoma Inc.

2915 Classen BlvdOklahoma City, OK 73106

T: 405-557-0200F: 405-524-1257

[email protected]@legalaidok.org

[email protected]

The Outcome Zone™

Training, Consultation, Coaching to Strengthen Organizational

Capacity for Success

Denise Caudill, DrPH2200 Dublin Road

Oklahoma City, OK 73120T: 405-755-6258

405-922-2814F: 405-755-6218

[email protected]