Using Program Data to Improve APS Performance Or How I Learned to Be A Geek and Enjoy It Karl Urban Director of Performance and Policy Development Texas.

Using Program Data to Improve APS Performance

Or How I Learned to Be A Geek and Enjoy It

Karl Urban

Director of Performance and Policy Development

Texas Adult Protective Services

What are the Objectives of This Presentation?

• Participants will learn about quantitative and qualitative measures of APS performance.

• Participants will learn how to use APS program data for employee accountability and performance improvement.

• Participants will learn about the technological support systems for using APS program data.

What Will You Get Out of This Presentation?

• This session will: – provide a review and discussion of data for measuring

APS performance; – explain how employees and managers are

encouraged and empowered to use data at the individual, unit and program level; and

– demonstrate the technology systems and tools for accessing and using data.

• Participants will not become geeks but they may just become better managers and decision-makers.

Why Am I Qualified to Talk About This?

• I am a policy wonk by education and disposition

• My prior experience with performance management using data– Texas Performance Based Planning and

Budgeting System– State Unit on Aging/AoA/GPRA

• My current job in Texas

Why Is This Topic Important for APS Programs?

• The GAO just recommended that the federal government explore collecting performance information from APS systems.

• Our experience in Texas suggests APS programs are a potentially data rich environment that can be used for accountability and program improvement.

What Are My Biases on Managing With Data?

• It is a means, not the ends.

• Data helps you to ask the right questions; rarely does it give definitive answers to policy or operational questions.

• The rewards of managing by data are usually worth the effort, but you need to pay attention to potential unintended consequences.

What is the Status of APS Programs Today?

• “It was the best of times; it was the worst of times”– Excitement – and a little apprehension – around EJA

and attendant national focus• How do we help the federal government know what is

important to measure? How do you actually do it across disparate state and county-based systems?

– Most state and local programs are hurting financially• How do we use performance information to help do more

with less?

• What else do you think is critical in the APS world these days?

…and Jerry Jones is actually from Arkansas

• Texas is really, really big, very culturally diverse, and an extremely red state.

10

• APS investigates allegations of abuse, neglect and exploitation of vulnerable adults in the community age 65 or older or people with disabilities age 18-64.– APS clients live in their own home, family homes or unlicensed

congregate settings such as personal care homes or room and boards.– APS conducted 82,802 in-home investigations in FY 2010.

• If needed, APS arranges for or pays for services to alleviate abuse and neglect, such as respite, and emergency assistance with food and shelter. – APS arranged or paid for services in 42,926 cases in FY 2010.

• APS also makes referrals to law enforcement and to guardianship programs.• Employs over 800 direct delivery staff with a total budget of almost $45

million

• APS is the safety net for the safety net in Texas

What Does the Texas APS’ In-home Program Do?

What Does It Mean to Be the Safety Net for the Safety Net?

Mental Health Neglect9.7%

Physical Abuse2.6%

Physical Neglect61.8%

Medical Neglect18.2%

Exploitation2.4%

Sexual Abuse0.1%

Emotional Abuse4.4%

Suicidal Threat0.7%

FY 2010 Cases By Type of ANE

12

What Do Texas APS’ MH&IDD Investigations Do?

• APS investigates abuse, neglect, and exploitation of clients receiving services in the following state-operated and/or contracted settings: – State Supported Living Centers– State Hospitals– State Centers– Community mental health/mental retardation centers – Privately-operated Intermediate Care Facilities for Persons with

Mental Retardation – Mental retardation waiver providers

• APS completed 9,922 MH&IDD related investigations in FY 10.

• Employs about 150 staff with a total budget slightly over $8 million

What Does the APS Program Look Like from a Metric Perspective?

14

Department of Family and Protective ServicesAdult Protective Services

15


16


17


18


19


20


21


Texas has a robust system of using qualitative (case reading) and quantitative (performance)

data for employee accountability and program improvement.

So?

What is Qualitative Data?

• Based on case reading by 7 quality assurance specialists– 2 (closed) cases per worker per month– Scored against standards based on policy

• Examples:– Was the case initiated in accordance with policy?– Was the client assessment tool completed appropriately?– Did the investigator collect all the necessary evidence?

• Rolls up into 3 scales– Investigation: how good was the investigation?– Outcome: was the client protected and services delivered?– Process: did we cross our Is and dot our Ts?

• Rolls up into worker, unit, region, statewide data

What is Quantitative Data?

• Based on documentation in case management system by workers

• Extensive system of available reports from MRS (Management Reporting System)

• Examples:– Timeliness of documentation, face-to-face contacts– Case durations

• There are multiple levels of reports: – Twelve month summaries that are used for evaluation purposes

and – Monthly reports that provide a case detail list for the months

summarized on the twelve timeliness reports– There are also quarterly, weekly and even daily reports

How Does APS Use Program Data?

• Provide data to inform employee and program accountability.

• Provide data for managing and enhancing the program.

• Provide data to assess and enhance quality of casework and investigations.

How Does APS Use Data for Employee Accountability?

Provide data to inform employee and program accountability.

• Achieve client-centered outcomes (intake, investigations, risk assessment, delivery of services).

• Inform job performance standards.• Establish method of accountability, including

awareness and procedures for monitoring.• Provide information for conducting periodic

employee performance reviews.

Task Performance Standard Data Source

Task 1: Case Initiation (Quantitative)

Caseworker will initiate each case within 24 hours.

Average for Timeliness in INV Stage Closed Investigations - 24 Contact during the review period(5) Distinguished = Above 98% (4) Commendable = 96%-98%(3) Competent = 94%-95%(2) Needs Improvement = 92%-93%(1) Needs Major Improvement = Below 92%

(See Data Warehouse Report -> APS -> Employee Development -> INV Timeliness in INV Stages Closed (12 Month) - INV APS 11 % 24 Contact Met)

Task 2: Investigation (Quantitative)

Caseworker will make face-to-face contacts within policy timeframes for the priority.

Average for Timeliness in INV Stage Closed Investigations - Face to Face Contact during the review period(5) Distinguished = Above 94% (4) 92%-94% = Commendable(3) Competent = 89%-91%(2) Needs Improvement = 80%-88%(1) Needs Major Improvement = Below 80%

(See Data Warehouse Report -> APS -> Employee Development -> INV Timeliness in INV Stages Closed (12 Month) - INV APS 11 % FTF Contact Met)

Task 4: Investigation (Qualitative)

Caseworker will fully investigate allegations and all factors that present a threat to the client's health or safety.

Investigation Scale average for the sample of cases read during the review period.(5) Distinguished = Above 93% (4) Commendable = 87%-93% (3) Competent = 80%-86%(2) Needs Improvement = 73%-79%(1) Needs Major Improvement = Below 73%

(See APS Case Reading System -> Reports -> Three Scales Report -> Investigation Scale Investigation Scale Score - %)

Task 6: Service Delivery (Qualitative)

Caseworker will work to alleviate the client's state of abuse/ neglect/ exploitation and remove threats to client's health and safety by designing and implementing, a service plan that addresses problems identified during the investigation.

Client Outcome Scale average for the sample of cases read during the review period.(5) Distinguished = Above 93% (4) Commendable = 87%-93% (3) Competent = 80%-86%(2) Needs Improvement = 73%-79%(1) Needs Major Improvement = Below 73%

(See APS Case Reading System -> Reports -> Three Scales Report -> Client Outcome Scale Client Outcome Scale Score - %)

Task 7: Process Compliance (Qualitative)

Caseworker will meet deadlines, perform consultations and notifications, and document the casework according to policy.

Process Compliance Scale average for the sample of cases read during the review period.(5) Distinguished = Above 93% (4) Commendable = 87%-93% (3) Competent = 80%-86%(2) Needs Improvement = 73%-79%(1) Needs Major Improvement = Below 73%

(See APS Case Reading System -> Reports -> Three Scales Report -> Process Compliance Scale Process Compliance Scale Score - %)

How Does APS Use Data for Managing and Enhancing the Program?

• Systemic/periodic data collection (management reporting system/tools).

• Enhance employee performance.• Measure the operationalization of

certain policies.• Regional/unit/worker development.

Region Investi- gation

Out-come

ProcessCompli-

ance

*Case Initiation

*FTF Contact

*Monthly Status

Contact

Doc. Time. ****

Inv Duration

Mean

SD Duration

Mean

Sum INV + SD Duration

ValidationRate

Supervisor Rejection

Rate

Rapid Closure

Rate

% Inv Progress

Svc

*** Caseload Aver Daily

1 94% 98% 93% 97% 95% 91% 87% 32 51 83 83% 13% 6% 73% 332 94% 99% 96% 97% 97% 92% 96% 21 29 50 78% 6% 10% 56% 233 90% 94% 90% 95% 89% 87% 87% 45 49 94 65% 13% 15% 34% 414 93% 96% 92% 97% 95% 90% 86% 38 47 85 76% 3% 9% 61% 315 89% 94% 91% 94% 90% 80% 76% 42 51 93 77% 8% 11% 58% 396 89% 93% 89% 96% 91% 77% 83% 36 47 83 72% 8% 15% 49% 307 93% 96% 93% 94% 95% 85% 90% 41 49 90 59% 13% 14% 37% 368 84% 87% 84% 90% 87% 63% 70% 49 81 130 64% 5% 12% 47% 439 95% 98% 95% 97% 95% 89% 93% 30 37 67 80% 12% 9% 61% 2410 89% 92% 88% 91% 92% 84% 79% 53 51 104 71% 11% 11% 48% 5111 94% 96% 94% 98% 95% 91% 87% 45 52 97 68% 6% 11% 53% 37

State 91% 94% 91% 95% 92% 76% 85% 40 51 91 70% 9% 12% 49% 36

State FY10 91% 94% 91% 95% 92% 76% 85% 40 51 91 70% 9% 12% 49% 36State FY09 91% 92% 88% 95% 91% 71% 80% 43 51 94 73% 11% 12% 52% 32State FY08 93% 91% 86% 94% 90% 70% NA 44 54 98 73% 11% 11% 51% 30State FY07 91% 89% 85% 94% 87% 58% NA 47 68 115 74% 11%** 12% 51% 36State FY06 90% 90% 88% 95% 84% 52% NA 52 67 118 75% 12% 52% 51

Casereading Database HHSC Data Data Warehouse

FY 2010 In-Home Program Performance Dashboard

In-home Performance Summary: Region 3 Compared to State Measure 2006 2007 2008 2009 2010

Number of Intakes Region 13,605 13,016 14,307 15,817 18,888 State 82,029 77,238 83,605 89,489 103,401 Average Investigation Duration (Days) Region 65 62 60 52 51 State 58 52 49 48 45 Averaged Service Delivery Duration (Days) Region 74 85 59 45 49 State 67 71 54 51 51 Timeliness of Documentation Rate (%) Region NA NA 79% 84% 87% State NA NA 83% 83% 84% Average % of Investigations Pending 60+ (Days)

Region 30 31 28 22 23 State 27 23 23 23 19 Rapid Closure Rate Region 14% 15% 14% 14% 15% State 11% 12% 12% 12% 12% Monthly Status Contacts Rate Region 47% 55% 64% 72% 87% State 48% 58% 70% 71% 82% Face-to-Face Initiation Rate Region 80% 84% 87% 87% 89% State 95% 94% 94% 95% 92% Overall Case Reading Scores Investigation 3 Scale Region 88% 89% 93% 90% 90% State 90% 91% 93% 91% 91% Outcome 3 Scale Region 89% 87% 90% 91% 94% State 90% 89% 91% 92% 94%

Monthly Summary of Pending APS State Supported Living Center Investigations

DATEUnder

10 Days

10 Days or

More

10+ Days with Extension

10+ Days without Extension

Percent 10+ Days

Percent 10+ Days without

Extension

1-May-2011Sunday

2-May-2011 22 23 22 1 51.11% 2.22%

3-May-2011 20 23 23 0 53.49% 0.00%

4-May-2011 25 23 22 1 47.92% 2.08%

5-May-2011 24 23 22 1 48.94% 2.13%

6-May-2011 26 24 24 0 48.00% 0.00%

How does APS Use Data to Assess and Enhance Quality of Casework

and Investigations?

• Appropriate notifications to law enforcement and/or other entities.

• Capture critical components of casework.• Establish clear investigation performance standards. • Ensure compliance with documentation

requirements.• Establish process for reviewing staff decisions.• Reporting to external and/or internal Stakeholders.

Where Does All This Data Come From?

• The intranet, of course– Case reading database– MRS data warehouse

• Accessible to:– EVERYONE!

For Discussion:

What are the consequences - positive and negative –

of using data to drive

accountability and performance?

What are the consequences - positive and negative - of using data to drive accountability and performance?

• Provide data to inform employee and program accountability

• Objective, 3rd party, consistent, measures compliance with the most important policy

• Valuable resource for conversations between supervisors and employees about employee’s development

• When used in evaluations, QA data is perceived as punitive, not

constructive• Case reading sample size is small at employee level• What should determine employee performance: subjective,

inconsistent supervisors or biased, inconsistent case readers?


• Provide data for managing and enhancing the program

• “What gets measured gets moved”

• “All management cares about is ‘the numbers’”


• Provide data to assess and enhance quality of casework and investigations.

• Zero in on cause and effect by disaggregating and examining

• Track and trend key issues

• Data must be accurate and reliable (“faceplate game”)

Karl Urban

Director of Performance and Policy Development

Texas Adult Protective Services

[email protected]

mailto:[email protected]

Using Program Data to Improve APS Performance Or How I Learned to Be A Geek and Enjoy It Karl Urban Director of Performance and Policy Development Texas.

Documents

aps program data

aps systems

aps world

aps clients

texas slide

status of aps programs

discussion of data

performance improvement