Outcomes Assessment in School Mental Health Webinar and dialogue sponsored by the Quality and Evidence-Based Practice (QEBP) Practice Group and the IDEA.

Outcomes Assessment in School Outcomes Assessment in School Mental HealthMental Health

Webinar and dialogue sponsored by the

Quality and Evidence-Based Practice (QEBP)

Practice Group and the IDEA Partnership

March 16, 2011

Today’s webinar Today’s webinar • Overview of QEBP practice group

• Highlights of QEBP practice group discussions at 2010 CSMH conference (Albuquerque, NM)

• Evaluation of school mental health (SMH) programs in Minnesota Public Schools (Dr. Mark Sander)

• Psychosocial, Placement and Cost Assessment in Maryland’s SMH programs for youth in special education (Dr. Sharon Stephan)

• QEBP updates from the CSMH (Dr. Nicole Evangelista)

• Call for Papers on SMH outcomes assessment - special issue of Advances in School Mental Health Promotion(Dr. Michael Kelly)

QEBP Practice Group Mission and Priorities

• Mission: To provide resources and promote sharing of information across individuals and groups interested in improving the quality of school mental health (SMH) programs and services.

• Priority areas include: • improving dissemination and sharing of evidence-based

practices in SMH; • bridging the research-practice and practice-research gaps

in the field; • and, understanding and promoting the use of the best

student-, program-, and school-level evaluation strategies.

QEBP practice group discussions at 2010 CSMH conference (Albuquerque, NM)

• Split into two groups (research and front line) to answer 3 questions:1) What are our most pressing concerns related to

Quality and Evidence-based Practice?

2) What is one specific action sep for our group (research or front line) this coming year?

3) What is one specific action step you would like to see from the “other” group this coming year?

Next Steps for QEBP Practice Group

• Future webinars and dialogue

Mark Sander, Psy.D., LP

Hennepin County/Minneapolis Public [email protected]

612-668-5489

School Mental Health: Local Evaluation and Research Lessons

Learned

School Mental Health Programs Across MN

Minneapolis Public Schools established in 2005 in five schools

Gradually expanded to 15 schools

County wide – 10 agencies, 14 school districts, over 70 schools

State wide – 21 grantees, serving 63 counties, about 200 school districts, and 550 schools

Collaboratively Building a Evaluation Model

What is “must have” data? o Mental Health Outcome (SDQ and CASII)o Educational Outcome (Suspension and Attendance)o Access and Engagement

Who informs it?o Stakeholder “interviews”o Going back and talking to stakeholderso Survey of Principal, Assistant Principals and Social

Workers (later years)

Data Informed Sustainability Planning

How do you do it?o We went low tech for the first 4 yearso Now we have developed on-line database

How do you fund it?o Used SS/HS funding to develop database and work with

statistician on analysis

How do you use it?o Developed preliminary evaluation modelo Referral and service trends o Impact on programming decisiono Impact on financial model and cash flow for agencies

Utilizing the Data to Inform and Sustain

Referral and Service Trendso Impact on programming decisiono Impact on financial model and cash flow for agencies

Case for Ancillary and Supportive Serviceso Teacher Consultation o Care Coordinationo Training for educational staff; classroom

presentations

Case for educational Benefit o Reduction in Suspensions o Reduction in Suspensions for students with 3 or moreo Reclaimed instructional time (not analyzed yet)

Referral Data SY06

ESMH Ref er ral SY06

31

23

37

56

1719

23

35

12 11

1

0

10

20

30

40

50

60

J ul 05 Aug 05 Sept 05 Oct 05 Nov 05 Dec 05 J an 06 Feb 06 Mar 06 Apr 06 May 06 J un 06

Ser ies1

Service Data SY07

0

100200

300400

500600

700800

900

MPS Student Demographics SY06-SY09

1017 Students 61% male and 39% female General Ed: 55% and 45% Special Ed (SPED 15.9%)Ethnicity

NATIVE AMERICAN – 8.3% (4.5%) AFRICAN AMERICAN – 48.9% (39.6%) ASIAN – 2.1% (9.0%) HISPANIC – 28.1% (17.1%) WHITE – 12.6% (29.8%) NATIVE HAWAIIAN PAC ISL – 0%

ELL – 31.2 % (23.2%)

District wide percentages are in parenthesis

Evaluation of MPS ESMHP

Access and sustained engagement in treatment 85 % of students seen once face to face; 70% within 10

days; 65% 1st time receiving services Average 17 visits per school year

Improved mental health functioning Parents and teachers report decreases in the emotional

and behavioral problems Improved school functioning

Decrease in school suspensions for students receiving mental health treatment

Principals and school social workers reported reduced office referrals and student suspensions

Impact on Suspensions

SY07 N=298 Seen 4x and had 1 suspension SY06 (n=82); only 27.5% had suspension in SY06

Impact on suspensions: SY06 to SY07 – N=82 50% reduced 1-6 32% stayed the same

Multiply Suspended SY06-07

SY06-SY07 Multiply Suspended (N=19) SPED=64% ELL=37% Male=75% African American=63%

2 suspensions or less=78.9%

52.6% to 1 or less

Moved to Zero=37%

Multiply Suspended SY06-SY07 (N=19)

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Individual Students

Nu

mb

er o

f S

usp

ensi

on

s

SY06

SY07

Change in Mean Suspensions: SY06-SY07

Impact on Suspensions: 156 Tx and 133 Comparison

0.92

0.54

0.83

1.80

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2.00

Baseline After Treatment

Aver

Days S

usp

en

ded

Full Tx

Full Comp

MPS – SY07 Change Mean Suspension: Full/African American

Impact on Suspensions: Full: 156 Tx, 133 Com; AA: 83 Tx, 76 Comp

0.92

0.54

0.83

1.80

1.30

0.61

1.131.17

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2.00

Baseline After Treatment

Full Tx

Full Comp

African Amer Tx

African Amer Comp

Change in Attendance: SY06 to SY07 (N=35, N=20)

Total Sample of Treatment kids = 159 40 (25%) had less than

90% Attendance in SY06 (5 kids missing data in SY07; sample 35 students)

Total Sample of 133 comparison 20 (15%) of them had

lower than 90% attendance in SY06

Treatment Comparison

N % N %

35 100 20 100

22 62.9% 7 35%

Psychosocial, Placement and Cost Outcomes in Maryland’s SMH Programs for Youth in

Special Education

Sharon H. Stephan, Ph.D.

University of Maryland School of Medicine

Center for School Mental Health

Acknowledgements• Maryland State Department of Education

• Dr. Carol Ann Heath• Dr. Jodi King

• Prince George’s County Public Schools• Dr. Pamela Downing-Hosten• Dr. Florence Foreman

• Baltimore City Public Schools• Ms. Kim Lewis

• Center for School Mental Health• Drs. Dana Cunningham, Eric Slade, Mark Weist, Nancy

Lever• Christianna Andrews, Kerri Chambers, Emily Sidway

Background• Nationally, there has been an upward trend in students

identified with emotional disturbances (ED) requiring intensive services to access educational standards.

• This trend is evident in Maryland with the Maryland State Department of Education listing ED as a high growth category increasing approximately 75% over the last ten years.

• In the five years prior to initiating the Prince George’s County SMH Initiative, the use of non-public placements in Maryland had increased by 20%, with some districts their use by over 30%.

Current Issues in Nonpublic Placements• Non-public placements cost local school districts over

40K per student

• Many districts do not have mental health services that are devoted to students identified as disabled

• There are multiple indicators that the use of non-public placements are racially disproportionate

• The outcomes associated with these placements are inconsistent

Rationale for Implementation

• Initiation of PGSMHI (2006)• Collaboration between MSDE, state

university, and local school system• Concern over high rate of referrals to

non-public schools• Concern over racial disproportionality in

placement in non-public schools• High costs of non-publics• Relative lack of school based services

Rationale for Implementation

• Initiation of BSMHI (2009)• Collaboration between MSDE, state

university, and local school system• Concern over high rate of referrals to

non-public schools• High costs of non-publics• Documented success of PGSMHI

Target PopulationTarget Population• Students in special education who are at Students in special education who are at

risk of entering non-public settings due to risk of entering non-public settings due to an increase in behavioral and/or emotional an increase in behavioral and/or emotional problemsproblems

• Students in non-public settings who are Students in non-public settings who are prepared to return to their home schoolprepared to return to their home school

Program Model

• Therapists

• Case managers

• Psychiatric consultation

• Family Support Partners (BSMHI)

Limited Local Capacity for Data Analysis

• The data on highly restrictive placements was not organized to answer questions and guide interventions

• Breaking this barrier:• Identified risk factors that contributed to restrictive

placements - Age, Grade, Compliance Issues, Suspensions, Disability, Gender

• Analyzed national, state, local and individual data sources

• Targeted efforts on suspected risk factors• Collected extensive data on clients

Breaking Down Barriers

• Financing• Total costs of Non-public and alternative programs are

hard to determine• Non-public placements involve financing from two

sources- the state and local level• Completed low and high estimates of cost savings for

both the state and local district• Moved thinking from “GRANT” thinking to “DIVERSION”

thinking.

Economic Evaluation

• Determined costs associated with PGSMHI based on 3 years of data and 77 students• Personnel• Trainings• Equipment and supplies

• Determined average cost of nonpublic placements• Education costs• Transportation costs

Economic Evaluation• Moved from dichotomous thinking ….stay or go, Emotionally

Impaired or not, etc.• to every day counts = continuous variable success

for each student

• Question changed to how many days was the student maintained in the public setting?

• Unit of analysis is the number of days prevented in non-public (not whether the student was placed)

• Cost determined by the number of days placement was delayed

or averted times cost differential per day

CostsAggregate Per student

per dayPer student per year

Total Nonpublic Costs

3,964,928 163 39,038

Total PGSMHI Costs

732,487 30 7,212

Total Savings

3,232,440 133 31,826

Note: Data is based on a program year of 240 days

Cost Savings

• $133 per student/day averted• For students receiving intense

services and are considered at imminent risk for non-public placement

• These savings are not expected for students who have emotional conditions but are not at risk for non-public placements

• These savings are not expected for “usual care” mental health services provided in a school

Limitations of Evaluation

• Did not assess indirect costs of adverse consequences for school staff and students of retaining students in public schools

• No comparison sample of students in nonpublic setting

• Assume that students would have been quickly placed in a nonpublic setting without intervention

Student GPAStudent GPA

0

0.5

1

1.5

2

2.5

Enroll 2nd Qtr 3rd Qtr 4th Qtr

Courses PassedCourses Passed

0

1

2

3

4

5

6

7

8

Enroll 2nd Qtr 3rd Qtr 4th Qtr

Student ObservationsStudent Observations

0

5

10

15

20

25

30

35

Off Task TchrPrompt

Neg Resp Attn

Obs 1Obs2

All SymptomsAll Symptoms

0

0.5

1

1.5

2

2.5

3

3.5

Youth Parent Teacher

Intake Time 2

Youth Intake N = 59Parent Intake N = 47

BSMHI Placement Outcomes

• 93% of the students enrolled in the BSMHI were maintained in their current placement.

• The number of students referred to non public placement at primary school decreased 50% from the 2008-2009 school year to the 2009-2010 school year.

BSMHI Cost Outcomes

• Students were diverted a total of 24.3 years from more restrictive placements. With an estimated additional cost of $31,915* per student per year for a nonpublic placement, approximately $775,534 was saved the first year of this project.

$43,000 Nonpublic Placement for 1 Year- $11,085 Public School Cost for 1 Year $31,915 Add’l Cost for Nonpublic Placementx 24.3 Placement Years Deferred$775,534.50 Cost Savings

Center for School Mental Health

• Objective 1.2 Continue and strengthen ongoing CSMH efforts to conduct comprehensive analyses of established, data driven and impactful local and state SMH programs, with data-driven findings and case examples shared in yearly summary reports (12/11, 12/12, 12/13).

• Objective 1.3 Enhance CSMH efforts to assist state and local program leaders to identify and measure targeted student and school-level outcomes and to incorporate data-based decision making into practice via a quarterly SMH outcomes/data management newsletter, annual webinar (5/11, 5/12, 5/13), and a regularly maintained compendium of online resources.

• Objective 2.7 Narrow the research-practice divide and improve communication and knowledge sharing, and facilitate partnerships between researchers and other SMH stakeholders via the development and convening of a SMH Research-Practice Network and encouraging increased participation of researchers in the annual conference and National COP.

Call for Papers!• Special Issue on School-Based Mental Health Practice, Data-

Driven School Mental Health Practice:  (How) Is It Possible?

Advances in School Mental Health Promotion

This issue will seek to better understand how data-driven and research-based educational interventions are being manifested in school mental health delivery. 

For more information, please contact the Special Issue Guest Editor Michael S. Kelly at [email protected]. 

The deadline for manuscript submission is June 15, 2011.  Submissions should be sent to: [email protected]

Discussion

• Please unmute your phone so that you can participate in the discussion.

• You also can type in questions on the iLinc website using the instant messaging.