Face to Face ESC/TEAblog.esc12.net/accountability/files/2015/09/Notes-Pages-FINAL-PBM...Consider lessons learned from PBM work over the last few years. Consider individually (2 min)

9/10/2015

1

ESC Region 12

September 15, 2015

Connecting TAIS framework to PBMAS.

Being proactive- using data early and in combination with other accountability reports.

Creating common language across state –Refocusing on program effectiveness.

Addressing all PBMAS concerns in targeted improvement plans.

Copyright © Texas Education Agency 2015. All rights reserved.

Statewide paradigm shift…◦ What is our understanding of PBMAS in general?

◦ How do we think about and what do we do with data?

DECPAS-DASPBMAS

9/10/2015

2

Performance-Based Monitoring

Analysis System (PBMAS)

Implementation began in

2004

SP

ED

CT

E

BE

/ES

L

Stage of Intervention




NC

LB

Required Determination Levels:

Meets Requirement, Needs Assistance, Needs

Intervention, Needs Substantial Intervention

Meets Requirement = LEA Allowable Maintenance of Effort (MOE) Flexibility

Federally Required LEA

DeterminationsImplementation began

in 2005

Federally Defined Elements

1. Compliance SPP Indicators 9,10,11,12,13

2. Valid, reliable, and timely data submission

3. Uncorrected noncompliance

4. Financial audit findings

State Defined Elements

5. PBMAS SPED stage of intervention (2008)

6. Significant Disproportionality (2012)

On-site Reviews

Residential Facility (RF) Monitoring

Implementation began

in 2006

Federally Required LEA

Determinations

Required Determination Levels:

Meets Requirement, Needs Assistance,

Needs Intervention, Needs Substantial

Intervention

Federally Defined Elements

1. Compliance SPP Indicators

9,10,11,12,13

2. Valid, reliable, and timely data

submission

3. Uncorrected noncompliance

4. Financial audit findings

State Defined Elements

5. PBMAS SPED stage of intervention

6. Significant Disproportionality

Meets Requirement =

LEA Allowable MOE Flexibility


Performance Based Monitoring Analysis System

(PBMAS)

SP

ED

CT

E

BE

/ES

L

NC

LB





Integrated Intervention(assigned a stage of intervention for more than one program area)

(required engagement in the Texas Accountability Intervention System (TAIS))


Unified State &

Federal

Accountability

System

Performance Based Monitoring (PBM)

Unified Performance Based Monitoring System (PBM)

SPED: Integrated analysis of:

1.PBMAS indicators

2.Compliance SPP Indicators 9, 10, 11, 12, 13

3.Valid, reliable, and timely data submission

4.Uncorrected noncompliance

5.Financial audit findings

CT

E

BE

/ES

L

NC

LB




Stage of Intervention/Federal LEA Determinations:

Stage 1 or Not Staged = Meets Requirement

Stage 2 = Needs Assistance

Stage 3 = Needs Intervention

Stage 4 = Needs Substantial Intervention

Integrated Intervention(assigned a stage of intervention for more than one program area)

(required engagement in the Texas Accountability Intervention System (TAIS))



Unified State &

Federal

Accountability

System

Coming Mid-October

9/10/2015

3

Percentages

Data

Student performance

Connections

Ownership

Support

Disproportionality

Proactive

Common language

Continuous improvement

Processes

Responsibility

Relationships

Interventions

Analysis

TODAY IS TODAY IS NOT

Reframe our thinking

Analyze data

Ask questions

Seek connections

Consider cross-program implications

Filling out templates Technology support with

templates A conversation about what is

out of our control

Consider lessons learned from PBM work over the last few years.

◦ Consider individually (2 min)

◦ Discuss with table group/come to consensus (3 min)

◦ Share with whole group (4 min)

9/10/2015

4

When it is okay or not okay to work in silos?

Who are the right people to bring to the table? (How do you know)?

Why don’t we blame or point fingers? But…at the same time how do we “dig” to find the root cause or “data driver?”

Others?

PBMAS Updates

Changes to Cut Points for BE/ESL Indicator #9 (TELPAS Reading Beginning Proficiency Level Rate) have been implemented

Indicator most missed by districts in our region Grades 3-8 Mathematics & Reading for students served by an ESL program

9/10/2015

5

There were no changes to cut points for 15-16 for these 10 indicators

Be aware that some indicators give credit for improved requirement (#3, #4, #5, #8, #9, #10) and some do not (#1, #2, #6, #7)

Problem Indicators for many in Region 12 - #1 Title I, Part A STAAR 3-8 Passing Rate – Writing/Math

CTE Indicators: #1-4 added Performance Level Assignment for Social Studies and ELA remains report only

New! #4-The hold harmless provision described in Section II of the manual applies to this indicator.

PAR vs. MOA

Most changes in Performance Based Monitoring

9/10/2015

6

Program Area and Indicator Number

Indicator Name 2015 PBMAS

SPED #1(i-v) SPED STAAR 3-8 Passing Rate

(M, R, S, SS, W)Add STAAR A and STAAR Alternate 2 results.

Changes & additions to PL’s

Continue ELA indicator as RO.

One year of data.

No RI or SA.SPED #3(i-iv) SPED STAAR EOC

Passing Rate

(M, S, SS, ELA)


16

BE/ESL, NCLB, and

SPED YAE Indicators

PL 0 PL 1 PL 2 PL 3

Mathematics 70.0% - 100% 60.0% - 69.9% 50.0% - 59.9% 0% - 49.9%

Reading 70.0% - 100% 60.0% - 69.9% 50.0% - 59.9% 0% - 49.9%

Science 65.0% - 100% 55.0% - 64.9% 45.0% - 54.9% 0% - 44.9%

Social Studies 65.0% - 100% 55.0% - 64.9% 45.0% - 54.9% 0% - 44.9%

Writing 70.0% - 100% 60.0% - 69.9% 50.0% - 59.9% 0% - 49.9%

Copyright © Texas Education Agency 2015. All rights reserved.17

SPED IndicatorPL 0 PL 1 PL 2 PL 3 PL 4

1(i): Mathematics 70.0% - 100% 55.0% - 69.9% 40.0% - 54.9% 25.0% - 39.9% 0% - 24.9%

1(ii):Reading 70.0% - 100% 55.0% - 69.9% 40.0% - 54.9% 25.0% - 39.9% 0% - 24.9%

1(iii):Science 65.0% - 100% 50.0% - 64.9% 40.0% - 49.9% 25.0% - 39.9% 0% - 24.9%

1(iv):Social Studies 65.0% - 100% 50.0% - 64.9% 40.0% - 49.9% 25.0% - 39.9% 0% - 24.9%

1(v): Writing 70.0% - 100% 55.0% - 69.9% 40.0% - 54.9% 25.0% - 39.9% 0% - 24.9%

All EOC Indicators,

Except SPED #3 and

CTE #4

PL 0 PL 1 PL 2 PL 3

Mathematics 60.0% - 100% 50.0% - 59.9% 40.0% - 49.9% 0% - 39.9%

Science 60.0% - 100% 50.0% - 59.9% 40.0% - 49.9% 0% - 39.9%

Social Studies 60.0% - 100% 50.0% - 59.9% 40.0% - 49.9% 0% - 39.9%

ELA Report Only Report Only Report Only Report Only

Copyright © Texas Education Agency 2015. All rights reserved.18

SPED #3 & CTE #4PL 0 PL 1 PL 2 PL 3 PL 4

Mathematics 60.0% - 100% 50.0% - 59.9% 40.0% - 49.9% 30.0% - 39.9% 0% - 29.9%

Science 60.0% - 100% 50.0% - 59.9% 40.0% - 49.9% 30.0% - 39.9% 0% - 29.9%

Social Studies 60.0% - 100% 50.0% - 59.9% 40.0% - 49.9% 30.0% - 39.9% 0% - 29.9%

ELA Report Only Report Only Report Only Report Only Report Only

9/10/2015

7

For SPED Indicator #1(i-iv), #3(i-iv), and CTE #4• It stipulates that any district that received a PL 0 or 0 RI on the SPED

STAAR Modified Participation Rate indicator in the 2014 PBMAS that would otherwise receive a PL 3 or PL 4 on SPED Indicator #1(i-iv) or #3 (i-iv) in the 2015 PBMAS will receive a PL 3 HH or PL 4 HH, as applicable for that subject area(s).

• For 2015 PBMAS interventions purposes, the count of PL 3 HH or PL 4 HH will not be considered in a district’s total PL 3 or PL 4 count in the program area.


19

Program Area and Indicator Number

Indicator Name 2015 PBMAS

SPED #17, 18, 19 SPED Discretionary DAEP, ISS, OSS Placements

Begin transition to a new PL structure by reporting disproportionality rates (as Report Only) in addition to percentage point differences.


20

Why are we transitioning to a new PL structure for the discipline indicators?

◦ The original expectation was that focusing on percentage point differences (DIFF) would encourage districts, regardless of PL assignment, to address issues of disproportionality, but this has not typically been the case. (these indicators’ progress rates have been slower than expected)

◦ Focusing on percentage point differences can mask very high rates of disproportionality and it may have even given some districts the impression they do not have disproportionate discipline placements, when they actually do.


21

9/10/2015

8

Focusing on districts’ rate of disproportionality is a

more meaningful, reliable way to evaluate

disproportionality and will provide more transparent

information.

Unlike percentage point differences,

disproportionality rates will yield more consistent PL

cut points across all three discipline indicators.


22

Disproportionality rates are not a significantly different methodology from what’s currently used; they basically take the current calculations one step further and tell us how much higher the special education rate is compared to the all students rate, e.g., 50% higher, 10% higher, 200% higher.

The new PL structure is scheduled to be implemented with the 2017 PBMAS.


23

SPED Discretionary OSS Placements

SPED OSS SPED Students

SPED OSSPlacements

12.2 170 1,388

OSS All Students

All OSS Placements

6.0 672 11,224

Difference = 6.2(12. 2 – 6.0 = 6.2)

Copyright © Texas Education Agency 2015. All rights reserved. 24

Question: How much higher is 12.2 than 6.0?

Answer: Calculate the Disproportionality Rate

percentage point difference (6.2)

all students OSS placements rate (6.0)= 103% higher

9/10/2015

9

6.0

0

2

4

6

8

10

12

SPED OSS Rate All Students OSS Rate

SPED OSS Rate Compared to All Students OSS Rate

■ This district’s special education OSS placement rate is 103% higher than its all students OSS placement rate.


25

■ High disproportionality rates indicate overrepresentation.■ This rate = PL 3 RO.

Each district’s disproportionality rate will be reported based on the percentage ranges below:


26

Report OnlyPL 0 (RO) PL 1 (RO) PL 2 (RO) PL 3 (RO)

Disproportionality Rate MIN - 10.0% 10.1% - 29.9% 30.0% - 49.9% 50.0% - MAX

2015 Leaver Data Validation district reports will be posted on the TEASE Accountability application on October 30, 2015.

2015 Discipline Data Validation district reports will be posted on the TEASE Accountability application on November 20, 2015.

2015 Student Assessment Data Validation district reports will be posted on the TEASE Accountability application on December 18, 2015.

◦ Indicator preview will be part of the September 24, 2015 TETN.


27

9/10/2015

10

Incident occurs

=PEIMS

Reason Code

(C165)

Disciplinary action

assigned=

PEIMS Reason ACTION

code(C164)

OSSNumber of Days per incident

3 for regular districts, 10 for charters

PBM pulls data for PBMAS

indicators

Special Education Discipline Indicators

Over representation inISSOSS

DAEP

ISSREPORT ONLY

Public report and potential interventions

EXPULSIONAppropriate

Reason Codes (C165)

andAge

Parameters

DVM validates districts’ discipline data and process

Action codes that remove student from classroom are governed by

TEC Chapter 37Appendix E

District inputs

Reason and Action

codes into PEIMS

DISCRETIONARY DAEPAge and

Placement RatesRepresentation

Indicator 1 Indicator 2 Indicators 3, 4Indicators 5, 6, 7, 8

Due Processand

Age Parameters

OSS – Out-of-School SuspensionISS – In-School Suspension

Potential connection to overrepresentation of African American in special education

Texas Accountability Intervention Framework



StateAccountability System

PBMAS

Index 1-Student Achievement

STAAR 3-8 Passing Rates

Index 3-Closing Performance Gaps

STAAR 3-8 Passing Rates

Index 4-Postsecondary Readiness

Annual Dropout RateRHSP/DAP Diploma RateGraduation Rate

System Safeguards STAAR 3-8 Passing Rates

9/10/2015

11

State Accountability Index Reports and System Safeguards (August)

Do not include math data

Do not include STAAR A data

Do not include STAAR Alt 2 data

PBMAS Reports

Do include math data

Do include STAAR A data

Do include STAAR Alt 2 data

A bridge study was used to include math data in 2015 PBMAS

There is not as strong a correlation between Indexes and PBMAS indicators in 2015 due to the differences in the data sets.

A stronger correlation will be available between federal system safeguards scheduled for release in September and the PBMAS indicators for this year.

District System SafeguardsPage 1

District System SafeguardsPage 2

How can I support students better?

Which program is contributing the most –

bilingual or ESL?

PBMAS

Do you think there is a disconnect

between TAIS and PBMAS? If so, why?

9/10/2015

12

District-wide Ownership and Accountability

High Expectations

Sense of Urgency

Clear Vision and Focus

Operational Flexibility


Capacity and Resources

Communications

Processes/Procedures

Organizational Structures


Discuss what the Support

Systems look like in action in your district.

9/10/2015

13

Clearly defined roles for program coordinator

Consider and discuss with your table team how you might use this activity to help make connections between the TAIS framework and PBMAS performance and program effectiveness.

How could this process be enhanced in your district?

9/10/2015

14

A district receives PBMAS data which indicates if programs are effective.

If data reflects that programs are not effective, something must be done to change that.

We need to intervene.

What does it mean to intervene?

the act of becoming involved in something in order to have an influence on what happens


Data AnalysisNeeds

Assessment

Improvement Plan

Implement & Monitor


9/10/2015

15

Be thinking about the reasons data meetings fail.

Identify

the

Problem

Annual GoalProblem StatementELLs have a

50% pass rate in reading

ELL student group will

have 60% pass rate on 2016

STAAR reading

Root Cause

Strategy

Lack of teacher understanding for the instructional

strategies needed to meet linguistic

needs of ELLs

Implement sheltered

instruction campus-wide

9/10/2015

16

ELLs have a 50% pass rate

in reading

Lack of teacher understanding for the instructional

strategies needed to meet linguistic

needs of ELLs

Annual GoalProblem Statement

Root Cause Strategy

Implement sheltered

instruction campus-wide

ELL student group will

have 60% pass rate on 2016

STAAR reading

Problem Statement

Gap

in the Data

Performance Levels

0 1 2 3 4

Report Only

0 Required Improvement(0 RI)

Not Assigned (NA)

Special Analysis (0, 1, 2, or 3 SA)

Hold Harmless (3 HH or 4 HH)

48Copyright © Texas Education Agency 2015. All rights reserved.

9/10/2015

17

Non-Example Example

Teachers not implementing behavior RtI process has led to a PL3 in overrepresentation of African American students in special education.

Written objectively

Avoids causation

Includes specific details

District has a 12% difference in the representation of African American students in special education (38%) as compared to the representation of African American students within the entire population (26%). Concise Gap in data Verified by data Written objectively Single manageable issue

?

Non-Example Example

Teachers not implementing behavior RtI process has led to a PL3 in overrepresentation of African American students in special education.

Written objectively

Avoids causation

Includes specific details

District has a 12% difference in the representation of African American students in special education (38%) as compared to the representation of African American students within the entire population (26%). Concise Gap in data Verified by data Written objectively Single manageable issue

?This is the rate on

which the PL is based. Page 72 of the 2015

PBMAS Manual.

9/10/2015

18

A district report such as PBMAS is made up of data from campuses.

Districts must understand this relationship.

Mayberry ISD received a performance level 2 in the percent of students with disabilities who are removed to DAEP. They receive a PL3 in the report only indicator that looks at the disproportionality of DAEP placements for students with disabilities.

What’s wrong with that?

What could this be affecting?

What would a district need to focus on in reviewing this situation?

ALL?

Which campuses are contributing to this indicator?

9/10/2015

19


SOME?


ONE?

The data from a campus must be analyzed to calculate the PL for this indicator.

The central office can influence what happens, but the campus has to make the change happen because they work with the students!

9/10/2015

20

District

Campus

Department

Grade level

Classrooms

Individual students

Where will I look in the manual? (Who)?

Where do I dig to identify trends with student groups?

How are indicators connected and impacting one another?

How do I look across programs, instructional settings, etc?

We identified an issue with ESL STAAR passing rate.

◦ Who are the ESL STAAR passing students?

◦ Were they SPED?

◦ Was inclusion used effectively?

9/10/2015

21

We have a high number of discretionary placements to DAEP.

◦ Which campus?

◦ What offenses?

◦ What time of day/location/setting?

◦ Which administrators?

◦ Are they repeat offenders?

◦ How is instruction being delivered?

Happy ISD has a PL3 in SPED discretionary DAEP placements. I am meeting with the administrators from all 3 campuses in the district.

◦ What will you ask them to bring with them to the meeting.

◦ How will you frame the issue (your actual conversation)?

◦ What questions will you ask?

◦ What are the next steps that will follow this meeting?

◦ What will your role as the district level administrator be in “next steps?”

Activity:

◦ Identify one indicator on your PBMAS report with a PL 2, 3 or 4.

◦ Consider Bil/ESL, CTE, NCLB, SPED. (what other indicators is your chosen indicator impacting)

◦ What questions do we ask to dig deep into the meaning of the data and connections to other areas? (ex. 5 whys)

9/10/2015

22

3 new learnings

2 questions

1 immediate application

Copyright © Texas Education Agency 2009. All rights reserved. 64

Reminder: go into ISAM to view your local determination

Target Date: October 21

Can view PBMAS report as well asdetermination rubric

Not Staged or Stage 1 = Meets Requirements

Stage 2 = Needs Assistance

Stage 3 = Needs Intervention

Stage 4 = Needs Substantial Interventions

9/10/2015

23

Develop a game plan

Bring the right people to the table

Dig, dig, dig into the data

Consider program improvements and plan accordingly

Implement systems for monitoring improvement strategies

Keep the attention away from “filling out templates for TEA”

Maintain momentum of continuous improvement

As you go forth in your work with PBMAS…remember the focus:

• Improving student performance

• Improving the effectiveness of district programs

Face to Face ESC/TEAblog.esc12.net/accountability/files/2015/09/Notes-Pages-FINAL-PBM...Consider lessons learned from PBM work over the last few years. Consider individually (2 min)

Documents