IT KEEPS GETTING BETTER! Using School-wide Data for Continuous Quality Improvement Kelsey R. Morris, EdD—University of Oregon Nadia K. Sampson, MA—University.

IT KEEPS GETTING BETTER!Using School-wide Data for Continuous Quality Improvement

Kelsey R. Morris, EdD—University of OregonNadia K. Sampson, MA—University of Oregon

Session C13

Session Overview• Value of data-based decision making• Nexus of data-based decision making and SWPBIS• Cycle of continuous quality improvement

• Student Outcomes• Implementation Fidelity

Goal

• Craft precise problem statements• Craft solution-based action plans• Engage in continuous quality improvement

Maximizing Your Session Participation

• Where are you in your implementation of the concepts presented?• Exploration & Adoption

• Installation

• Initial Implementation

• Full Implementation

• What do you hope to learn?

• What new learning do you take away from the session?

• What will you do with your new learning?

Data Collection & Analysis• Data are “the numerical results of measuring some

quantifiable aspect of behavior” (Mayer, Sulzer-Azaroff, & Wallace,

2012, p. 130).

• Data collection involves:• Observation of a behavior

• Notation of the behavior characteristics and context

• Data analysis involves:• Converting numerical results into graphs

• Using graphed results for instructional decision making

Value and Utility of Data

• Repeatedly giving people the right information, at the right time, in the right format is the single most effective way to improve decision making and achieve valued outcomes (Gilbert, 1978).

Data

PracticesSystems

Valued Outcomes

Performance Gap & Cause Analysis

Cur

rent

Rea

lity Valued

Outcom

es

Environment/System

1—Information • Clear expectations• Timely, specific

feedback

2—Resources • Materials, tools• Time• Processes

3—Incentives • Financial & non-

financial encouragement

Individual Persons

6—Knowledge • Requisite knowledge

and skill base

5—Capacity • Ability to learn

and do

4—Motives • Desire to work

and excel

Data-based Decision Making• Effective teams use data to document progress and outcomes, guide

decisions, and inform stakeholders (Boudett, City, & Murnane, 2006; Burke, 2010; Deno, 2005; Hill 2010; Newton, Algozzine, Algozzine, Horner, & Todd, 2011; Newton, Horner, Algozzine, Todd, & Algozzine, 2009; Pidgeon & Gregory, 2004; Renfro & Grieshaber, 2009)

• A critical predictor of sustained implementation of SWPBIS (Coffey & Horner, 2012; McIntosh et al., 2013)

• Fidelity and student outcome data are essential (Fixsen, Blase, Metz, & Van Dyke, 2013)

• Continues to be a struggle for schools (Dunn, Airola, Lo, & Garrison, 2013; Schildkamp, Ehren, & Lai, 2012; Telzrow, McNamara, & Hollinger, 2000)

• Advances in computer technology could provide efficient means for data management (Wayman, 2005)

Components of SWPBIS

Defined Behavior Expectations

Teaching of Behavior Expectations

Acknowledgment Systems

Consequence Systems

Evaluation

Evaluation of Effectiveness• Essential Question

• Is the student successful with this level of support?

• Intensity is a two-way street.• Improved student outcomes are

the result of continually monitoring and modifying (as needed) instruction, interventions, and supports.

Goal:Increase prosocial behavior and enhance quality of life

Continuous Quality Improvement

Plan

Implement

Evaluate

Adult Behaviors Cause Student Change

Outcomes Fidelity

School-wide Behavior Data• Critical Questions

• How often are problem behaviors occurring?

• When are problem behaviors frequently occurring?

• Where are problem behaviors frequently occurring?

• What problem behaviors are frequently occurring?

• Who is frequently engaging in problem behaviors?

School-wide Behavior Data

September = rate of 3.50 average referrals/day/month October = rate of 4.53 average referrals/day/month

Spikes at 9:45 AM and 1:00 PM – 2:30 PM

Top 3 non-classroom locations = bathroom, playground, & hallway

School-wide Behavior Data

Tuesday and Wednesday are the school days with the highest frequency.

Top 3 Problem Behaviors = inappropriate language, defiance, disruption

5th & 8th grades have the highest frequency

25 students have more than 1 referral

Problem Solving with Precision• The statement of a problem is important for team-based

problem solving.• Everyone must be working on the same problem with the same

assumptions.

• Problems are often framed in the “primary” form. • Raises awareness

• Not useful for problem solving

• Precise problem statements result from a detailed data review and are solvable.

Problem Solving with Precision

Primary Statements

• There are too many referrals

• Gang behavior is increasing

• The cafeteria is out of control

• Student disrespect is a big problem

Precision Statement

• There are more ODRs for aggression on the playground than last month. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Problem Solving with Precision• There are more ODRs for aggression on the playground

than last month. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

What? Where? When? Who? Why?

Aggression Playground 1st RecessLarge number

of students

To get new playground equipment

Data Analysis for Precision

Location

Problem Behavior

Time of Day

Persons Involved

Motivation

Precise Problem

Statement

Continuous Quality Improvement

Plan

Implement

Evaluate

Solution Development & Action PlanningEssential Elements

Explanation

1. PreventionHow can we avoid the problem context?• Who? What? When? Where?

2. TeachingHow can we define, teach, and monitor what we want?• Teach appropriate behavior, use problem behavior as the

non-example

3. RecognitionHow can we build in systematic acknowledgment/rewards for positive behavior?

4. ExtinctionHow can we prevent the problem behavior from continuing to pay off? (tied to motivation/function of behavior)

5. ConsequencesWhat are efficient, consistent consequences for problem behavior?

6. EvaluationHow will we collect and use data to evaluate our fidelity and outcomes?

Solution Development & Action Planning

Remember to include the

precise problem statement as well as a statement of

the goal.

Continuous Quality Improvement

Plan

Implement

Evaluate

Jan-14 May-14 Sep-140

10

20

30

40

50

60

70

80

90

100

39

54

32

Hamlin Middle SchoolTFI Subscales

Tier ITier IITier III

Evaluation—SWIS was installed for

data collection and analysis

Hamlin Action Planning• PBIS Team used TFI results to identify items of importance

1.3 & 1.4 Behavioral Expectations & Teaching Expectations

1.6 Discipline Policies (e.g., flowchart)

1.9 Feedback & Acknowledgement (e.g., menu of reinforcers)

• PBIS Team worked on items during team meetings• Solicited feedback from whole faculty

• Revised as needed

• Created lesson plans to teach all staff the aspects of school-wide PBIS during in-service week

Jan-14 May-14 Sep-140

10

20

30

40

50

60

70

80

90

100

39

71

5446

32

65

Hamlin Middle School TFI Subscales

Tier ITier IITier III

Jan-14 May-14 Sep-140

10

20

30

40

50

60

70

80

90

100

39

71

90

5450

32

44

Hamlin Middle SchoolTFI Subscales

Tier ITier IITier III

Enablers of Sustainability

Admin

istra

tor S

upport

Staff

Buy-in

Fidel

ity

Data

Team

ing

Resourc

es

Stake

holder

Invo

lvem

ent

Train

ing

SWPBS P

hiloso

phy

Motiv

atio

n

Distri

ct S

upport

0

10

20

30

40

50

60

70

Nu

mb

er o

f R

esp

on

ses

McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014).

Barriers to Sustainability

0102030405060708090

Nu

mb

er o

f R

esp

on

ses

McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014).

What is most related to high sustainability?

McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014).

IT KEEPS GETTING BETTER!Using School-wide Data for Continuous Quality Improvement

Kelsey R. Morris, EdD—University of OregonNadia K. Sampson, MA—University of Oregon

Session C13