We would like to thank the Institute for Education Sciences for their support and funding.

We would like to thank the Institute for Education Sciences for their support and funding. Please direct any questions to [email protected]

735 S. State Street | Ann Arbor, MI 48109

Principal Investigators:

Brian Jacob & Susan DynarskiUniversity of Michigan

Barbara Schneider & Ken FrankMichigan State University

Thomas Howell Center for Educational Performance & Information

Joseph MartineauMichigan Department of Education

What is MCER?

Michigan Consortium for Educational Research

IES-funded collaboration between: University of Michigan Michigan State University Michigan Department of Education Michigan Center for Educational Performance

and Information.

Objectives of MCER

Engage stakeholders and education experts in research for the benefit of public education in Michigan

Provide research-based evidence to policymakers in Michigan

Inform national policy initiatives for improving education

Inaugural Research Questions

What is the effect of the Michigan Merit Curriculum on… course-taking patterns, student achievement high school graduation, postsecondary attendance

What was the effect of the Michigan Promise Scholarship on… college entry, college choice, college completion

Do these effects vary by school or student characteristics?

MCER Data Goal

Detailed Student-Level Longitudinal Dataset

Attainment:

HS graduation,

GED, college degrees

K12:

test scores, enrollment,

demographics, GED, ACT

scores

Postsecondary: NSC, MI public

college transcripts

Michigan Merit Curriculum Study As of 2011, all graduating MI HS students must pass

16 rigorous courses (e.g., Algebra II, Biology, Chemistry, Physics) and complete end-of-course exams to measure content mastery

The evaluation will compare student outcomes before & after the MMC

Standardized test scores, HS graduation College entry, choice & completion

Random sample of 100+ schools: assess fidelity of implementation

End-of-course exams Student transcripts Interviews

Michigan Promise Study Intervention

$4,000 college scholarship for students with qualifying score on Michigan Merit Exam

Students must maintain 2.5 college GPA

Methods Regression discontinuity: compare outcomes

just above/below MME threshold College enrollment/choice/persistence (NSC) Scholarship receipt (Treasury)

Emerging Research Questions

• Has mandatory ACT (now part of MME) improved college attendance and choice?

• What is the “value-added” of individual schools, once we control for student characteristics and initial achievement?

• Are Michigan’s charter schools raising student achievement?

Example studyThe Impact of the Threat of School Sanctions: A

Regression Discontinuity Study of Being on a Probationary List

Guan K. Saw, I-Chien Chen, Barbara L. Schneider, Kenneth A. Frank

Michigan State University

• This study analyzes the effect of the first stage of school sanctions in Michigan, being on a probationary list. We pay attention to its possible impact on student achievement in high-stakes and low-stakes subjects at school level.

Purpose of Study

Background

• School sanctions, increasingly used as instruments of education policy, have been the focus of debate at federal and state levels.

• The goal of sanctions is to incentivize schools that fail to meet academic standards to improve their students’ educational performance.

• How does it work to make school change?

Two possible explanations1. Probations serve as a Social Stigma

– Stigmatizing or labeling is a potent tool for guiding individuals to conform to social norms.

– For failing schools, stigmatization becomes a motivating factor to make change (Figlio & Rouse, 2006; Ladd & Glennie, 2001; Mintrop, 2004; Sim, 2007, 2009).

– “Being on a probationary list” can be seen as a social stigma, which may have a “labeling effect”.

2. Effects of Sanction Threats– Not only imposing sanctions, but also

threatening sanctions can change an individual’s behavior (Lacy & Niou, 2004).

– In economic sanctions literature, some argue that sanctions threatened are often more effective than those that are deployed (Drezner, 1999; Drury & Li, 2006; Lacy & Emerson, 2004; Smith, 1996).

– In education, the effects of sanction threats on low-performing schools are mixed (Chiang, 2009; Figlio & Rouse, 2006; Springer, 2008; Winters et al., 2010).

Crowding-out hypothesis– There is a growing concern that test-based

accountability may cause schools to shift inputs from low-stakes subjects (Corbett & Wilson, 1991; Kohn, 1999; Nichols & Berliner, 2007; Whitford & Jones, 2000).

– This crowding-out hypothesis was supported by some qualitative evidence (Au, 2007; Groves, 2002; King & Mathers, 1997; Murillo & Flores, 2002).

– In contrast, some quantitative studies report that high-stakes testing policies led to significant gains in low-stakes subjects (Jacob, 2005; Winters, Trivitt, & Greene, 2010).

Michigan Context: PLA and Watch List

• Since 2009, Michigan Department of Education (MDE) has annually published a list of the lowest performing 5% schools, named the Persistently Lowest Achieving list (PLA list).

• The PLA list is established by certain criteria: (a) 2-year average percent proficiency in math and reading;(b) 4-year slope of percent proficiency in math and reading; (c) whether a school made Adequate Yearly Progress (AYP) status over the past two years; and

(d) whether a school had a 4-year graduation rate below 60% for three years in a row.

• The PLA list schools have to make significant gains in student achievement within a short time to get off the list.

• If not, further sanctions may be imposed including turnaround, restart, and closure of schools.

• With labeling and sanction threat effects, we hypothesize that the PLA list schools tend to positively affect student achievement.

• In addition to the PLA list (bottom 5%), MDE also publishes a “Watch list” of schools in the lowest quintile (6-20%), which were identified as being in danger of falling under the 5% mark.

• This does not affect the PLA ranking but provides an alert to these schools to keep them out of the PLA category.

• Without a strong threat of further sanctions, we expect that the labeling effect of being on the watch list is relatively limited.

Michigan Context: Watch List

Data Sources• Our longitudinal school-level data

constructed with:(1) Michigan Educational Assessment System (MEAS);(2) Common Core of Data (CCD).

• We only focus on a sample of regular high schools (only 333 schools ranked by state-wide achievement base percentile ranking).

Measures• Treatment 1: Being on 2008-09 PLA list

(<5%)Treatment 2: Being on 2008-09 Watch list (6-20%)

• Forcing variable: State-wide achievement base percentile ranking

• Outcomes: % of students met proficiency level in (a) high-stakes subjects: math, reading, writing,(b) low-stakes subjects: science, social studies

• Covariates: % of free/reduced lunch students, % of black students, school size, and pupil teacher ratio.

Analytic Method• We employ a regression discontinuity

design (RDD) method. This is a quasi-experimental design, in which treatment status depends on whether an observed covariate exceeds a fixed threshold(Lee & Card, 2008; Shadish, Campbell, & Cook, 2002).

• In our case, the fixed threshold is the cutoff (5% or 20%) of percentile ranking for being on the PLA or watch lists.

• RD Analysis of PLA List

• RD Analysis of Watch List

0 5

0 205 35

10 100

100

PLA list

Watch list

Percentile ranking

Percentile ranking

RDD with covariates

• Given the imbalance of covariates between treatment and control groups, we include the covariates in the RD models, which can (Imbens & Lemieux, 2008): (1) reduce small sample bias;(2) improve precision if covariates correlated with potential outcomes (as in analyses of

randomized experiments)

Findings: PLA List0

2040

6080

% o

f stu

dent

s m

et m

ath

prof

icie

ncy

leve

l in

2010

-201

1

0 5 10Tier 2 percentile rank in 2008-09

Figure 1a. Percentage of Students Met Proficiency Level in 2011, by Tier 2 Percentile Rank in 2008-09

020

4060

80%

of s

tude

nts

met

read

ing

prof

icie

ncy

leve

l in

2010

-201

1


020

4060

80%

of s

tude

nts

met

writ

ing

prof

icie

ncy

leve

l in

2010

-201

1


020

4060

80%

of s

tude

nts

met

sci

ence

pro

ficie

ncy

leve

l in

2010

-201

1


020

4060

80%

of s

tude

nts

met

soc

ial s

tudi

es p

rofic

ienc

y le

vel i

n 20

10-2

011


Mathematics Reading Writing

Science Social Studies Cutoff = 5%<5% (PLA list) = 19 schools

5-10% (Control group) = 19 schools

Effects of Being on the PLA List (2008-2009)

• RDD analyses show a positive “list” effect on all subjects 2010-2011.

• In models with presence of covariates, only the positive “list” effect on writing achieves statistical significance.

• No negative effect on low-stakes subjects of science and social studies was observed.

020

4060

80%

of s

tude

nts

met

sci

ence

pro

ficie

ncy

leve

l in

2010

-201

1

0 10 20 30 40Tier 2 percentile rank in 2008-09

Findings: Watch List

Figure 1b. Percentage of Students Met Proficiency Level in 2011, by Tier 2 Percentile Rank in 2008-09

Cutoff = 20%5-20% (Watch list) = 54 schools


020

4060

80%

of s

tude

nts

met

mat

h pr

ofic

ienc

y le

vel i

n 20

10-2

011


020

4060

80%

of s

tude

nts

met

read

ing

prof

icie

ncy

leve

l in

2010

-201

1


020

4060

80%

of s

tude

nts

met

writ

ing

prof

icie

ncy

leve

l in

2010

-201

1


020

4060

80%

of s

tude

nts

met

soc

ial s

tudi

es p

rofic

ienc

y le

vel i

n 20

10-2

011



Science Social Studies

Effects of Being on the Watch List (2008-2009)

• We found no effect of being on the watch list for all subjects in 2010-2011.

• This finding holds across estimation models using different bandwidths (5%, 10%, and 15% below and above cutoff).

Robustness Test• We created a pseudo PLA list using the

previous school year data (2007-08), before the state policy mandating assignment of low-performing schools to a probationary list had been enacted.

• Then, we compare the results of 2008-09 PLA list effects to 2007-08 pseudo PLA list effects.

• We expect that there would be no effect of being on the pseudo PLA list since these schools did not receive a labeling treatment or real threat of sanctions.

Findings: Pseudo PLA List

Figure 1c. Percentage of Students Met Proficiency Level in 2010, by Pseudo Tier 2 Percentile Rank in 2007-08

Cutoff = 5%<5% (Pseudo PLA list) = 18 schools


020

4060

80%

of s

tude

nts

met

mat

h pr

ofic

ienc

y le

vel i

n 20

09-2

010

0 5 10Pseudo tier 2 percentile rank in 2007-08

020

4060

80%

of s

tude

nts

met

read

ing

prof

icie

ncy

leve

l in

2009

-201

00 5 10

Pseudo tier 2 percentile rank in 2007-08

020

4060

80%

of s

tude

nts

met

writ

ing

prof

icie

ncy

leve

l in

2009

-201

0

0 5 10Pseudo tier 2 percentile rank in 2007-08


Effects of Being on the Pseudo PLA List (2007-08)

• Results from the 2007-08 pseudo list analysis indicate no “list” effect on all subjects in 2009-2010.

• This finding further testifies the robustness of the effects of 2008-2009 PLA list.

Conclusion• Being on PLA list as a social stigma

combined with possible following incremental sanctions may spur school improvement.

• Without a real and strong threat of further sanctions, just being labeled by the watch list does not stimulate school performance.

• Crowding-out hypothesis was not supported by our data.

Limitation and Discussion• Limitations:

(1) given three years of data, we only can claim a short term effect of PLA list.(2) the limited variables in the data set provide no information to uncover real changes being implemented in the schools.

• Puzzles:(1) Are the PLA effects stronger for writing? Why? (2) What organizational processes occurred in the

schools placed on the PLA list?

…take home message…

Within the school sanctioning context:• The labeling effect is limited. • The threat of sanctions does motivate low-

ranked probationary schools to make changes.

• Crowding-out effects may not occur but spillover effects may be present.

“On the List” Example of Evaluation of Institution

• State implemented• Involves state data as well as school

composition (common core)• Apply to Michigan Merit Curriculum

– Graduate students– Data– Collaborative faculty

We would like to thank the Institute for Education Sciences for their support and funding. Please direct any questions to [email protected]

735 S. State Street | Ann Arbor, MI 48109

Principal Investigators:

Brian Jacob & Susan DynarskiUniversity of Michigan

Barbara Schneider & Ken FrankMichigan State University

Thomas Howell Center for Educational Performance & Information

Joseph MartineauMichigan Department of Education

We would like to thank the Institute for Education Sciences for their support and funding.

Documents

michigan promise scholarship

michigan merit exam

college choice

college scholarship

student outcomes

student characteristics

threat of school sanctions

stage of school sanctions