When Accountability Strategies Collide: Do policy changes that raise standards also erode public satisfaction? Rebecca Jacobsen, Ph.D. Assistant Professor,

When Accountability Strategies Collide:

Do policy changes that raise standards also erode public satisfaction?

Rebecca Jacobsen, Ph.D.Assistant Professor, Michigan State

University

Andrew Saultz and Jeffrey W. SnyderDoctoral Students, Michigan State

University

We appreciate partial support from MSU’sEducational Policy Center

Popular Accountability StrategiesTwo Theories of Action for School Improvement

• The strategy of public pressure• Public information is critical to successful democratic oversight (Plank & Boyd, 1994; McDonnell, 2009)

• Often asymmetry of information hindering public oversight• Draws on democratic accountability theories

• The strategy of raising the bar• Goal setting, especially challenging goals, leads to higher levels of achievement across a range of outcomes (Locke & Latham, 2006)

• Ensures continuous improvement• Draws on industrial and organizational psychology theories

• Research has focused on how these strategies have impacted student achievement (i.e. test scores) and many studies find significant, positive impact.

When Accountability Policies Collide

• No Child Left Behind Requires Both Strategies• Many states/districts in the US are implementing both strategies simultaneously

The Potential Problem• Raising the bar results in a decline in scores

• Unlike private industry, this decline is publicized widely

• The public, notorious for holding unsophisticated policy knowledge, may misunderstand the decline• Falsely attribute the decline to failing reform efforts

• As a result, the public may become dissatisfied with the performance of their public schools

When Accountability Policies CollideThe Potential Problem

• Some public skepticism is healthy for democratic institutions• Improves representatives responsiveness• BUT … you can have too much of a good thing

• The impact of declining public satisfaction• Other public services (e.g. police and fire services)

• Satisfaction predicts willingness to maintain public funding (Glaser & Hildreth, 1999; Simonsen & Robbins, 2003)

• Satisfaction related to democratic participation and exit (Lyons & Lowery, 1986)

• May erode public satisfaction with the public schools in inappropriate ways, ultimately hindering public support for future reform

Research Question

• Do the two popular accountability strategies in the U.S. collide and produce a potentially corrosive effect for public opinion?• Should we worry?

• Exemplar Case Study of New York City

New York CityImplementing the Two Strategies Simultaneously

• The strategy of public pressure• New York City Progress Reports provide letter grades – A through F – to evaluate each school’s performance

• Progress Reports sent home with students, released to the media at a press conference, and are available online

• “The purpose…is to generate pressure to get [the schools] moving forward.” – James Liebman, Former NYC Chief Accountability Officer.

• The strategy of raising the bar• After 97% of schools earned top grades (an A or B) in 2009, Chancellor Klein announced the city would raise the bar to limit the number of schools receiving top grades

• “We’ve seen that the setting of targets or lines have a substantial effect on schools’ efforts to achieve outcomes with their students.” – Liebman

Data• School Level Data• Publicly available data through both NYC/NYS• NYC report cards• NYC Schools Survey

• Asks parents their satisfaction with their child’s school.

• Elementary Schools (n=542)• At least two of the tested upper elementary grades (3rd, 4th, and/or 5th).

• Excluded combined K-8 schools• Excluded those with missing NYC school grades

• Excluded response rates on School Survey < 30%

Descriptive Changes

71% of schools in our sample saw a decline of at least one letter grade

34% of schools saw a decline of two or more grades

94% of the schools saw a decline in average parent satisfaction. (Average change = -0.16, or about a 4% decline)

Overall Grade Change from

2009-2010

Parental Satisfaction Change

from 2009-2010

Increased 14 schools 23 schools

Unchanged 157 schools 0 schools

Decreased 410 schools 558 schools

Did Performance Change?Change in Scale Scores

ELA* Math*

Grade 3 -0.8 +0.7

Grade 4 +2.9 -5.1

Grade 5 +0.7 -1.0

*Standard deviations for these tests ranged from 7.4 to 9.1

Performance change far more mixed when examining scale scores.

Changes were, on average, small.

Empirical StrategyFirst Difference Model

Υs = β0 + β1Us + β 2Vs + β3Ws + β4Xs + β5Zs+εs

• Y = satisfaction level difference from 2009-2010

• U = response rate difference from 2009-2010• V = enrollment difference from 2009-2010• W = vector of student demographic differences from 2009-2010

• X = vector of teacher demographic differences from 2009-2010

• Z = overall school letter grade change from 2009-2010

Results: First Difference Model

Enrollment0.000(.000) **

Response rate (%)0.001(.000) ***

Frpl (%)0.001(.000) **

LEP(%)-0.007(.002) ***

Less 3 years T exp-0.001(.001) **

Decreased Grade-0.028(.009) ***

* = p < .1** = p < .05*** = p < .01

Highly Statistically Significant

Substantively Small Impact

Discussion• Raising the bar has, in some cases, been shown to raise achievement

• The public has a right to know how well its schools are performing

• In NYC, we find evidence, though substantively small, that the collision of these two accountability strategies was negative for parent opinion.• While small, we remain concerned about this decline

• Likely underestimates impact on broader public opinion

• Parents often have additional information upon which to base their opinions

• Parents reported on their child’s education.

Unanswered Questions• Satisfaction – an intermediate outcome

• Will this prompt increased engagement?• Will this prompt withdrawal and removal of funding?

• Future Research• Is one year enough to cause concern?

• Short-term blip vs. long-term decline

• When does a decline in satisfaction prompt action?

• Is the decline in NYC large enough to warrant concern? Is the size of the decline related to which course of action the public takes?

• Can public information campaigns stave off the potential negative effects of declining satisfaction?

Conclusion: Should We Worry?

• Policy Feedback• Policies, once implemented, can shape future political dynamics and political support

• Accountability strategies produce outcomes beyond test score impacts that must also be considered

• Given our findings, we believe that concern is warranted for the development of a “vicious chain of low trust-declining resources-poor perceived government performance-lower trust” (Holzer & Zhang, 2004, p. 238)