-
April 22, 2013
Market-oriented educationreforms rhetoric
trumps realityThe impacts of test-based teacher
evaluations, school closures, and increasedcharter school access
on student
outcomes in Chicago, New York City, andWashington, D.C.
B Y E L A I N E W E I S S A N D D O N L O N G
BROADER, BOLDER APPROACH TO EDUCATION WWW.BOLDAPPROACH.ORG
-
Table of contents
Executive summary
.....................................................................................................................................................3
The reformers and their claims
..............................................................................................................................7
A brief synopsis of the evidence regarding core components of
the reform agenda
...............................................10
Test-based accountability systems
...........................................................................................................................10
School closures
.......................................................................................................................................................12
Increasing the reach of
charters...............................................................................................................................13
Turnover and instability
.........................................................................................................................................15
A no excuses take on the influence of poverty
......................................................................................................16
The reforms have not strengthened school systems in the three
cities
....................................................................16
Washington, D.C.: Firings and closures increased churn
........................................................................................19
New York City: Merit pay failed while school reshuffling
produced mixed results
..................................................26
Chicago: Neighborhood school replacement increased instability
but not performance
..........................................33
The bottom line: Highlighted reforms in the three cities
produced upheaval, overshadowing smaller promising
programs
................................................................................................................................................................39
The reforms have not improved student
outcomes..................................................................................................41
Washington, D.C.: Race and income-based achievement gaps grew
.......................................................................42
New York City: Nine years of market-based reforms failed to
improve test scores or narrow achievement gaps .......49
Chicago: As achievement gaps grew, African American students
fell further behind, and college readiness remained
disappointing
.........................................................................................................................................................54
The reforms have not saved money
..........................................................................................................................62
Washington, D.C.: Increased spending without public and parental
input produced few benefits ..........................62
New York City: Reforms hinged on likely unsustainable spending
increases
...........................................................64
Chicago: Spending increases mirrored those of other large
cities, but donors policy influence outstripped their
funding
..................................................................................................................................................................65
The bottom line: More spending, bolstered by private donations,
depressed democratic input without helping
students..................................................................................................................................................................67
Conclusion................................................................................................................................................................68
About the authors
.....................................................................................................................................................73
Appendix tables
........................................................................................................................................................73
Endnotes
...................................................................................................................................................................76
References
.................................................................................................................................................................83
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
2
-
Executive summary
T op-down pressure from federal education policies such as Race
to the Top and No Child Left Behind, com-bined with organized
advocacy efforts, is making a popular set of market-oriented
education reforms lookmore like the new status quo than real
reform. Reformers assert that test-based teacher evaluation,
increasedschool choice through expanded access to charter schools,
and the closure of failing and underenrolled schools will
boost falling student achievement and narrow longstanding race-
and income-based achievement gaps. The 2010 docu-
mentary Waiting for Superman presented these policies as sure
fixes for education woes closely correlated with
child poverty.
This report from the Broader, Bolder Approach to Education
examines these assertions by assessing the impacts of these
reforms in three large urban school districts: Washington, D.C.,
New York City, and Chicago. These districts were
chosen for study because all enjoyed the benefit of mayoral
control, produce reliable district-level test score data from
the National Assessment of Educational Progress (NAEP), and were
led by vocal proponents who implemented versions
of this reform agenda. Indeed, former reform leaders in all
three cities have become high-profile national proponents
who disseminate the agenda across multiple districts and
states.
The report finds that the reforms delivered few benefits and in
some cases harmed the students they purport to help. It
also identifies a set of largely neglected policies with real
promise to weaken the poverty-education link, if they receive
some of the attention and resources now targeted to the touted
reforms. Specifically the report finds:
Test scores increased less, and achievement gaps grew more, in
reform cities than in other urban districts.
Reported successes for targeted students evaporated upon closer
examination.
Test-based accountability prompted churn that thinned the ranks
of experienced teachers, but not necessarily
bad teachers.
School closures did not send students to better schools or save
school districts money.
Charter schools further disrupted the districts while providing
mixed benefits, particularly for the highest-needs stu-
dents.
Emphasis on the widely touted market-oriented reforms drew
attention and resources from initiatives with
greater promise.
The reforms missed a critical factor driving achievement gaps:
the influence of poverty on academic performance.
Real, sustained change requires strategies that are more
realistic, patient, and multipronged.
Test scores increased less, and achievement gaps grew more, in
reform cities than in other urban districts. Leaders
promised that the reforms would raise test scores, especially
those of minority and low-income students, and close race-
and income-based achievement gaps. Analysis of the most
reliable, comparable dataNational Assessment of Educa-
tional Progress scoresshows that the rhetoric did not match the
reality. While test scores increased and achievement
gaps shrank in most large urban districts over the past decade,
scores stagnated for low-income and minority students
and/or achievement gaps widened in the reform cities.
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
3
-
Between 2005 and 2011, in large, urban districts, Hispanic
eighth-graders gained an average six points in reading
(from 243 to 249), black eighth-graders gained 5 points (from
240 to 245), and white eighth-graders gained 3
points (from 270 to 273) (NCES 2005b, 2011b). In District of
Columbia Public Schools, however, Hispanic
eighth-graders scores fell 15 points (from 247 to 232), black
eighth-graders scores fell 2 points (from 233 to 231),
and white eighth-graders scores fell 13 points (from 303 to 290)
(Levy 2012c).
Reported successes for targeted students evaporated upon closer
examination. Reformers in all three cities claimed
that they had boosted student achievement and closed achievement
gaps. But when state test scores were recalibrated
to make standards consistent, compared with NAEP scores, and
disaggregated by race and income, gains vanished or
turned out to have accrued only to white and high-income
students.
New York City Mayor Michael Bloomberg claimed to halve the
white/Asian to black/Latino achievement gap in
city schools from 2003 to 2011, but scores on state-administered
tests, averaged across fourth and eighth grades in
reading and math, show that the achievement gap had stagnated;
it was 26.2 percentage points in 2003 and 25.8
percentage points in 2011 (a 0.01 standard deviation change).
Columbia University professor Aaron Pallas, who
calculated the 1 percent reduction, noted, The mayor has thus
overstated the cut in the achievement gap by a
factor of 50 (Pallas 2012b).
Test-based accountability prompted churn that thinned the ranks
of experienced teachers, but not necessarily bad
teachers. Reformers said that using student test scores to
evaluate teachers, to reward and fire teachers, and to target
schools for closure would improve the quality of teachers in
low-income schools. But instead, narrow, unreliable met-
rics turned off great teachers, increased churn, and drained
experience from teacher pools, with no boost to student
achievement.
District of Columbia Public Schools IMPACT system, which bases
teacher evaluations (and dismissals) heavily on
test scores, is associated with higher teacher turnover. The
share of DCPS teachers leaving after one year increased
from 15.3 percent in 20012007 (before IMPACT began in 2009) to
19.3 percent in 20082012; the share leaving
after two years increased from 27.8 percent to 33.2 percent; the
share leaving after three years increased from 37.5
percent to 42.7 percent; and after four years fully half (52.1
percent) of teachers left the system, up from 45.3 per-
cent (Levy 2012g). Few teachers reach experienced status,
generally considered at least five years and, by some
experts, seven years or more.
School closures did not send students to better schools or save
school districts money. Reformers closed schools
deemed failing so students could transfer to better-performing
schools. But most students whose schools were closed
went to other schools with even lower test scores, and the
disruption (some had multiple moves) was exacerbated by
longer commutes and spikes in gang violence as established lines
were crossed.
Although Duncan closed Chicago public schools deemed
underperforming in order to move students to better
schools, the closings had almost no effect on student
achievement because almost all displaced elementary school
students transferred from one low-performing school to another,
according to a study of 18 schools closed between
2001 and 2006 (de la Torre and Gwynne 2009). Only the 6 percent
who moved to better schools with greater
resources had improved outcomes.
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
4
-
Charter schools further disrupted the districts while providing
mixed benefits, particularly for the highest-needs
students. Reformers say charters offer better options and
outcomes for students in failing public schools. But charter
outcomes in these cities and across the country are uneven.
Charters serve fewer of the highest-needs students than do
regular public schools and can disrupt school districts
logistically and financially. High-performing charters may also
spend more per student than regular public schools.
The Chicago Public School (CPS) system uses its own value-added
metric to measure school performance, with
schools scoring lower on the distribution identified for
closure. By this measure, if students in the types of schools
most likely to be closed moved to charters, they would move to
lower-performing schools. Specifically, students
who moved from high-poverty regular public schools at the 47th
percentile in performance would go to charter
schools at the 40th percentile, and those moving from intensely
segregated schools at the 43rd percentile would
end up in charters at the 33rd percentile. Random-lottery
enrollment schools, which, unlike charter schools, do not
select out students via a challenging application process,
outperform their demographically comparable charter
counterparts: Students who moved to charters would drop from the
52nd percentile to the 40th percentile (Caref
et al. 2012).
Emphasis on the widely touted market-oriented reforms drew
attention and resources from initiatives with greater
promise. Less-publicized strategies for boosting student
achievement were piloted in these cities but not widely replic-
ated or expanded to scale because leaders and funders focused on
the market-oriented reforms. These promising but
overlooked reforms are more multifaceted and holistic than
reforms that seek quick fixes and rely on narrow, unreli-
able metrics.
Between 2003 and 2005, New York Mayor Michael Bloomberg created
100 new, small, and reportedly academically
rigorous secondary schools across the citys five boroughs
(Goldstein 2010). These schools conferred many benefits
for at-risk students, including a reported 10 percentage-point
increase in the share of ninth-grade students on track
to graduate (from 48.5 percent to 58.5 percent) and a 7.6
percentage-point increase in Regents exam scores indic-
ating college readiness in English. Among the added resources
small schools delivered were community partners
providing students with relevant learning opportunities inside
and outside the classroom and providing school fac-
ulty with additional staffing support and start-up resources.
For example, a law firm offered internships to students
of the Urban Assembly School for Law & Justice and helped
staff integrate real-world examples into the curriculum.
Another partner helped the Urban Assembly New York Harbor School
grow one of the largest oyster beds in the
region (Bloom and Unterman 2012, 2). Extra staff and other
resources also enabled the schools to nurture the
strong personal teacher-student interactions that likely
contributed to the academic gains.
Duncan worked to improve the college readiness of low-income and
minority students in Chicago by increasing
their access to AP courses, providing high schools with college
counselors, and holding principals accountable for
ensuring that students applied for financial aid. Scholarship
aid nearly doubled in one year (20112012). Budget
cuts have since removed counselors from almost all schools.
Just before Rhee became chancellor, DCPS expanded its
high-quality, full-day prekindergarten program to serve
middle- and upper-income as well as low-income 3- and
4-year-olds whose parents requested seats for them. The
district also adopted a holistic, hands-on curriculum designed
to nurture healthy development of all early childhood
domains: cognitive, emotional, physical, and behavioral. Data
showed that third-graders who had participated in
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
5
-
the program had higher test scores than their nonparticipating
peers. However, Rhee implemented a policy that
required parents to enter a lottery to get a spot for the coming
year, limiting access. And pre-K is not even on the
agenda of StudentsFirst, the well-funded education reform
organization founded by Rhee to promote the reforms
she says worked in Washington, D.C.
The reforms missed a critical factor in achievement gaps: the
influence of poverty on academic performance. In
all three cities, a narrow focus on market-oriented policies
diverted attention from the need to address socioeconomic
factors that impede learning. In 2010, student eligibility rates
for free- and reduced-price meals were 67 percent in
Washington, D.C., 72 percent in New York City, and 77 percent in
Chicago (NAF 20092010). Clearly child poverty
has been a significant factor contributing to low student test
scores and graduation rates in these three cities. Failing
to provide supports that alleviate impediments posed by poverty
ensures continued low student test scores and gradu-
ation rates, and large gaps between average test scores of white
and affluent students and test scores of minority and
low-income students.
Real, sustained change requires strategies that are realistic,
patient, and multipronged. In each city, the initiatives
showing more promise than the touted reforms demonstrate that
achievement gaps can only be closed when the
opportunity gaps driving them are addressed. The hands-on
experiences and consistent, intensive teacher-student rela-
tionships of New York Citys small schools must replace reforms
test preparation, novice teachers, and churn. Heavy
reliance on college- and career-readiness test metrics should
give way to CPS-style college- and career-readiness sup-
ports: helping students choose courses and schools, access AP
classes, and match their skills with career goals; and hold-
ing schools accountable for scholarship applications. DCPSs
high-quality prekindergarten program, which is designed
to nurture all aspects of childrens development, should serve as
a model for all cities and students, not be sidelined in
reform agendas.
Districts that recognize the impact of poverty and address it
head-on find the greatest success. Though it is higher-
income, Montgomery County, Md., serves a student body that is as
ethnically diverse as any of these urban districts,
and has a large and growing share of low-income students. In
contrast to the reformers, however, Montgomery County
Public Schools (MCPS) Superintendent Joshua Starr staunchly
opposes using test scores to evaluate teachers, instead
employing a peer-assisted review system that focuses on teacher
support, development, and collaboration (Strauss 2012).
MCPS has no charter schools. Rather, it channels extra
resources, including targeted professional development for
qual-
ified teachers, the smallest classrooms, and intensive literacy,
to the neediest schools. It has developed a holistic, creative
curriculum to nurture in-depth, critical thinking. This includes
art, music, and physical education teachers in every
school. MCPS also leverages the countys mixed-use housing
policies to integrate schools (Schwartz 2012). Finally, it
employs high-quality prekindergarten, health clinics, and
afterschool enrichment to further close income-based oppor-
tunity gaps. As Starr highlights, all of this has produced some
of the highest test scores among minority and low-income
students of any district, smaller and shrinking achievement
gaps, and high school graduation and college attendance
rates that are the envy of the country (Starr 2012).
Every school district has unique needs and resources. But
providing all students with the enriching experiences that
already help high-income students thrive would represent a big
step forward, and away from narrow reforms that miss
the mark.
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
6
-
The reformers and their claims
A specific set of policies labeled reforms have dominated
education policy discussions in recent years. Self-identified
reformers such as former New York City schools chancellor Joel
Klein, former Washington, D.C., school chancellor
Michelle Rhee, and Secretary of Education Arne Duncan, emphasize
three reforms, in particular, as the answer to the
declining student achievement and widening race- and
income-based achievement gaps that they assert represent the
state of American education:
1) The use of student test score data to make decisions about
hiring, firing, and rewarding teachers and principals.
2) The use of student test score data to target failing schools
for turnaround and/or closure.
3) The expansion of student and parent choice about schools, in
the form of increased access to charter schools (often
as replacements for neighborhood schools closed for
underenrollment).
Although student achievement is in fact increasing, and
race-based achievement gaps have been shrinking, these market-
oriented reforms have become embedded in federal policies and
organized advocacy efforts to change how school sys-
tems and districts operate. Under the Obama Administrations
signature Race to the Top (RTTT) initiative, states must
establish plans to enact these reforms in order to attain
grants. States seeking waivers from some of the requirements of
the federal No Child Left Behind (NCLB) program must undertake
dramatic efforts to turn around the lowest-per-
forming schools, and their plans, too, usually rely on these
reforms.
These same reforms represent the basic policy platforms of a
number of well-funded education reform organizations,
including StudentsFirst (founded by Rhee), Stand for Children,
50CAN, and Democrats for Education Reform
(DFER), all of which have state affiliates across the country.
Combined, these efforts represent a concerted push to
establish this set of reforms as the new norm. In its mission
statement, StudentsFirst states its intent to build a
national movement and demand that legislators, courts, district
administrators, and school boards create and enforce
policies consistent with these reforms (StudentsFirst 2012). And
its recent State Policy Report Cards ranked every
This paper compares and contrasts reformers promises and claims
of improvement with actual outcomes.
Use of reformer claims as benchmarks does not indicate and
should not be interpreted as endorsement or
adoption by the Broader Bolder Approach to Education (BBA) of
those benchmarks. For example, reformers
assert that using student test scores to evaluate teachers and
take action based on those evaluations can
improve student outcomes, and even turn around struggling
schools in relatively short periods of time. BBA
does not believe that this is true, nor that truly troubled
schools can turn around in a sustained manner in
a short period or as the result of isolated reforms. Nor does
BBA endorse any specific college or career read-
iness standard(s). Similarly, though student test scores are
discussed in the report, because they are widely
employed as a key measure of student achievement, as discussed
below, BBA does not believe that they can
be reliably used to measure student skill level or growth over
time, but rather that NAEP scores are the
appropriate benchmark for such analyses.
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
7
-
state and the District of Columbia on the degree to which it
implemented them, finding the majority severely lack-
ing (StudentsFirst 2013). Indeed, with StudentsFirst targeting
races for statewide public office, 50Can mounting state
policy campaigns, Stand for Children working at the local school
board level as well as in statehouses, and DFER advoc-
ating local, state, and federal policy changes, policies
consistent with these reforms are being promoted and adopted in
districts and states across the country (Cavanagh and Sawchuk
2012). Further evidence of their growing hold is seen in
the number of consulting companies that employ them as the focus
of their advising efforts on behalf of these states,
cities, and districts.
The persistence of problematic achievement gaps between students
of different races and income levelsgaps particu-
larly afflicting students in poor urban schools and districtsis
not in dispute. Indeed, 50 years of rising income inequal-
ity has accompanied a rapidly widening income-based achievement
gap, which has eclipsed the race-based gap (Reardon
2011). The question is how we close these gaps.
Some reformers position their policies as higher minded than the
policies advocated by others. Rhee and Klein advance
a no excuses response to those who say poverty is an impediment
to education, and frequently label those with whom
they disagree as defending the status quo (StudentsFirst 2011).
Others, such as Duncan, acknowledge the impact of
poverty and promote a larger range of policies, while still
emphasizing the same core set of reforms. But the question
most critical for the millions of at-risk students and their
familiesand the nation as a wholeis not whether one
group or another is reforming or making excuses, but what works
and what does not.
This paper addresses that question directly. First, it reviews
the reformers claims against the body of academic research
devoted to each of these reform policies. As with most areas of
policy, the evidence is mixed. Some of the policies
are more promising than others, but none is even close to
securing definitive endorsement based on the data. Indeed,
policies that are tested in the social science equivalent of a
laboratory inevitably vary, depending upon the laboratory
employed (and many other variables), resulting in different
outcomes in different cases.
Second, and of greater significance, the paper assesses these
policies by reviewing their actual impact in three major cit-
ies. Over the past decade, different iterations of this set of
reforms were implemented in Washington, D.C., New York
City, and Chicago. The school districts in all three cities were
controlled by their mayors, and thus had the potential
to implement the reforms fully and flexibly. The outcomes in
these three cities can therefore be seen as the best that a
district enacting similar reforms could expect, with the fewest
possible barriers to implementation.
Reliable National Assessment of Educational Progress data on
changes in test scores, and thus achievement gaps, as well
as somewhat comparable high school graduation rates, are
available for all three citys school districts, as are data
from
other large, urban districts that did not employ the same
reforms, or that did so to a much lesser degree, providing a
basis for comparison. Unlike city or state test scores, the NAEP
data are comparable across districts in different states,
thoroughly measure student learning and knowledge, and are
sampled, and thus not subject to the system-gaming that
can affect state test score data. Importantly too, NAEP data are
available for a representative group of all students, and
for minority and low-income students, the subgroups purported to
benefit from these reforms. Specifically, the report
explores how the reforms in each city have influenced:
Teacher turnover and the stability of students educational
contexts
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
8
-
Student outcomes as measured both by disaggregated NAEP scores
in math and reading and by the cities or states
own assessments
High school graduation rates
School budgets
Although cost savings was not a promise, or goal, of reformers,
tracking changes in school budgets is important, because
public officials in many of the other cities and states that
have recently endorsed similar reformsincluding Cleveland,
Detroit, and Philadelphiasay these policies will not only
improve student outcomes but also save money and, thus,
help close major deficits. For example, the Boston Consulting
Groups plan, commissioned by the Philadelphia School
Reform Commission, states, Faced with both a financial crisis
and persistently unsafe schools with low student achieve-
ment, the Philadelphia public schools are undergoing a
transformation in order to ensure that all students have access
to safe, high-quality schools, while bringing the budget back
into balance. A spreadsheet detailing the financial aspects
of the plan asserts that the budget will be balanced by 2014
(School District of Philadelphia 2012, 2 and 28).1
The findings from the three cities do not show that the reforms
have succeeded in fulfilling either their backers
promises or their claims. While in each city the mayor or
chancellor reported large increases in test scores and/or
decreases in achievement gaps since reforms were implemented,
the improvements were often refuted soon after they
were announced. In these districts, most test scores have not
increased any more than in other, nonreform districts or
than in the period predating the new policies; some test scores
have fallen; and achievement gaps by race and income are
as wide or wider than they were prior to the reforms. New
teacher evaluation systems based heavily on student test scores
seem to have increased teacher turnover, rather than improved
teacher quality. Graduation rates among low-income and
minority students have improved somewhat, narrowing citywide
gaps in those rates, though it is not entirely clear that
these cities have made gains in graduation rates that are
greater than those in other cities.
Moreover, these reforms have had significant costs. All three
cities received substantial private donations and enacted
major budget increases, with New York City virtually doubling
education spending (in real dollars) in just a decade.
While the reforms are not the main cause of the increases, some
aspects have introduced new costs. Moreover, the costs
raise questions about the viability of the reforms today in
other cities facing major budget shortfalls, and the sustainab-
ility of the reforms in these three cities.
At the same time, officials in these cities have implemented
other new policies that produced positive results. New
schools with a focus on real-life experiences and strong
teacher-student relationships have improved outcomes for high-
risk students. Smarter teacher recruitment efforts have brought
qualified educators to hard-to-serve schools. Enhanced
and expanded prekindergarten (pre-K) programs may be boosting
low-income students elementary school achieve-
ment. However, these promising policy changes have largely been
ignored. Because of the excessive focus on promin-
ent reforms, less visible programs with real potential to narrow
achievement gaps havent had the kind of support or
resources they need to succeed or expand.
In short, some reform elements are working, but most are not.
And policies that address students noncognitive needs
are lacking. But multidimensional, nuanced efforts with a
long-term trajectory and mixed outcomes can be difficult to
communicate when the urgency to implement quick fixes prevails.
As such, reformers have mostly advanced simpler
messages that fail to tell the real story. Districts, states,
and the federal government should take a hard look at the grow-
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
9
-
ing evidence to ensure that what actually works gets replicated,
and that popular but ineffective or counterproductive
reform policies do not become the new status quo.
A brief synopsis of the evidence regarding core components of
thereform agenda
An American University forum in Washington, D.C., in March 2012,
Education Now: Cities at the Forefront of
Reform, promoted the basic tenets of the reform movement.
Representing the federal government and two of the three
cities featured in this report were U.S. Education Secretary
Arne Duncan, Chicago Mayor Rahm Emanuel, and New
York City Mayor Michael Bloomberg, along with their respective
superintendents. While Washington, D.C., leaders
did not participate, Los Angeles Mayor Antonio Villaraigosa and
Los Angeless schools superintendent were there, rep-
resenting a district without mayoral control (American
University 2012). The participants attributed a decline in U.S
global competitiveness to unions protection of poor teachers, an
insufficient emphasis on STEM (science, technology,
engineering, and math) instruction, and a failure to prepare
students for at least a two-year college. To correct these
deficiencies and others, they advocated for more
school-performance data to be used to eliminate what they
termed
low-quality schools (and to enable parents to make informed
choices among regular district schools and an expanded
number of charter schools) and greater use of test scores to
assess teachers and principals effectiveness and make hiring
and firing decisions.
All three leaders asserted that this set of changes would
improve student achievement and narrow race- and income-
based achievement gaps.2 As the literature review below
illustrates, however, these claims are exaggerated in some
respects and false in others. There is no sound theoretical
basis for the premise that market-oriented reforms can fix
income-correlated gaps in educational attainment.
Test-based accountability systems
Bloomberg was perhaps the strongest proponent among the
panelists of test-based accountability. Whereas Duncan
acknowledged that No Child Left Behind had led to excessive
reliance on testing, Bloomberg defended the practice of
teaching to the test as reflecting state standards and
curriculum. All the mayors, along with Duncan, emphasized fir-
ing bad teachers as a key means of improving quality.
If you care about education, you have to care about the quality
of the teachers and get those out of the system who
cannot do the job, Bloomberg said (American University
2012).
Research does not show that it is effective or reliable to rely
on student test scores to evaluate, fire, or provide merit
bonuses to teachers (Baker et al. 2010). First, test scores are
influenced by myriad factors, both in and out of school,
most of which are beyond the ability of teachers to influence,
but which nevertheless lead to judgments regarding teach-
ers effectiveness (Rothstein, Jacobsen, and Wilder 2008).
Even value-added methods (VAM), which try to account for the
impact of student background characteristics on test
scores, inevitably fail to capture many factors beyond teacher
control. Though they use a complex statistical technique
called regression analysis, which incorporates multiple factors,
value-added measures still have many flaws. The initial
development of value-added systems in the 1990s enabled
evaluators to move beyond the use of raw student scores,
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
10
-
which only reflect a students knowledge at a given point and do
not capture change by taking into account where he
or she began. Value-added methods correct for that problem by
using two tests, one from the current year and another
from the prior year, or, one from the beginning and another from
the end of the school year, thus capturing student
growth over a period. Also, some versions of the regression
analyses control for easier-to-measure individual student
characteristics that are associated with learning and
achievement, such as use of free-and-reduced-price lunch, and
race
and/or ethnicity. Nonetheless, value-added methods still employ
narrow tests as the only measure of student knowledge
and skill, incorporate only a very limited number and range of
out-of-school factors, and cannot capture many of the
other in-school influences, such as tutors, team teaching, or
counseling, that can influence student scores as much as, or
even more than, the individual teacher being assessed (Weiss
2011).3
As a result, value-added scores are problematic. After New York
City published value-added scores for every teacher in
the city in 2012, veteran teachers considered by parents and
principals as among the citys best were depicted as poor
performers, scores shot up and down from year to year, and many
of the scores were found to be wildly inaccurate and
based on incorrect data (see, for example, Pallas 2012a and
Winerip 2012). According to the Board on Testing and
Assessment of the National Research Council of the National
Academy of Sciences, VAM estimates of teacher effect-
iveness should not be used to make operational decisions because
such estimates are far too unstable to be considered
fair or reliable (BOTA 2009).
Second, basing important decisions on student test scores has
negative consequences. Teachers are driven to teach to
the test, adjusting instruction to cover material on the test or
even teaching test-taking skills rather than content in
order to produce the higher scores demanded (Baker et al. 2010;
Scherrer 2011; Rothstein, Jacobsen, and Wilder 2008).
Test-based accountability systems also give teachers incentives
to choose classes and students who are likely to score
higher, and/or to gain more from one year to the next, and,
perhaps most troubling, to avoid serving students who are
less likely to do well on tests (Peneston et al. 2011).4 As
such, some experts caution that value-added methods and other
test-based teacher evaluation systems may lead to the loss of
qualified and strong teachers in hard-to-staff schools and
districts (Weiss 2011). At the extreme, some observers note that
the increasingly high stakes attached to raising student
test scores have led to a growing number of district-wide
cheating scandals (FairTest 2012).5
Proponents of using student test scores to award merit-pay
bonuses to teachers assert that current salary systems provide
teachers with insufficient incentive to focus on improving
student outcomes, and that bonuses for raising test scores
would thus improve teaching and student outcomes. The same
shortcomings of VAM in other contexts apply to test-
driven merit-pay systems, in particular the failure of VAM to
account for factors influencing test scores that are beyond
teacher control, and the added incentive to narrow instruction
to test content in order to raise scores.
Experiments assessing test-driven merit-pay outcomes have been
conducted in several sites, but none has yet found
these systems effective. One large pilot conducted in the
Metropolitan Nashville Public Schools over three school years,
20062009, was designed to avert the inadequate pay and rigid
strategies blamed for failures in other, prior pilots. It
offered large bonuses, from $5,000 for a small test-score bump
to $15,000 for raising scores to the very top of the
continuum (Springer et al. 2010). The structure of the
experiment also allowed teachers to employ any strategy they
chose to raise scores, from obtaining more professional
development or coaching to collaborating with others. Despite
the thoughtful design and major investment, however, the bonuses
produced no improvement in student test scores
(Springer et al. 2010). Moreover, teachers surveyed did not
believe that those who received bonuses were better teachers.
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
11
-
These findings were supported by a RAND evaluation that found
that New York Citys experimental program had no
impact on student test scores. According to the authors, the
results of this evaluation add to a growing body of research
from the United States that finds no effects on student
achievement of narrow pay-for-performance policies that focus
only on financial incentives without other features, such as
targeted professional development or revised teacher evalu-
ations (Marsh et al. 2011).
A different model funded by a federal grant and employed in
Chicago downplayed individual teacher rewards and
spread bonuses throughout the schools staff, instead focusing on
other aspects of teacher support to boost student
achievement and teacher retention. Nonetheless, results at the
two-year mark were disappointing: Neither test scores
nor teacher retention had improved (Sawchuk 2010).6 At four
years, there was some increase in school-level teacher
retention, but still no impact on student achievement (Glazerman
and Seifullah 2012).
In his 2009 best-seller Drive, labor policy expert Daniel Pink
contends that merit pay programs do not induce teachers
to improve their instruction because, unlike corporate
executives, teachers intrinsic motivation for entering the
profes-
sion, and for improving student outcomes, is not fundamentally
rooted in salaries or compensation. Moreover, because
the field requires conceptual and creative thinking, a
simplistic reward-for-higher-performance mechanism does not
apply (Pink 2009).
Research confirms Pinks suggestion that recruiting good teachers
to low-income and struggling schools and retaining
them relies much more heavily on their working conditions than
on salaries, or students income or background. The
findings from a recent Harvard Graduate School of Education
study show that guaranteeing an effective teacher for
all studentsespecially minority students who live in
povertycannot be accomplished simply by offering financial
bonuses or mandating the reassignment of effective teachers
(Moore Johnson, Kraft, and Papay 2012, 18).
School closures
Another facet of the school reform agenda is the intentional
closure of neighborhood schools. School closures can be
driven by the desire either to reduce district spending (by
shuttering an underutilized building) or to transfer students
from a failing school to one that is higher-performing, and thus
able to provide better educational opportunities.7
Studies of school closures show that both premises are flawed.
First, the determination that a school is underenrolled is,
itself, subject to manipulation, as explored in the section of
this paper reviewing real-life outcomes. Also, as discussed in
more detail later in this paper, purportedly underutilized
neighborhood schools are often converted to charter schools
that serve even fewer students than before conversioncharter
schools which are not deemed underenrolled.
The premise for closing failing schools, often targeted for
closure based on student test scores, is that many of the
teachers, the principal, and other aspects of the school are too
problematic to remedy, so improvement must start from
scratch.8 Here, too, the research suggests that the policy does
not improve student outcomes overall and may even
exacerbate existing problems. A 2009 study of students from 18
Chicago elementary schools closed between 2001
and 2006 found no overall effect on those students academic
performance, but raised a concern that since most stu-
dents transferred to very low-performing schools, while only 6
percent went to high-performing schools, the disruption
offered no benefits (de la Torre and Gwynne 2009, 16). Using
data from an unnamed high school in a large urban
school district in the western United States, a 2010 study
assessed the academic performance and experiences of Latino
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
12
-
and African American high school students in the year following
their schools closure (Kirshner, Gaertner, and Pozzo-
boni 2010). Among the findings were declines in academic
performance and added stressors [for] students who were
already contending with challenges associated with urban
poverty. And a recent study of a district with declining
enroll-
ment that used student achievement to target schools for closure
found adverse effects on test scores and attendance,
though the study noted that these effects could be minimized by
transferring students to higher-performing schools
(Engberg et al. 2011).
In all, the studies point to a critical limitation: Districts
that use low student test scores to label schools as failing
and
close them are unlikely to have a sufficient number of
high-performing schools to serve transferred students. As such,
even if students could, hypothetically, benefit from such moves,
beneficial moves may not be possible in many, if not
most, current contexts.
Increasing the reach of charters
Expanding access to charter schoolsframed as key to parental and
student choicewas another major priority for the
panelists of the March 2012 American University forum, and a
common focus of reforms in the three cities assessed
in this paper. However, assertions that charter schools improve
educational outcomes are not supported by rigorous
studies. Research findings on the effectiveness of charter
schools in boosting achievement for low-income students and
in narrowing achievement gaps are mixed at best. The largest and
most comprehensive study to date was conducted in
2009 by the Center for Research on Education Outcomes (CREDO) at
Stanford University. Multiple Choice: Charter
School Performance in Sixteen States analyzed outcomes of
charter school students in 70 percent of all charter schools in
the United States by using simulations to compare them to their
virtual peers in the traditional public schools they
would otherwise have attended. The report found wide variation
in performance. In particular, math test scores were
better for 17 percent of charter school students, were the same
for roughly half of charter students, and were worse for
34 percent of charter students. In other words, twice as many
students lost as gained from being in a charter school, a
discovery that the researchers call sobering (CREDO 2009b, 3).9
Reading scores in charter schools versus traditional
public schools were similarly mixed, with gains differing in
elementary versus middle and high schools, across ethnic
groups, and in first versus later years in the charter
school.
Charter schools are public schools, and the vast majority of
their funding comes from public sources. As such charter
schools, like other public (and private) schools, vary
tremendously in terms of resources, teacher qualifications,
school
day and year, services provided, and a host of other factors. In
the case of charters, these factors include state policies
authorizing, funding, and monitoring them. In addition, unlike
regular public schools, some charters, like KIPP
schools, employ established, tested models, while others are
newly minted and employ practices that have not been
assessed.10 While most are nonprofits, some are for-profits.
Given this variation across a range of factors, it is not sur-
prising that studies of specific types and brands of charters,
and of charters in particular cities and states, have found
substantial variation in their success, relative to comparable
traditional public schools.
New York and Ohio illustrate many of the substantial divides in
state-level findings. A study of New York City charter
schools by the same Stanford researchers found significant
benefits for most groups of students who enroll, including
low-income, minority, and special-education students, though
none for English language learners (ELL) nor for those
who were retained in grade (CREDO 2010). Conversely, the
Stanford study found that charter schools in Ohio, a
national leader in the authorization and opening of new charter
schools, performed significantly worse than comparable
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
13
-
public schools (CREDO 2009a). A second in-depth study of the
history of charter laws and schools in Ohio found that
the states charter schools were plagued by mismanagement,
financial misconduct, and lack of accountability (Dinger-
son 2008, 17).11 These disparities mirror the broader findings
in the multi-state 2009 CREDO study, which discovered
substantial variation across states over and above existing
differences among states in their academic results (CREDO
2009b, 3).12
These findings, in turn, should inform broader discussions
regarding the role of extended learning time, enrichment
activities, prekindergarten (pre-K) and other early childhood
programs, health clinics, the mix of students served, and
other features of charter schools that may differ from their
traditional public school counterparts and which are known
to influence student outcomes. Such discussions are critical
both to understanding why and which charter schools out-
perform their regular public school counterparts and to guiding
changes in education policy irrespective of the type
of school.
One consequence of the expansion of charter schools in a school
district is a corresponding loss of students in the dis-
tricts regular or neighborhood public schools. Of course,
districts lose students for many reasons; for example, many
rural districts have lost students in recent decades as their
populations declined. The district public-to-charter school
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
14
-
shift, too, is often described as a decline in student
enrollment, though it often does not, as do rural losses, reflect
an
overall reduction in the number of students who are served in
the district.
The shift nonetheless presents problems for school districts, as
if a real decline had taken place. Because the number
of public school students determines district budgets, schools
that lose students to charters may have similar overhead
expenses but fewer dollars to pay for them (Dingerson 2008,
31).13 The way in which charters operate also can com-
plicate planning, hiring, and other logistical matters for
traditional public schools. In Ohio, for example, charter
recruit-
ment and closures during the school year mean more student
mobility, leading to disruption and lack of capacity.
In Dayton, where public schools begin two weeks earlier than
most charter schools, the district must staff the pub-
lic schools to accommodate all enrolled students for the first
weeks; when students enrolled in charter schools move
to those schools two weeks into the school year, the school
district has no way to pay the teachers who are no longer
needed. In the longer term, districts known to fire or attempt
to hire teachers midyear in response to outflows of stu-
dents to or influxes from charter schools have an increasingly
hard time recruiting high-quality teachers. A description
of how parents in Washington, D.C., scramble to get their
children into the right school at the start of the year, and
reserve slots at multiple schools until they choose one,
illustrates much the same problem posed by some aspects of
charter-based choice (Brown 2012b).
Turnover and instability
The asserted goal of test-based teacher evaluations, elimination
of bad teachers and principals, and closures of failing
schools is improved teacher quality and better student outcomes
(Rhee and Klein 2010; StudentsFirst 2011). As
Anthony Bryk and other scholars have found, however, the cake
mix required to turn around troubled schools is a
complex one, and requires, among other ingredients, stable and
cohesive teacher and leadership efforts that evolve and
cohese over time (Bryk et al. 2010). This makes sense; with so
much else unstable and uncertain in their liveshousing,
meals, parental attentionlow-income students need their schools
to provide that stability. New schools and new teach-
ers, especially when they become a pattern repeated over several
years, can further disrupt their already chaotic lives
(Hanushek, Kain, and Rivkin 2004).
If these reforms lead to instability (and the next section of
this paper shows that they do), they are producing a negative,
unintended consequence.
This is not to say that bad teachers should not be fired, that
teachers cannot do their jobs well after only a few years, or
that there is never a reason to close a school. However, a large
body of research explains the advantages of experienced
teachers over lower-paid novices, and of the importance of
continuity and stability in improving student outcomes (e.g.,
Haycock 2006; Holzman 2012).14 As Tom Carroll, president of the
National Commission on Teaching and Americas
Future, pointed out, research clearly shows that with each year
of experience, teachers improve their proficiency and
effectiveness during the first seven years. National Board for
Professional Teaching Standards (NBPTS) certification
demonstrates that many teachers are still gaining in proficiency
and improving their effectiveness after an average of 11
years of teaching (Carroll and Foster 2010, 12). In fact, a
recent study on the impact of teacher turnover concluded
that, distinct from the relative quality of teachers who may be
brought in to replace those who leave, teacher turnover
itself harms a school.15 Turnover lowers school morale and
professional culture, depletes the staff s store of knowledge
about students and the community, and impedes the collegiality,
professional support, and trust that teachers need to
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
15
-
improve student achievement (Loeb, Ronfeldt, and Wyckoff 2012).
In order for increased teacher turnover to improve
student outcomes, then, teachers who leave must be replaced by
teachers who are substantially more effective.
A no excuses take on the influence of poverty
A common refrain underpinning espousal of the reforms discussed
above is the dismissal of the impacts of child, family,
and community poverty on student achievement and school success.
When discussing their reform agendas, leaders
such as Michelle Rhee have employed a no excuses rhetoric,
asserting that any teacher who cites poverty as a driver of
student or school failure is excusing him or herself from the
responsibility to teach well. Indeed, Rhee alleges that, what
often happens when we start to talk about wraparound services
[that acknowledge the need to address povertys impacts]
is a lot of people start to give up responsibility (Maxwell
2012). Former New York City schools leader Joel Klein has
gone further, explicitly rejecting decades of research that
demonstrate the benefits of alleviating poverty-related
obstacles
to effective learning.
No single impediment to closing the nations shameful achievement
gap looms larger than the culture of excuse
that now permeates our schools. Too many educators today excuse
teachers, principals, and school superintend-
ents who fail to substantially raise the performance of
low-income minority students by claiming that schools
cannot really be held accountable for student achievement
because disadvantaged students bear multiple bur-
dens of poverty. The favored solution du jour to minority
underachievement is to reduce the handicap of
being poor by establishing full-service health clinics at
schools, dispensing more housing vouchers, expanding
preschool programs, and offering after-school services like
mental health counseling for students and parents.
America will never fix education until it first fixes povertyor
so the argument goes. In fact, the skeptics of
urban schools have got the diagnosis exactly backward. The truth
is that America will never fix poverty until it
fixes its urban schools. (Klein 2009)
But rigorous studies have revealed the many ways in which
growing up with limited resources and being raised by poorly
educated parents impede students development and academic
achievement (Rothstein 2004; Barton and Coley 2007;
Berliner 2009; Ladd 2011; and Yoshikawa, Aber, and Beardslee
2012). These include the large achievement gaps that
emerge long before children enter kindergarten (Galinsky 2006;
Isaacs 2012; Rolnick and Grunewald 2007). These
achievement gaps are exacerbated by gaps in nutrition and health
status and care that similarly divide along lines of eco-
nomic class and race (Rothstein 2004 and Berliner 2009). In
addition, loss of learning in afterschool hours and over the
summer account for large shares of income- and race-based
achievement gaps (Alexander, Entwisle, and Olson 2007).
The reforms have not strengthened school systems in the three
cities
As briefly summarized in the previous section of this report,
extensive research has assessed the effectiveness of the three
main components of the reform agenda: using standardized test
scores to make decisions regarding schools and edu-
cators; closing schools that are labeled failing and/or
underenrolled; and expanding access to charter schools to give
par-
ents and students greater choice. Results have been
inconclusive, and policymakers have used the findings to make
cases
both for and against these policies. As such, these studies have
led to renewed debate, rather than to consensus, regarding
which policies best improve outcomes for underachieving
students. Indeed, education policy has become increasingly
politicized, and discussions hostile, as advocates of these
reforms and their opponents cleave to their respective
positions
(Bushaw and Lopez 2012).
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
16
-
Cities looking to implement similar plans will thus find it more
informative to understand how these reform efforts
have actually worked in practice.
The main sections of the paper, below, assess the reforms
implemented by Michelle Rhee during her tenure as DC
Public Schools (DCPS) chancellor, with many of the same policies
in place under her successor, Kaya Henderson; by
Joel Klein when he was New York City Schools chancellor; and by
Education Secretary Arne Duncan when he was
Chicago Public Schools chief education officer (CEO), with his
policies largely followed by his successors Ron Huber-
man, Jean-Claude Brizard,16 and, it appears, new Chicago Public
Schools CEO Barbara Byrd-Bennett.
Rhee was appointed by Mayor Adrian Fenty in 2007, and she left
after he failed to win reelection in 2010. Klein served
from 2002, when he was appointed by Mayor Michael Bloomberg, to
2010, when he left to become an advisor to media
mogul Rupert Murdoch. As CEO of Chicago Public Schools (CPS),
Duncan worked with Mayor Richard Daley from
2001 through 2008 to implement Renaissance 2010market-oriented
reforms that emphasized school closings and
consolidations and charter schools (Lipman 2009).17
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
17
-
The publicly reportedstatistics used to holdschools and
districtsaccountable for mak-ing academic progressare not accurate
meas-ures of progress.
Chicago, New York, and Washington, D.C., are, to a large degree,
pioneers in implementing the three specific policies
assessed in this paper. While many other cities have instituted
some of the same policies or are beginning to do so,
none enacted them as comprehensively or with as much mayoral
backing, as these three cities. Indeed, Rhee, Klein, and
Duncan all cite mayoral control as one key to effective
education reform.
Thus, we can make a valid comparison between the reform cities
of Chicago, New York, and Washington, D.C., with
nonreform cities. For the purposes of this paper, the latter
include other large, low-income, urban districts for which
similar datain particular, test scores from the National
Assessment of Educational Progress, and graduation rates that
are comparable across districts have been collected in recent
years.
The problems driving achievement gaps are complex, and policies
designed to narrow them go beyond those explored in
this paper. Any leader of a major school district promotes
and/or institutes a variety of policies and programs, and makes
changes to many others. This paper does not assess all of the
policies and programs that these three districts enacted or
implemented during this period. It focuses on the three that
were core to these leaders tenures, and that they and other
reformers are most widely promoting and replicating elsewhere,
for that reason. It also, however, reviews certain smaller
policy changes that, while overshadowed by the big three, may
have contributed to the outcomes reported, and/or have
shown promise in boosting future achievement.
Note that this report tracks gains (and losses) based on NAEP
scores across the three cities during their periods of major
reform, and compares changes with trends in other large, urban
districts and nationwide. Data for large, urban districts
are from NAEP Trial Urban District Assessment (TUDA) reports
unless otherwise noted. TUDA uses NAEP scores
from a representative sample of students in large, urban
districts with high proportions of low-income and minority
students. TUDA began in 2002 with six districts and had 21 as of
2011.18
As mentioned earlier, the use of NAEP, versus city or state test
scores, allows
for apples-to-apples comparisons across cities in different
states, and makes
these comparisons more valid and reliable than many that are
commonly
cited; NAEP is a more thorough measure of student learning and
knowledge
than many state tests, and it is sampled, and thus not subject
to the system-
gaming that can affect test scores. The report also notes scores
on other tests,
since they are often used as the basis for claims of gains in
specific schools or
types of schools. That non-NAEP tests should be viewed with
caution was
underscored in a study of CPS reforms by the Consortium on
Chicago
School Research.
Many of the findings in this report contradict trends that
appear in publicly reported data. For instance, publicly
reported statistics indicate that CPS has made tremendous
progress in elementary math and reading tests, while
this analysis demonstrates only incremental gains in math and
almost no growth in reading. The discrepancies
are due to myriad issues with publicly reported dataincluding
changes in test content and scoringthat make
year-over-year comparisons nearly impossible without complex
statistical analyses, such as those undertaken for
this report. This leads to another key message in this report:
The publicly reported statistics used to hold schools
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
18
-
and districts accountable for making academic progress are not
accurate measures of progress. (Luppescu et al.
2011, 5)
As this report details, when the market-based policies at the
center of the reform agenda play out in a comprehensive
manner across many years, the results, as captured in reliable
data, are not encouraging. This evidence should give pause
to supporters of this popular set of reforms, including
policymakers who are implementing similar policies. Reforms
that produce a lack of progress on improving test scores or
closing achievement gaps are no different from the status
quo that they purport to break.
This section of the report also examines data on school district
budgets over the period of the reformers tenures. While
the reformers in the three cities that are the focus of this
paper have not claimed such savings as a benefit of reforms,
other reform advocates make this promise. It is therefore
important to understand not only the potential of these reform
strategies to improve student outcomes, but the degree to which
they can, indeed, provide such savings without doing
further harm to students.
Washington, D.C.: Firings and closures increased churn
During Michelle Rhees three-and-a-half years as chancellor of
the DC Public Schools, several hundred teachers were
fired. Many more were laid off or left voluntarily, along with
dozens of principals, increasing the rate of turnover in
the schools, especially those serving the Districts
lowest-income students. Closing underenrolled and
low-performing
schools further increased churn for those students without
visible benefits or budget savings.
Test-based teacher evaluations and turnover. In 2009, DCPS
instituted a system, called IMPACT, to use student test
scores as a basis for evaluating individual teachers, in order
to reward those found to be effective or highly effective
and fire those deemed ineffective or minimally ineffective.
Under Rhee, 50 percent of a teachers IMPACT score was
based on his or her students test scores on the DC Comprehensive
Assessment System (DC CAS), the districts stand-
ardized test, which is given each April. The remaining 50
percent was based heavily on classroom observations, with a
small portion devoted to collaboration and professionalism (DCPS
2011). Complaints about the system from teach-
ers and others led to several changes under Rhees successor,
Kaya Henderson, including allowing some teachers who
received minimally effective ratings to keep their jobs, and
exempting teachers with two consecutive highly effective
ratings from three of five annual observations (Huffington Post
2011). More recently, Henderson reduced the weight
accorded to value-added scores from 50 to 35 percent of the
total IMPACT rating (Gartner 2012b).
Prior to IMPACT, DCPS had in place a rarely used 90-day plan
that allowed principals to place teachers in a semi-
probationary status that required that teachers develop a
remedial plan, be observed by the principal while they taught,
and fulfill other requirements or face rapid dismissal (Turque
2008). Rhee employed this plan to eliminate a small num-
ber of teachers before IMPACT was implemented.
Turnover, which generally is high among urban and low-income
schools and has been identified as a key impediment
to student success, increased sharply after mayoral takeover and
Rhees arrival in 2007, and the subsequent institution
of market-oriented reforms, as illustrated in Figure A. While
the high rate of turnover was to some extent inten-
tionalRhee announced when she was hired that many weak teachers
would have to leave in order to improve the
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
19
-
FIGURE A
District of Columbia Public Schools teacher attrition rates
before and after mayoraltakeover in 2007
Source: DCPS teacher retention data, 20012012 (Levy 2012g)
Years on the job before leaving
Perc
ent l
eavi
ng D
CPS
15.3%
27.8%
37.5%
45.3%
19.3%
33.2%
42.7%
52.1%Before mayoral takeover (20012007)
After mayoral takeover (20082012)
1 year 2 years 3 years 4 years0
20
40
60%
systemit increased the share of the teaching pool consisting of
novice teachers, who lack the experience needed to
boost instructional capacity.
The share of teachers leaving after one year in the system
increased from 15.3 percent in 20012007 to 19.3 percent in
20082012; the share leaving after two years increased from 27.8
percent to 33.2 percent; the share leaving after three
years increased from 37.5 percent to 42.7 percent; and the share
leaving after four years increased from 45.3 percent
to 52.1 percent (Levy 2012g). Among the districts charter school
teachers and new teacher hires, the rates were much
higher: Over half of new teacher hires left in their first two
years, and eight of 10 were gone by the end of their sixth year
(Simon 2012). While there are no perfectly comparable data from
other cities, these numbers suggest that DCPS has
one of the nations highest new teacher turnover rates. Scholars
estimate that, on average, 30 percent of new U.S. teach-
ers leave the profession after five years (Ronfeldt et al.
2011). That rate is as high as 45 percent in the
highest-poverty
school systems (Hunt and Carroll 2003).19
While Rhee made a point of firing teachers deemed ineffective,
the higher attrition rate cannot be attributed largely
to teacher terminations. Comparisons of data on teacher
termination with reports on the changes in DCPS and news
articles all suggest that most teachers left of their own
volition.
At the end of Rhees first year, 20072008, 350 of the systems
roughly 4,000 teachers, or about 8 percent, were ter-
minated, for reasons that were not clearly specified (Levy
2012h).20 Then, in 2009, Rhee fired 131 teachers for poor
Beforemayoraltakeover
(20012007)
After mayoraltakeover
(20082012)
1year 15.3% 19.3%
2years 27.8% 33.2%
3years 37.5% 42.7%
4years 45.3% 52.1%
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
20
-
appraisals under the 90-day plan that preceded IMPACT (Levy
2012h). She also fired 117 teachers for lack of licen-
sure. In July 2010, Rhee fired 165 more teachers who had
received ineffective ratings under the new IMPACT system
and announced that hundreds of others rated minimally effective
had one year to improve their performance before
facing dismissal (Turque 2010c).
In 2011, Rhee fired 65 teachers who had been rated ineffective
and another 141 who had been rated minimally effective
for two years in a row. These terminations for ratings of
ineffective or minimally effective accounted for roughly 3
percent, 4 percent, and 5 percent of the total DCPS base
teaching force in 2009, 2010, and 2011, respectively (Levy
2012h). With attrition rates of 20 percent or more for
first-year teachers and higher among the overall pool of
teachers,
the vast majority of exits, then, were due to other factors.
Altogether, around 1,000 educators were fired during Rhees
three-and-a-half-year tenure, about half for ineffective rat-
ings, and many of the rest ostensibly for budgetary reasons. For
example, in July 2009, the D.C. Council cut the DCPS
budget, directing that most of the money be taken out of summer
school, which would not take effect until the next
year. Rhee disobeyed the councils line item authority and
instead laid off 266 teachers. Rhee later suggested that she
had taken advantage of the budget problem to fire teachers who
had hit children, had sex with students, or had other-
wise seriously broken the rules. When questioned by policymakers
about why such teachers had not already been fired,
Rhee offered some details regarding what she said were eight of
the 266 teachers, but did not provide either back-up
information or names (Perdido Street School 2010). At the same
time, DCPS continued hiring; between March 2009
and January 2010, DCPS hired 2,211 new employees, including
1,034 teachers. Three quarters of these new hires took
place in August, just before Rhee fired the 266 teachers (Levy
2012e). Then-Council Chair Vincent Gray pointed to
DCPSs decision to hire hundreds of new teachers just before
firing the nearly 300 others, suggesting that Rhee and
Fenty used the budget as an excuse to replace experienced
teachers with lower-paid novices (Turque 2009).
Other data show that turnover affected some students more than
others. Levy has calculated individual DCPS school
attrition data, which show 30 percent, 40 percent, and even
higher percentages of teachers leaving in a single year, such
that in some schools, the majority of teachers are new at the
start of the school year. For example, at Anacostia High
School, where almost the entire staff (88 percent) was new,
having been specially selected in fall 2010, one third left
the following spring (Levy 2012f ). This happened despite Rhees
appointment of a charter management organization
to run the school and improve its performance. Another
distressed high school, Ballou, lost two-thirds of its teachers
over just two years (20102012). While not as astronomical,
attrition rates in several less-distressed schools were still
quite high. Bancroft Elementary in Mount Pleasant, which serves
a high proportion of Latino students, had a two-year
teacher turnover rate of 40 percent, and Barnard Elementary
School, housed in a beautiful new building in the rapidly
gentrifying Petworth area, lost 45 percent of its teachers in
two years (Levy 2012f ).
These high turnover rates harm students, especially the
low-income students concentrated in most DCPS schools. Mary
Levy, a former school finance litigator who has been a
consultant for the districts city council, noted the negative
impact
of these high rates on both students and the systems budget: We
lose half of our new teachers, our new hires within
two years of their hiring. This is, of course, expensive because
it costs money to recruit. We have a lot of teachers in
their first three years of teaching. We have a lot at the high
end, [too], and very few in between. This does not bode well
for the future (Levy 2011a).21
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
21
-
The National Commission on Teaching and Americas Future attached
a rough price tag to this high level of churn,
using data on the actual cost to the district of time and money
spent on activities associated with teacher leavers
including: recruitment and advertising; special incentives;
administrative processing, training for new hires, first-time
teachers, and all teachers; learning curve, and transfer
(Barnes, Crowe, and Schaefer 2007, 1314) As Tom Carroll,
president of the commission, notes, this figure ignores what may
in fact be the largest costs of teacher turnover: lost
teaching quality and effectiveness (Carroll 2011, 4).
In estimating the high cost to DCPS, Carroll explained that
teaching is no different than any other professionexperience
matters. Researchers have found that teachers
reach peak effectiveness with about seven years of experience.
But 80 percent of the teachers hired by D.C. this
year will be gone before they get there. [As a result,] the
District is burning about $12 million a year on
teacher churn$12 million that is spent hiring and replacing
teachers with no gain in school performance.
(Simon 2012)
For a teacher replacement strategy to effectively improve
student performance, then, the system must be losing mostly
bad teachers. But it may rather be losing good teachers, who are
leaving voluntarily (or by accident). The termination
data provided above make clear that the majority of teachers who
left DCPS during Rhees tenure had not been rated
ineffective, and that even among those who were terminated, only
half had ineffective ratings. One likely reason that
teachers who were not fired left anyway is an atmosphere of
mistrust that intensified under Rhee. As Levy noted, DCPS
has never been a nice place to work, now its an awful place to
work (Levy, personal communication 2012). Indeed, a
recent report on rising attrition asserts that DCPS has become a
teacher turnover factory that has a hard time retain-
ing teachers who are committed to their school and the community
it serves (Simon 2012).
It could also be that teachers who knew they were likely to get
a bad rating retired or resigned to avoid that stigma. If
the ratings accurately identified ineffective teachers, this
might be a good thing. Because they are based heavily on stu-
dent test scores, however, DCPS teacher evaluations may not be
very accurate. One highly publicized example suggest-
ing inaccuracy is seen in Washington Post education reporter
Bill Turques depiction of the termination of MacFarland
Middle School teacher Sarah Wysocki, who had been praised by her
principal, fellow teachers, and especially parents of
her students as one of the best (Turque 2012a). Turque recounted
the flaws with the value-added scoring method that
failed to account for Wysockis stellar performance in classroom
observations and led to her firing. The article also noted
Wysockis concerns that many of her students starting test
scoresagainst which the growth in her classrooms was
measuredwere impossibly high, because students came from a
school in which widespread cheating was under invest-
igation. The suggested inaccuracy of her ineffective score was
borne out by the fact that, immediately upon being
fired by DCPS, Wysocki was hired by Fairfax County Public
Schools, one of the nations wealthiest and best districts,
where she is considered a superb teacher (Brown 2011).22 While
this is just one anecdote, it illustrates several flaws with
IMPACT that are potentially applicable in many cases.
A lack of ratings reliability has not been the only problem.
Veteran teacher Mary Sutton, who taught for 11 years at
Malcolm X Elementary School in Anacostia and was one of few
teachers to receive an effective rating in a school with
a large share of low-income students, was excessed out of the
system in 2011. While DCPS asserted the need to elim-
inate even effective people we dont have an express need for,
Nathan Saunders, president of the Washington Teachers
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
22
-
Union, claimed that a push for younger, lower-cost hiressome
recruited from programs such as Teach for America
and D.C. Teaching Fellowshas wrongfully forced out seasoned
practitioners (Turque 2011a).
Saunders said Hendersons position would be understandable if the
District were not hiring hundreds of teachers
every spring and summer, many of them in fields in which the
excessed teachers worked. They include math,
reading and other core subjects. Payroll records show that the
city generally hires 300 to 400 teachers a year,
many of them younger and less costly than the veterans let go. A
recent independent study showed that the
proportion of first- and second-year teachers has grown sharply
in five years, especially in high-poverty com-
munities east of the Anacostia River. (Turque 2011a)
A 2009 Government Accountability Office (GAO) report, offered as
testimony for Congress on the DCPS reforms
under Rhee, provided other, data-driven reasons for concern. It
found that DCPS focused on a workforce replacement
strategy to strengthen teacher and principal quality (Ashby
2009, 8). As discussed above, without a strong plan to
ensure high-quality new hires and avoid destabilization, such a
strategy can backfire. The report found systemic flaws
and inconsistencies from the start of Rhees term that could
negatively affect teachers desire to stay in the district:
The district implemented too many new separate programs or
initiatives to boost student performance and, thus,
was forced to refocus; it failed to link employee expectations
and performance evaluations to organizational goals;
though it allowed principals to request changes to the staffing
model based on their schools needs, it did not estab-
lish or communicate clear guidance or criteria on how such
requests would be treated, resulting in city council and
parent allegations of unfairness and lack of transparency; and
it began hiring teacher coaches who were central to the
20092010 reforms later than expected, such that one in five
positions was not filled on time, and teacher coaches
were often uncertain about their responsibilities and how to
work with teachers, and received some conflicting guidance
from principals. In all, the report highlighted a lack of
coordinated planning, follow-through, and transparency, the
failure to communicate with relevant parties, and even exclusion
of key constituencies, including parents, from much
of the decision-making process (Ashby 2009, 10).23
Principal turnover is no higher than before Rhee and Fenty
controlled the system. Principals leave DC schools at a rate
of about 20 percent each year, and 25 percent of schools open
with a new principal (Levy 2012d). However, there are
two respects in which Rhees execution of reform policies may
have exacerbated the lack of strong leadership in high-
needs schools. The new principals she brought in had
particularly high attrition rates; of the 90 she recruited, the
major-
ity were already gone by the time she left in October 2010, and
most lasted no more than two years (Levy 2012d). And
a few particularly hard-to-staff and -serve schools had
substantially greater turnover. Anacostia High School,
transferred
by Rhee to Friendship Public Charter Schools to manage, had six
principals in six years (Brown 2012a). While such
rapid turnover is high, it is unfortunately not unique; Dunbar
High School, Hardy Middle School, MacFarland Middle
School, Kelly Miller Middle School, and Powell Elementary School
each had four principals in five years (20072012),
and Johnson Middle School and Patterson Elementary School each
had five principals in six years (Levy 2012d).
School closures. In addition to firing 1,000 teachers, Rhee
closed 24 regular public schools and three special education
centers during her three-and-a-half years as chancellor,
asserting the need to save money in areas of declining
enrollment.
Overall student enrollment in the district did not fall during
her tenure; rather, as in other reform cities and elsewhere,
it continued to shift from regular public to charter schools.
Charter schools enrolled 38 percent of D.C. public school
students by the time Rhee left in the 20102011 school year, up
from 30 percent when she arrived (Levy 2012a). At
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
23
-
FIGURE B
District of Columbia Public Schools enrollment; all, traditional
public, and publiccharter schools, from 19981999 to 20112012
*Excludes tuition students
Source: Authors analysis of audited enrollment data for DCPS
traditional and public charter schools, 19982012 (Levy 2012a)
the same time, the decline in regular public school enrollment
that had been the trend prior to Rhees arrival leveled off.
The result was a net total increase of over 4,000 students
during Rhees tenure, as illustrated in Figure B.
In alignment with the GAO reports finding that key DCPS
decisions were made with insufficient public input, a Pew
research study concluded that the closures happened too quickly
and without community input, generating distrust
(Samuels 2011). Moreover, while Rhee said that the students in
the schools slated for closure were struggling, when
their schools were closed, the students went, on average, to
schools with even lower test scores and lower odds of making
adequate yearly progress (AYP), as required under federal No
Child Left Behind guidelines (Levy 2011b).
The projected cost savings that Rhee had cited likewise failed
to materialize. An audit of the school closures made in
2008 found that the actual cost to the city of shuttering the 23
schools was steep: $39.5 million, quadruple the initial
reported cost of $9.7 million (Gartner 2012a). Unanticipated
costs included additional moving expenses beyond those
initially accounted for, 2010 demolitions of two elementary
schools, patrolling of closed schools, transportation of stu-
dents to further-away schools, and, heftiest of all, capital
asset impairment losses of $21.8 million, or the near-total
loss of the value of eight buildings when they were no longer
used as schools.
And while the current chancellor, Kaya Henderson, and Mayor
Vincent Gray have announced plans to close dozens
more schools, it will be hard to do so because, among other
problems, young children will have no safe (or reasonably
fast) way to get to their new schools (Scott 2012).24 Many of
the schools closed in the first round were close to others
BROADER, BOLDER APPROACH TO EDUCATION | APRIL 22, 2013 PAGE
24
-
that provided a viable alternative, but that is no longer the
case; with duplicative neighborhood schools now closed,
schools are increasingly farther from students homes. The
district provides school buses only for special education stu-
dents whose Individualized Education Programs (IEPs) require it;
other students beyond walking distance take public
transportation, paying half the normal fare. Most schools are
not near subway stations, and bus routes, designed princip-
ally to bring workers downtown, are not set up to serve
students. Transportation is expensive for families with
multiple
children or with young children whose parents need to ride with
them. The combination of distance and cost impedes
attendance, particularly for secondary students, who have
farther to travel, and teachers report that students sometimes
fail to attend for lack of bus fare. For those already
disengaged, the added barriers may tip the scales toward
dropping
out altogether. The mayor and chancellor should assess these
considerations as they target additional schools for closure.
Charter schools. Another aspect of the reform platform is
increased access to charter schools. As documented above,
the proportion of DCPS students served in charter schools was on
the rise before Rhee arrived and continued to
increase during her time in office. However, while Rhee and
Mayor Fenty had envisioned using their authority over
DCPS to reopen the closed public schools as charter schools,
they did not do so. Rhee did turn over operations of
two high schools to New York-based charter operator Friends of
Bedford: Dunbar High School in 2008 and Coolidge
High School in 2010. She also transferred management of Stanton
Elementary School in Ward 8 to Philadelphia-based
charter organizer Scholar Academies in 2010 (Turque 2010b).
When the overhaul of Stanton was announced in June 2010,
Washington Post education writer Bill Turque noted that
Stanton, along with five other schools to be reconstituted for
failing to make Adequate Yearly Progress under No
Child Left Behind, ha[d already] undergone waves of federally
mandated improvement and restructuring [yet four
of the six] had not met annual progress benchmarks for at least
seven years (Turque 2010a).
None of the three schools turned over to charter operators