-
. . . . . . . .
..~.........~..~...~..~...~~~~~..~~~~.....~...~....~~~~......~...~............................~..University
of Wisconsin-Madison * National Center for Improving Science
Educafion
Workshop Report
Research on SWhat Have We We Need to Know?
Synthesis of t
Volume 2: Br
William H. Clune,Dianne C. Bowcoc& Ricardo Mesquit
L. Webb,
Funded bNation
-
National Institute for Science Education (NISE) Publications
The NISE issues papers to facilitate the exchange of ideas among
the research anddevelopment community in science, mathematics,
engineering, and technology (SMET)education and leading reformers
of SMET education as found in schools, universities,and
professional organizations across the country. The NISE Occasional
Papers pro-vide comment and analysis on current issues in SMET
education including SMET inno-vations and practices. The papers in
the NISE Research Monograph series report find-ings of original
research. The NISE Conference and Workshop Reports result
fromconferences, forums, and workshops sponsored by the NISE. In
addition to these threepublication series, the NISE publishes
Briefs on a variety of SMET issues.
The preparation of this paper was was supported by a cooperative
agreement between the NationalScience Foundation and the University
of Wisconsin-Madison (Cooperative Agreement No. RED-9452971). At
UW-Madison, the National Institute for Science Education is housed
in the WisconsinCenter for Education Research and is a
collaborative effort of the College of Agricultural and
LifeSciences, the School of Education, the College of Engineering,
and the College of Letters and Science.The collaborative effort is
also joined by the National Center for Improving Science
Education,Washington, DC. Any opinions, findings, or conclusions
are those of the author and do not necessarilyreflect the view of
the supporting agencies.
-
Workshop Report No. 4
Research on Systemic Reform:What Have We Learned? What Do We
Need to Know?
Synthesis of the Second Annual NISE ForumVolume 2:
Proceedings
William H. Clune, Susan B. Millar, Senta A. Raizen, Norman L.
Webb,Dianne C. Bowcock, Edward D. B&ton, Ramona L. Gunter,
& Ricardo Mesquita
National Institute for Science EducationUniversity of
Wisconsin-Madison
December 1997
i
-
The writers wish to acknowledge Andrew C. Porter for his
valuable comments on our manuscripts, SarahK. A. Pfatteicher for
organizing the development of this document during spring 1997,
Deborah Stewart
for her skilled copy editing and document management, and Paula
A. White and Becky S. Torrisi forattending to the many other tasks
needed to complete the document.
-
Contents
Contents of Volume 1 . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .v
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . .. . . . . . .
..*..............................*...........................................................
vii
Forum Agenda. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
..*................*.........*.*...*.........................
ix
Status and Trends of Systemic Reform: Research on Systemic
Reform
Pascal D. Forgione,
Jr.................................................................................................
1Luther S.
Williams........................................................................................................
3
What Have We Learned? What Do We Need to Know?
William H. Clune
..........................................................................................................
5Susan Fuhrman
............................................................................................................
8Jane Butler Kahle
.......................................................................................
. ............... 1 1Uri Treisman
..............................................................................................................
1 6Daryl Chubin
.............................................................................................................
2 1
The Role of Curriculum in Systemic Reform
Rodger W. Bybee
........................................................................................................
23Margaret B. Cozzens
..................................................................................................
28Richard Greenberg
....................................................................................................
39Sylvia T. Johnson
.......................................................................................................
4 1Thomas A. Romberg
...................................................................................................
46
The Role of Teaching and Learning in SystemicReform
Christopher Dede
.......................................................................................................
50LeRoy Lee
...................................................................................................................
52Susan
Loucks-Horsley................................................................................................
54Shirley Malcom
.........................................................................................................
.60Lauren B. Resnick
.....................................................................................................
.62
The Role of Evaluation in Systemic Reform
Tom B. Corcoran
.......................................................................................................
64Paul B. LeMahieu
......................................................................................................
69Norman L. Webb
........................................................................................................
7 1Iris R. Weiss
...............................................................................................................
73Andrew A. Zucker & Patrick M. Shields..
.................................................................
.75
. . .1 1 1
-
Forum Highlights and Looking AheadCora B. Marrett
.........................................................................................................
80Michael W . Kirst
........................................................................................................
83Larry E. Suter
.............................................................................................................
87
Appendixes
A: Current Agencies Involved in Curriculum Improvement
................................................. 8 9B:
Comprehensive Mathematics Curricula & Funded Projects with
Contact Information.. .90C : Comprehensive Science Curricula..
.................................................................................
.92D: Criteria for Judging SSI
Strategies....................................................................................
9 3
iv
-
Contents of Volume 1
Contents of Volume 2 . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . V
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vii
Forum Agenda . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . ix
Status and Trends of Systemic Reform: Research on Systemic
ReformPascal D. For&one, Jr. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . 1Luther S. Williams . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . 3
An Introduction to the Papers and Think Piece Themes . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . 6William H. Clune and Norman L.
Webb
Themes Articulated in Forum Participant Think Pieces . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . 13Susan B. Millar, Dianne C. Bowcock,
Ramona L. Gunter, and Ricardo Mesquita
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . ..‘...............................................13
1 . The Definition of Systemic Reform . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
2. Implementation of Systemic Reform . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3. Curriculum in the Context of Systemic Reform . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . 34
4 . The Critical Value of Evaluation of Systemic Reform . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . 40
Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . 46
Forum Highlights and Looking AheadCora B. Marrett
.................................................................................................................
47Michael W. Kirst
................................................................................................................
50Larry E. Suter
......................................................................................................................
54
AppendixesA: Demographic Information about Forum Participants .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . 57
B: Contact Information for Forum Participants . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . 59
V
-
Preface
The goal of the Second Annual NISE Forumwas to generate and
share knowledge of systemicreform, including the role of
curriculum, the role ofteaching and learning, and the role of
evaluation.To reach this goal, panelists from around the
nationshared their expertise in these various areas
withparticipants who themselves brought variedexpertise to
structured conversations and network-ing sessions.
These two volumes are the result of a collabo-ration among four
NISE teams and, more broadly,of a collaboration among the NISE
researchers, theForum panelists, NSF staff, and the Forum
partici-pants. The Interacting with Professional Audiences(IPA)
team, Policy Analysis of Systemic Reform(PASR) team, Strategies for
Evaluating SystemicReform (SESR) team, and Formative Evaluation(FE)
team worked together to design, implement,and evaluate the Forum.
The four teams thencollaborated in the writing of Volume 1. The
FEteam (Susan Millar, Dianne Bowcock, RamonaGunter, and Ricardo
Mesquita) analyzed the 483think pieces written during the Forum and
pro-duced the “Themes Articulated in Forum Partici-pant Think
Pieces.” Senta Raizen and Ted Britton(IPA), William Clune (PASR),
and Norman Webb(SESR) supported the FE team’s analysis byframing
key themes based on their readings of thethink pieces. William Chme
and Norman Webbdrew heavily on the FE team’s analysis of the
thinkpieces as well as the panelists’ papers in writingtheir
introduction.
Given the way this document was developed,readers should view
the two volumes as elementsof a conversation through which a better
under-standing of systemic reform is emerging.
Volume 1: Analysis
l Introduction to the Papers and Think PieceThemes. In this
introductory section,
William Clune and Norman Webb integratethe other two sections by
developing someof the more compelling points raised by
theparticipants’ think pieces and the panelists’contributions.
William Clune is Voss-Bascom Professor of Law at UW-Madisonwith
research interests in education lawand policy, systemic reform, and
adequacyin school finance. Norman Webb is asenior research
scientist for WCER withresearch interests in mathematics
educa-tion, evaluation, and assessment.
l Themes Articulated in Forum ParticipantThink Pieces. During
three breakoutsessions, Forum participants were asked towrite
‘wlink pieces” in reaction to thepanelists’ remarks. This synthesis
of thethemes in the think pieces, including manyquotations from the
writers, provides thereader a window on issues, questions,
andcontrasting viewpoints about systemicreform that the Forum’s
varied participantsarticulated.
Volume 2: Proceedings
l Papers Presented. The ideas, knowledge,and experience of 24
panelists involved inimplementing, researching, and
evaluatingsystemic reform appear in this section.Most of these
papers were prepared aheadof time and distributed to
participants.Others are transcriptions of the panelists’remarks.
(Print and audio versions ofseveral of these are available on
theTEECH Web site, ht@://teech. terc.
edu/modes/papers/systemic4apers.cfi)
Both volumes include papers from the openingplenary session and
from the closing plenarysession.
vii
-
Second Annual NISE Forum: February 24-25,1997
AGENDA 1Research on Systemic Reform:
What Have We Learned?What Do We Need to Know?
Monday February 24
8:30 - lo:30 Opening PlenaryAndy Porter, chair
Status and Trends of Systemic ReformLuther Williams, Marshall
Smith
What Have We Learned? What Do We Need to Know?Bill Clune, Susan
Fuhrman, Jane Butler Kahle, Uri Treisman;Daryl Chubin,
discussant
lo:30 - lo:45 Break
lo:45 - 12:00 Breakouts
What Have We Learned? What Do We Need to Know?
12:lS - 1:30
1:30 - 3:30
3:30 - 3:45
3:45 - 5:oo
Lunch
Afternoon PlenarySenta Raizen, chair
The Role of Curriculum in Systemic ReformRodger Bybee, Midge
Cozzens, Richard Greenberg,Sylvia Johnson, Tom Romberg
The Role of Teaching and Learning in Systemic ReformChris Dede,
LeRoy Lee, Susan Loucks-Horsley, Shirley Malcom,Lauren Resnick
Break
Breakouts
Curriculum, Teaching and Learning in Systemic Reform
ix
-
Tuesday, February 25
8:30 - 9:30 Opening PlenarySenta Raizen, chair
The Role of Evaluation in Systemic ReformBernice Anderson, Tom
Corcoran, Paul LeMahieu, Norman Webb,Iris Weiss
9:30 - 9:45 Break
9:45 - lo:45 Breakouts
Evaluation in Systemic Reform; Discussions with Panelists
Monday’s Panelists Rodger Bybee, Jane Butler Kahle, Bill Clune,
Chris Dede,Richard Greenberg, Sylvia Johnson, Lauren Resnick,Susan
Loucks-Horsley, LeRoy Lee, Uri Treisman
Tuesday’s Panelists Bernice Anderson, Iris Weiss, Tom Corcoran,
Paul LeMahieu,Norm Webb
11:OO - 12:30 Closing PlenaryAndy Porter, chair
Forum Highlights and Looking AheadMike Kirst, Cora Marrett;Larry
Suter, discussant
X
-
Status and Trends in Systemic Reform:Research on Systemic
Reform
Pascal Forgione, JI:Commissioner of Education
Statistics,Department of Education
I bring you greetings from Acting DeputySecretary Marshall
“Mike” Smith who reallywanted to be here this morning. Mike asked
meto reflect briefly on the impact of the Third Inter-national
Mathematics and Science Study(TIMSS) and what it has done for the
characterand quality of the national conversation aboutsystemic
reform. While on the one hand, sys-temic reform is very complex and
challenging, itis also very easy to trivialize this construct.
It’seasy for people to define what they are doing assystemic
without actual substance or verification.
Fortunately, TIMSS is not about quick fixes.In fact, the Grade 8
data-all that’s been releasedthus far-suggest that there are no
simple solu-tions. TIMSS has provided data to explode threemyths. I
hope in June when the Grade 4 data areout that we confirm these
findings. First, eighthgraders in the United States have more hours
ofinstruction in science and mathematics than Ger-man or Japanese
students. Despite our shorterschool year, if you take the length of
the periodsand the frequency per week at eighth grade com-pared to
Japan and Germany, U.S. students inGrade 8 have more hours of
instruction.
Second, on our questionnaires, Americanteachers reported
assigning more homework thantheir Japanese or German counterparts.
Andwhen we looked at the after-school work, all ofthe work put
together, American students do asmuch homework as their German and
Japanesecounterparts.
Finally, if you take three hours of TV watch-ing per night as
heavy TV watching, as manyJapanese children are heavy TV watchers
as chil-dren in the United States. So these three simplemyths of
time, homework, and TV are not what
, it’s about.There are not going to be simple answers.
We’ll see whether the next two data sets-Grade
4 in June 1997 and Grade 12 in spring 1998-confirm these initial
findings.
The TIMSS design overtly includes the keyelements emphasized in
a notion of systemic re-form. The study had five major components
thatutilized a variety of methods to examine thebreadth and the
depth of the essential elements.Basically, one only has to look at
the table ofcontents of our report. This report tries to
giveAmerica a systemic view about achievement,curriculum, teaching,
the lives of our teachers,and the lives of our students. In fact,
the Grade 8summary report called Pursuing Excellence waswritten
explicitly to be read on an airplane in onehour by a businessman.
Now remember, this is astatistical report. It may not appear to be
a statis-tical report when you first look at it, becausethere are
no standard errors; we don’t talk aboutstandard deviations. We
tried to write in clearEnglish about what these findings mean. I
canassure you that every sentence went through astrict adjudication
review process. Everything inthe report is supported by data.
Let me highlight for two areas the kinds ofquestions that TIMSS
is raising for this Nation.The first is the area of curriculum
where Profes-sor Bill Schmidt has developed very
innovativemethodology to look at textbooks and curriculumof the
fifty countries. The kind of questions weasked in our report under
curriculum included:
. Who sets the curriculum?
. Is the curriculum as focused as in other coun-tries?
. Is it as advanced as in other countries?
. What do we mean by advanced?n What is the content that really
is advanced at
the eighth-grade level and how much time isspent in class?
I
-
These are the kinds of powerful questionsthat we need to engage
the Nation on and builddeeper understanding through the analysis of
theTIMSS database. It will take over a year to rollout the three
populations of TIMSS data, fromNovember 1996 (Grade 8), through
June 1997(Grade 4), to spring 1998 (Grade 12).
The second area that I want to emphasize isthe innovative work
of Professor Jim Stigler whoconducted a three-country classroom
video studyof mathematics teaching. This is only Grade 8mathematics
and only in Germany, Japan, and theUnited States. It’s unfortunate
that we were notable to include Grade 8 science, but this studywas
quite an innovation five years ago. Thisvideo study answers the
kinds of questions thatwe could never answer with questionnaires
oreven case studies:
.
.
.
.
.
.
How do teachers in the three countries teach?How do the
instructional practices differ?How do they organize and present
their les-sons?What are the cultural scripts of the lessons?What is
the role of the teacher and the stu-dent during seat work and then
class work?Are the students passive or active learners?What
proportion of class time is spent ininstructionally relevant
activities? in socialor housekeeping activities?Are teachers aware
of the reform?Do they think they’re doing it and do we seeit?
This video study included a half nationalsample in the three
countries. We have 81 lessonsof a U.S. teacher. We selected the
schools; weselected the teachers; and we selected the classperiod.
It’s a time warp capturing Americanteaching at Grade 8 in
mathematics, and we’venever had that before. We also have made
avail-able a CD-ROM and a videotape with six les-sons, two for each
country, from the study. Forexample, you are able to look at a
Japanese alge-bra lesson and compare it to an American one. It
stimulates a terrific conversation about qualityteaching.
TIMSS also taught us to be careful aboutdrawing conclusions,
even with statistical stan-dards such as reliability. There’s a
need to con-tinue to verify and to confirm the data. Let megive you
an example. American teachers, 95% ofthem, said they knew the NCTM
standards, and75% of them said they were implementing them.When we
examined the videotapes we found afocus on conceptual thinking in
only 20% of thelessons. Again, what people called high
levelmathematics often ended up being the processesand not the deep
mathematics. Thus, we werecautious in reporting the mathematics
question-naire data because the mathematics video datadid not
confirm the claim. In science, we hadvery similar data. Grade 8
science teachers saidthey also were implementing the standards.
Weheld off releasing the findings because we werenot able to
corroborate the data. So we need to becareful in our survey
reporting.
It’s also ironic that in TIMSS we probablyhave more information
about American curricula,textbooks, teaching, and instruction than
in anyother national database. This database is the kindthat we
need to build, with rich videos and cur-riculum components. It is a
limited database inthat it’s cross sectional and not longitudinal,
andit also doesn’t often go to the student level. ButTIMSS whets
our appetite.
I’m pleased to say that Mike Smith, NealLane, director of the
National Science Founda-tion, and Ernie Gibbons, the President’s
advisoron science, are working with the Domestic PolicyCouncil to
assure cross-agency follow throughand coordination. Wouldn’t it be
wonderful if thefederal government could be systemic about
ourpartnerships to support the challenging reformsin science
teaching and learning? I think we’rebeginning to have the collegial
and cooperativeconversations.
Thank you for this opportunity to share withyou a brief overview
of the TIMSS findings. Ilook forward to being your partner as we
moveforward in this exciting work.
2
-
Luther S. WliamsAssistant Director for Education and Human
Resources,National Science Foundation
I’m going to make a few comments bearingon the topic of status
and trends of systemic re-form. They are the end results of
observations,experiences, and thinking about systemic reformin the
context of NSF’s programs. I will try toframe them vis-&is your
conference. You askedtwo very important questions about research
onsystemic reform: What have we learned? Whatdo we need to know?
The answer to the secondquestion is, “Substantial”; it is probably
limitless.About the first question, if by research you meana
fundamental exercise informed by some theoryor hypothesis that
leads to substantial advance-ment of the knowledge base, the answer
is, “Verylittle.” I would argue that the size of the knowl-edge
base is one of the challenges in anantiquality context to systemic
reform. Few stud-ies, in my judgement, configure to equal
re-search.
promote a comprehensive restructuring in a sys-tem that is
highly resistant to Unitarian ap-proaches. That recognition, in my
view, from aresearch perspective, is one of the most impor-tant
factors that should bear on any research de-sign. There is
essentially no f=ed set that definesthe problem that one is going
to take up. None-theless, under the rubric of NSF’s and
others’programming, substantial progress has beenmade in a variety
of states and in large urbancommunities. My sense is that progress
has ob-tained insights that have done a very good job ofcontrolling
for a series of antisystemic orantiquality forces. I want to spend
a few minutessharing my list of those insights.
There’s another observation with respect tosystemic reform:
Today, almost every conceiv-able machination bearing on K-12
education,and now increasingly higher education, can ac-commodate
itself under the rubric of systemicreform. That’s okay at some
level, and that is in-tegral to the American culture. But it does
makevery challenging the maintenance of focus onsystemic reform,
particularly of a K-12 system. Imake that observation because
systemic reform isin the context of a system. In the case of
NSF’sprograms in the K-12 domain focusing only onscience and
mathematics, a very powerful lessonthat’s been learned over the
past several years ishow exceedingly complex that system is.
Thatfinding of complexity, I would submit, is rarelyacknowledged,
whether one is talking about astate or a large urban community.
Complexity ismade worse by the fact that systems are
highlyfragmented. In effect one is attempting to pro-mote reform in
a system that is elusive and insome instances presents itself in
multiple con-structions in real time. The system is also
ex-ceedingly conflicted--even contradictory-ow-ing to
counter-instance agendas, roles, partici-pants, and players.
Therefore one is attempting to
One of the most important challenges is toeffect the conversion
by process from a tmidi-mensional to a multifaceted approach. All
of theelements of the system are counter to a multifac-eted
approach; the system seeks to devolve to aunidimensional
construction.
The second issue that has to be overcome isconcerned with a very
challenging nonlinear,very dynamic, ever-changing problem
construc-tion. It is made more difficult if emphasis isgiven to
mathematics and science learning as op-posed to educational
processes. That is a veryimportant distinction.
Third, systems that have made substantialprogress have been very
explicit and have almostelevated to a mandatory level the required
infra-structure that is needed for mathematics and sci-ence
learning. Pat Forgione alluded to part of thisin talking about one
dimension of one of the vari-ables that equal this very, very
complicated con-struction. Let me extend that further. All of
theresearch he discussed has to do with eighth-grademathematics,
but his question is, “At the end ofthe day, what is the value of
the take home?’ Oneknows nothing about eighth-grade science;
oneknows nothing about the K-7 enterprise that ac-tually informed
mathematics learning. Unlessone thinks eighth-grade mathematics is
the endpoint, it is comparatively inconsequential relativeto the
rest of the sequence. I’m trying to empha-
3
-
size the vast domain about which one does nothave comparable
findings. We need to make thetransition to emphasis on science
learning in thecontext of the total system.
The fourth factor that has been very impor-tant where progress
has been made is having in-dividuals who operate from the context
of atheory of reform, at least for their urban schooldistrict or
their state. The reform was not haphaz-ardly deposited in a series
of machinations in asystem without some reference to why, in
fact,one is engaging in an explicit set of activities:What is the
chronology in which one engages inthose activities? What are the
goals? What arethe road maps? What are the strategies
associatedwith those? What kind of formative evaluation isconducted
in order to make early adjustments,and what outputs are valued? Is
it the orderlyprogression year by year through the system, oris it
the definitive outputs associated with sciencelearning?
Progress has also been made in systemswhere there has been
substantial attention to ef-forts to manage the role of others,
including thefederal government, which is not unitary in
itscontributions to standard mathematics and sci-ence education.
How does one reconcile all ofthe conflicting inputs that bear on
systems fromWashington through funding and through
othermachinations? Similarly, state apparatus is an en-tity that
requires some accommodation-witnessthe difference among the goals
for learning in thevariety of national or state assessment
systemsthat are present across the states. What do theycommunicate
differentially about the reasonableoutcomes?
Another factor is higher education. Scienceand mathematics
reform is critically dependanton the contributions from the higher
educationcommunity. But there is one matter on which Iwould argue
some thoughtful consideration, ifnot apprehension, needs to obtain.
The sugges-tion that higher education as a domain is an ex-emplar
of successful conclusion of reform or re-structuring requires one
to engage in a supremefiction. There are enormous contributions to
behad from the higher education community. (It ismostly scientific
and technical expertise.) It isnot a sector that schools should
turn to for under-
standing school systems. In contrast, some schooldistricts have
done a very good job of workingout arrangements with the business
sector. Thatsector actually does have experience in this arenaof
systemic reform or restructuring. The businesssector has taken on
very challenging multifacetedproblems and made progress. So there’s
a lot tobe learned from that sector. I would argue thatschool
systems that have made great progress be-yond having very
productive relationships withthe business sector have forged very
excellentrelationships with a variety of
community-basedorganizations that have been able to bring to bearan
advocacy that is not found in other sectors.They’ve also catalyzed
parents and employedthem in the support of their efforts.
Successful sites also recognize that they’reworking in a very
complicated, multifaceted,ever-changing, nonlinear system. Early
on, thesesites included “strategies for innovation replica-tion” to
develop some sense of how to scale up.Stringency is brought to
documenting, communi-cating, and securing gains, so that they in
effectbecome demonstrable nodes toward reform. Onecan continue to
build on them and try to drive thesystem toward greater
educational, financial, andintellectual economies while becoming
more so-phisticated in promoting reform. Many states re-quire doing
that in a systemwide fashion, con-verting a variety of resources
whether in highereducation, state departments of education,
orschool systems. In other words, all of these ele-ments about
which I’ve spoken have been contig-ured in a system that is
supporting an agendathat, if not unitary, has been reduced beyond
itsnatural limits of replication.
There is a challenging research agenda trans-forming what we
should learn. Some of the sites(maybe all of them in contrast,
because they’vehad very different experiences) represent excel-lent
source materials for mounting the research,not only in terms of
documenting reform inprogress, but also in trying to identify in a
multi-faceted domain the fmite number of overarchingand control
elements that are crucial. We reallydon’t know that number. An
exceedingly impor-tant and timely contribution from research
wouldexplicate within an undefined set of variables theminimum set
that’s really crucial to driving re-
4
-
form, leaving aside the variation that one finds would suggest
you do that with deliberatenessfrom system to system. and near
exclusion of the ever present and well
Again, I acknowledge the very important understood, which is of
little value. The impor-work you do and very much urge you to take
up tant research to be done is that which 95% of usthese matters
under your research agenda. But I do not already purport to
understand.
What Have We Learned? What Do We Need to Know?
William H. ClunePolicy Analysis of Systemic Reform, NISE Team
LeaderUniversity of Wisconsin-Madison
The research of the Policy Team of the NISEhas concentrated in
three areas: implementationof systemic reform, implementation of
the na-tional standards, and equity. I would like to re-port
findings and key research questions in eacharea and then turn to
what we are learning aboutthe links between systemic reform and
studentoutcomes and the implications for policy and re-search.
selective curriculum replacement? By curricuhmreplacement I mean
the implemented curriculum,as actually taught in schools.
Findings on the National Standards and LocalImplementation
The Knapp Synthesis of Research on Imple-mentation of Systemic
Reform
large “grain size,” unguided local discretiontension between
problem solving and formalanalysisconfusion over “deep
understanding” and“habits of mind”
. widespread planning
. incremental change in the classroom
. absence of broad and deep systemic change
. key knowledge gap: strategies leading towholesale v. selective
curriculum replace-ment
On the first topic of implementation, we
political controversy over content and conse-quenceskey
knowledge gap: alternative packages ofcurricula, materials,
assessments, develop-ment, technology thatfit the standards and
alternative educationalgoals and philosophiesdraw on knowledge in
NSF Systemic Initia-tives, TIMSS
commissioned a paper by Michael Knapp that heentitled “Between
Systemic Reforms and theMathematics and Science Classroom.” This
pa-per synthesized research not only on NSF’s Sys-temic Initiatives
but other systemic reforms suchas in Michigan and California. The
paper foundthat systemic reform had stimulated widespreadplanning
and discussion, in other words had be-come a major presence on the
policy landscape,but had produced only incremental change in
theclassroom, that is, an absence of broad and deepor systemic
change. The key research question is,Which strategies lead to
wholesale as opposed to
The fmdings on local implementation, orimplementability, of the
national standards werecomplementary. The standards are very
generaland comprehensive, which allows localities andtextbook
writers to claim compliance withoutmaking much real change. There
is a tension be-tween constructivists who emphasize problemsolving
and traditionalists who prefer formalanalysis. There is real
confusion about how todefine deep understanding of the subject
mattersand so-called habits of mind, as well as
politicalcontroversy over content and consequences, such
-
as for the traditional group of college-bound stu-dents. A key
knowledge gap is the availability ofpackages of curricula,
assessments, teacher de-velopment materials, and technology (such
asinteractive learning) that fit the standards, as wellas
alternative educational goals and philosophies,including both
college prep and vocational goals.In other words, it looks like
some of the disputescannot be resolved by consensus but instead
willrequire the design of highly developed curricularoptions.
Relevant knowledge of this kind existswithin NSF and many Systemic
Initiatives buthas not been pulled together adequately and
con-nected with systemic reform.
The Case of the Virginia Standards and theImportance of
Nonpoliticized Review
. quality should be recognized
. without blocking legitimate options
. without ignoring implementation in symbolicpolitics
It is important that we do serious quality re-views of the
entire chain of developing andimplementing standards, as
illustrated by thepolitics now surrounding the Virginia
standards.If these standards are indeed of high quality andare
appropriate for certain educational goals andphilosophies, that
conclusion should be firmlyestablished and widely publicized by
indepen-dent researchers. On the other hand, if alternativemodels
are needed, those, too, should receive ap-propriate support. And
there is an important issuebeyond disputes over competing goals and
phi-losophies, because politicization can present dan-gers for any
kind of standards-based reforrn If aparticular set of standards is
not appropriate forall educational goals and philosophies, a
political“victory” for one group may be costly to manystudents; and
symbolic politics over standardsmay actually prevent us from even
looking at anyreal development of the implemented curriculumand
teacher capacity.
We should not have expected that the searchfor “universal high
standards” or “hard stuff for
all kids” would be easy, but neither should weleave this central
question to politics. Perhapsinternational comparisons, such as
TIMSS, canshed some light on options for implementing uni-versally
high standards.
Findings on Equity
. ethnic/gender subgroups differ in outcomesn mixed evidence on
equalization through
course takingl controversy over equity and academic
coursesn controversy over how to measure the gap and
expand access. controversy over indicators as gatekeepersn key
research question: high standards,
greater access
We had two papers on educational outcomesin mathematics and
science by race and gender,written by Bill Tate and Albert0
Rodriguez.These updated and refined similar work done inthe past.
They found many substantial differ-ences across subgroups of race,
ethnicity, andgender and mixed evidence for the propositionthat
similar course taking will produce similarresults. Equivalent
courses narrow the achieve-ment gap but don’t eliminate it and are
not avail-able to many students. Clearly, the effort to mea-sure
and report on gaps in equity yields valuableinformation.
But we also encountered serious debatesabout equity indicators.
One is whether weshould stick with college preparatory courses,
orwhether less academic courses are also neededfor more equity.
This debate appears to be occur-ring in the field through a
conflict of contendingforces, rather than through rational design
andoptions. A related debate is whether the studentassessments used
to measure achievement gapsare the correct measure of equity and
especiallythe correct criteria for selective admissions.Thus, the
key research question is how to simul-taneously raise standards and
broaden educa-tional access.
6
-
The Need for Theory Linking Process andOutcomes (what we need to
know from re-search on systemic reform)
. outcomes are changed by process
. value of theory shown in example of coursetaking and
achievement
Across all three areas, we need a bettertheory linking the
process of systemic reform, orpolicies, with outcomes. Theories
that link pro-cess and outcomes in education, or any area ofsocial
science, are valuable and rare. It took along time, for example, to
establish the connec-tion between course taking and achievement
tothe point that we now consider course taking anintermediate
outcome, as much as a process vari-able.
Key Process Components of Successful Sys-temic Reform
A change strategy controlling
= local politics, resources, and administration, content and
pedagogy. school improvement= public acceptance= longitudinal data
on curriculum and achieve-
ment
If we look across sites attempting systemicreform, the more
successful sites have a changestrategy involving control over (and
through) thepower structure, curriculum and teacher develop-ment,
school improvement, public acceptance,and data on incremental
changes in curriculumand achievement.
Critical Flaws in Real Change Strategies
= political, ideological, racial conflict= political vision
without pedagogical delivery. school improvement without scaling
up. external accountability without internal com-
mitmentn public rejection of professional agendas. lack of
resources, limits of volunteerism
. absence of planned sequence of classroomchange
On the other hand, unsuccessful sites haveone or more of a
familiar list of flaws: lack ofpolitical integration and will,
political visionwithout a pedagogical delivery structure,
schoolimprovement or teacher enhancement projectswith no realistic
strategy for scaling up, externalaccountability or technical
assistance without in-ternal buy-in, unmanageable public
controversy,perhaps over a previously low-profile profes-sional
agenda, lack of resources and the limits ofvolunteerism, and the
absence of a planned strat-egy for incremental change in schools
and class-rooms.
Implications for Research and Systemic Initia-tives
. clarifying the profiles of successful process,e.g., policy
plus delivery, networks plus scal-ing up
. setting initial conditions and incentives
. matching resources to vision: no “moonshotson a
shoestring”
. cost-effectiveness of partially successful sys-temic
reforms?
. research question: enough real success forthe cost?
. skepticism plus sensitivity to small variations
If the conditions for successful systemic re-form are really
much more stringent than previ-ously suspected, what are the
implications forany policy seeking to expand the scope of
suchinitiatives? First, the profile of success wouldneed to be
clarified, so that sites could decidewhether the necessary elements
were in place.Realistically, there probably would be
severalprofiles, such as policy plus delivery and net-works plus
scaling up.
These profiles would have to be translatedinto initial
conditions for participation, but if wewant anyone to accept the
conditions, the incen-tives for participation should match the
intensityof the commitment. If we expect that a full-scalesystemic
reform will be built from scratch, seri-
7
-
ous attention should be given to the resourcesthat could support
this kind of effort, instead ofexpecting a “moonshot on a
shoestring.”
Perhaps the most interesting bottom-linepolicy question is the
cost-effectiveness of par-tially successful systemic reforms.
Suppose thatnothing like full-scale systemic change will
beproduced, in the sense of wholesale replacementof the curriculum
and substantial gains inachievement for all children. But also
supposesignificant progress in both process and out-comes, namely,
that a system has been built thatis capable of producing continuous
improvementin curriculum and instruction with assurances ofsocially
valuable change in curriculum andachievement. Then suppose that the
initial invest-ment is not very large, as is usually the case
inSystemic Initiatives.
The situation represents failure of systemicreform in the large
sense, but success of asystemic-type reform as judged
bycost-effectiveness. The key research questionwould be when there
are enough pieces of a com-plete reform to produce some significant
results
at a relatively modest cost, in other words, whena reform
crosses some as yet undeterminedthreshold of
cost-effectiveness.
Nothing is more common in policy than am-bitious rebuilding
projects being scaled down tomodest home improvements (except
perhaps forcomplete failure), and we should not lose sight ofthe
broader objectives, because they do appearfeasible in some places.
On the other hand, Iknow of nothing in policy that should prevent
usfrom accepting cost-effective policies, especiallyin this vital
area of social policy. Some commen-tators suggest that systemic
change is alwayslong-term, and progress is recognizable from
theconstant vector of change over time and acrossinstitutional
locations, rather than from anyrapid, wholesale transformation. At
base, it isthe old question of the half-full or half-emptyglass,
with the key questions being just howhalf-full and what is the cost
of the drink. An-swering those questions will require a kind
ofresearch that is appropriately skeptical of opti-mistic claims
yet sensitive to the importance ofsmall, but important, variations
in actual perfor-mance.
Susan FuhrmanDirector, Consortium for Policy Research in
EducationUniversity of Pennsyivania
My remarks focus primarily on the processof standards-based
systemic reform. I trust thatother speakers will address the
accumulating evi-dence on the effects of these reforms. For
ex-ample, we know that many teachers are at leastaware of new
policy directives and are favorablydisposed toward them. In some
schools and dis-tricts, important changes in teacher practice,
suchas increased use of real books and stories insteadof basal
texts and more hands-on activities in sci-ence, have been noted.
CPRE researchers sawpositive effects on 4,800 students in
transitionhigh school mathematics courses linked to ambi-tious
NCTM-like standards. Students inCalifornia’s Math A courses and New
York’sStretch Regents courses were much more likelythan
general-track students to complete a mini-mal college-preparatory
sequence by the end of
high school. And students in transitional math-ematics classes
posted better achievement testscores than students in general
mathematics (al-though they still lagged behind students in
col-lege-preparatory mathematics classes). We arealso finding that
professional development di-rectly linked to the course content has
contrib-uted to changes in practice and improved
studentachievement.
My choice to focus on the process of enact-ing and implementing
standards-based reformsreflects my background in political science,
but itis also appropriate to the current state of
reformdevelopment. The reforms are not yet in place inmost states
and districts; they are still under de-velopment. Their story is
still being written. Fur-ther, many of the most important
challenges fac-ing reformers, as well as lessons about the
-
progress of these reforms, concern the difficultiesassociated
with the reform process and politics.
I will focus on seven lessons about theprogress of
standards-based reform. I draw pri-marily on CPRE research in nine
states (CA, CT,FL, GA, KY, MN, NJ, SC, TX) and 25 districtsbetween
1990 and 1995 (Massell, Kirst, &Hoppe, 1997).
1. Standards-based reforms continue to makeprogress, despite
changes in leadership and po-litical turbulence. Although there was
oppositionto the reforms in each of our states, particularlyin the
1994-95 period, they were not dismantled.Despite the vocal, and
often virulent, objectionsof religious conservatives who asserted
that stan-dards interfered with the prerogatives of familiesand of
antigovernmental forces who saw stan-dards as infringements on the
authority of localschools, standards development continued.
Inwell-established American tradition, new re-forms, such as
charter schools, were developed torespond to those who advocated
devolution, andthey were simply added to the books while stan-dards
reforms continued to be developed andimplemented. Political
rhetoric focused on thenewer reforms, while under the surface,
andmore quietly than when they were originallychampioned, standards
documents, new assess-ments. and related policies were promulgated
andthe slow process of classroom implementationbegan. Standards
policies were modified in re-sponse to opposition, as noted below,
but the ideaof standards-based reform continued to be robust.
2. Much of the continuing momentum behindstandards-based reforms
can be traced to the ac-tivity of nongovernmental forces.
Professionalassociations, networks, and collaborations,
somenational and some state-specific, were importantsources of
support and expertise. For example,standards developers at both
state and districtlevels drew on national documents and
examplesfrom other states. Sometimes nongovernmentalorganizations
provided revenues. This is the casein eight districts undertaking
standards-based re-forms with the help of the Pew Charitable
Trusts.Certainly, the associations and partnerships en-hanced the
legitimacy of standards-based re-forms. Most national or regional
gatherings ofeducation policymakers became occasions to
highlight the record of states and districts consid-ered “in the
lead” in these reforms and to drawimplications from their
experiences. These meet-ings, and the diversity of groups-from
unions tobusiness leaders-supporting standards-basedreforms, were
very reinforcing.
3. The federal government was an importantsource of support.
Goals 2000 funds were usefulin many states, particularly for
subsidizing pro-fessional development. The standards frameworkof
the Improving America’s Schools Act ap-peared to lend greater
legitimacy to state efforts,but it was just taking shape when we
were last inthe field. We have plans to examine its influenceover
time. In the context of this Forum, the criti-cal contributions of
the National ScienceFoundation’s Statewide Systemic Initiatives
mustbe noted. The SSIs in seven of our nine states(CA, CT, FL, GA,
KY, NJ and TX) were respon-sible for developing the mathematics and
sciencestandards. And the SSI can sometimes be cred-ited with
giving standards reforms an essentialboost. For example, Georgia’s
subject-matter re-visions in most areas had been stalled for
years,but thanks to the SSI, work in mathematics andscience
progressed.
4. That support for standards reforms camefrom many sources was
essential for its politicalsurvival, but the very diversity of
supportersmade achieving coherent direction for educationmore
difficult. Standards were supposed to rep-resent agreement on what
students should knowand be able to do, but the many sources of
stan-dards-national associations, local and state de-velopment
committees, new specifications devel-oped by test
publishers-frequently varied in in-structional vision. Policymakers
and educatorsare drawing on multiple sources in developingtheir own
versions of standards, and an importanttopic for future research is
the coherence of theresulting products. For example, contrary to
thefears of conservative critics, local educators werenot
circumscribed by state standards. In mostcases, they used state
frameworks as only onesource of their own standards and found
statestandards too vague to be really useful. They de-veloped their
own frameworks, using many ex-amples and models.
9
-
5. An additional reason to worry about thecoherence of emerging
standards documents isthe trend toward adding “‘the basics” back
intostate standards. Part of the political and publicopposition to
standards centered on the new-fangled notions of learning and
pedagogy incor-porated in early standards documents. The
newapproaches seemed to threaten traditional skillsand common-sense
notions of what school wasabout-reading, writing, arithmetic,
spelling, etc.To respond, many states sought more “balance”in their
frameworks, for example, by incorporat-ing phonics into language
arts standards that pre-viously had a whole language orientation.
Simi-larly, emerging state assessments were modifiedto add
multiple-choice items back in and to in-clude norm-referenced as
well as criterion-refer-enced components. Will “balance” mean
morethoughtful integrations of approaches or simpleaggregations,
with traditional practices on onepage and new ideas on another?
Future researchshould attend to this question.
6. Early on in the standards reform move-ment, it became clear
that the work of standards-based reform is extremely demanding. The
ideaof making change in virtually every aspect ofpolicy-curriculum,
assessment, teacher prepara-tion, professional development-not just
at oncebut in a coordinated fashion so they all linked tostandards,
represented a huge challenge to a po-litical system that was expert
in incremental, dis-jointed change. Over time, the overwhelming
de-mands of standards-based reform have becomeeven clearer, as has
the need to sequence reformcomponents to make them manageable and
fea-sible. Teachers are furious when new assess-ments come on line
before the standards, towhich they are supposedly tied, are
developedand before any curricular materials are available.They are
even angrier when such new assess-ments are incorporated in
accountability systemsthat carry consequences. Yet such disconnects
insequencing are not uncommon at the state anddistrict level. In
the states we studied, some thattook more incremental, more
step-by-step ap-proaches to reform have had less upheaval thansome
of the previous reform leaders--one reasonmay be that the
incremental states were able to
be more deliberate about each step and plan morecarefully about
how to bring things on line.
7. The most important factor influencing theprogress of reform
development and ultimatelywhether standards reforms can support
meaning-ful changes in teaching and learning is the capac-ity of
the system. Capacity is essential at everylevel. States must manage
complex processes ofstandards development that balance public
andprofessional input, choose or create assessmentsthat balance
needs for adequate information forparents and teachers and the
desire to modelgood instruction through challenging items; re-vise
licensing systems; support schools and dis-tricts in curriculum
development and profes-sional development-and this is just some
ofwhat is required. Districts must do much of thesame, focusing on
daily support to schools goingabout the difficult process of
improving instruc-tion. Teachers and administrators must developnew
approaches to teaching and learning, whilefrequently managing new
site-based governancepolicies and developing ways of relating to
oneanother and to parents. Parents are expected tosupport new, more
challenging expectations fortheir children, understand and respond
to new,very complex accountability systems, and seethat schools are
supported with adequate re-sources. Everyone needs help.
There are some signs that the need forgreater capacity is
recognized. For example,Connecticut, Florida, and Minnesota were
amongthe states actively revising their teacher certifica-tion
processes to support more challenging vi-sions of instruction. Many
states have orches-trated or supported teacher networks that
groupteachers by grade level or subject in order to pro-vide
continuing support; some states and districtsare facilitating
school access to reform designsand technical assistance.
But capacity-building efforts to date aredwarfed by the need and
undercut by omissionsand contradictions. For example, in our
states,state and local central agencies continue to bedownsized. At
the state level, a number of agencybudgets were cut by about 25%
during this pe-riod, coming on top of earlier cuts during the1980s.
Little has been done to see to the quality
1 0
-
of professional development available to teach-ers. The
prevailing mode is to push money downto the school or district
(that is, where the state ismaking a new investment in professional
devel-opment at all) assuming that demand for profes-sional
development will be generated by newstandards and assessments and
that the demandwill in turn generate supply. This turns out to
betrue; suppliers do show up. But much of what isavailable is of
very low quality, and few placeshave tried to develop criteria to
guide selectionof professional development providers or to edu-cate
consumers about good choices. And, finally,a crying need is for
curriculum and materials, re-flecting the standards, that can be
used for dailyinstruction. Increasingly, teachers themselves
arequestioning the romantic notion that day-to-daycurricula should
be developed by collaborativeschool or department-level groups.
They haveneither the energy nor the desire to create fromwhole
cloth when others might have invented
Jane Butler Kahle*Condit Professor of Science EducationMiami
University
Although much has been written about thenature and policies of
systemic reform bypolicymakers, those papers provide only a
partialvision of systemic reform-one from the outsidein. Less has
been written from the field, i.e., fromthose who are actively
trying to promulgate re-form either in the classroom or at the
state level.Even less has been written about changes inteacher
practices and in student learning-yet,without those changes the
reforms eventuallywill wither and fade away. The encompassingnature
of systemic reform provides critical rolesfor national and state
leaders, for professional
* The preparation of this paper was sponsored in partby the
National Science Foundation, Grant # OSR-92500 (J. B. Kahle and K.
G. Wilson, Co-PrincipalInvestigators). The opinions expressed are
those of theauthor and do not necessarily reflect the position ofN
S F .
very usable and good materials. We need bettermethods of sharing
what is available and moreattention paid to development.
Systemic reform has proven to be both hardyand difficult.
Challenges multiply, but the basiclogic has enormous appeal and
staying power. Itis important to act on the lessons we are
learningabout the challenges, for example, by supportingmore
extensive efforts to develop and dissemi-nate excellent curricula.
It is also important tokeep tracking the results. Evidence of
positiveeffects will be critical to the reform’s stayingpower over
time. We must show that the hardwork is worth it.
ReferenceMassell, D., Kirst, M. W,, & Hoppe, M. (1997).
Per-
sistence and change: School reform in ninestates. Unpublished
manuscript, University ofPennsylvania, Philadelphia, Consortium
forPolicy Research in Education.
groups, and for individual teachers. Experiencewith several
systemic initiatives in one state,Ohio, forms the basis for the
following discus-sion of the challenges to reform and the
changesneeded for success.’ My discussion focuses onthe challenges
faced by one state’s efforts, onevidence of changes in teaching
practice and instudent learning, and on the meanings that maybe
drawn from that evidence. Specifically, Ohio’sreform was
characterized by the following pa-rameters. It
9 developed a regional infrastructure to sup-port and sustain
reform.
n focused on teacher professional developmentthat emphasized
content, provided in-depthexperiences in inquiry and problem
solving,and extended over one year.
. targeted middle schools for equity and eco-nomic reasons.
1 1
-
. included substantive involvement of univer-sity scientists and
mathematicians.
. avoided K-12 curriculum development.
Four years into the reform, changes in par-ents’ perceptions of
science, in principals’ sup-port for inquiry instruction, in
teachers’ practiceof inquiry, and in students’ learning of
sciencewere assessed. These findings articulated boththe challenges
and changes of the systemicinitiative’s efforts.
What Are the Challenges?
The challenges describe five aspects of sys-temic reform that
must be addressed and aligned.Because each one poses risks to a
part of the edu-cation community, varying levels of success
havebeen reached. However, the lessons learned inattempting to meet
the challenges provide in-sights and directions for the future.
Challenge One: Sustained professional de-velopment of a
validated model can produce aculture shift in participants, but it
is costly andtime intensive. Given a state cohort of over
7,500teachers per grade level, there are neither the hu-man nor the
financial resources to reach morethan a small fraction of the
target audiencewithin a five-year period.2
Lesson Learned Not only are human and fi-nancial resources
limiting, but the pool of teach-ers who can, or will, undertake
sustained profes-sional development is limited. Teachers who maybe
characterized as needing professional devel-opment the most---ones
in poor schools, oneswith general licenses, ones with few courses
inscience or mathematics, ones teaching out of cer-tification
areas, ones who are disenchanted ordisenfranchised-do not readily
volunteer for arigorous summer of mathematics or science.Rather,
they must be reached in their communi-ties and schools, the
academic program must beat the level at which they teach science
and/ormathematics, and the materials used must be di-rectly
applicable in their classrooms and withtheir students.
To meet this challenge, research validatedcurricula were
identified. Next, teachers, who
had had at least one year of professional develop-ment, offered
local, 40-hour workshops for theirpeers. Districts supported the
teacher-instructorsand often required all science or
mathematicsteachers at the targeted grade to attend. Ostensi-bly,
the workshops were to help teachers learn touse standards-based
curricula; in reality, muchmathematics and science was learned.
Challenge Two: Any reform has a limited andunique function.
Although it must offer resourcesthat are not available during its
lifetime, thoseresources eventually must be assimilated into
theongoing educational system.
Lesson Learned At the beginning of the re-form, both the
Systemic Initiative and the OhioDepartment of Education divided the
state intoeight professional development regions. Two setsof
centers were established that tested two differ-ent paradigms for
professional development. TheSystemic Initiative insisted upon
regional col-laboration before identifying and supporting
itsregional centers, which, then, delivered thesix-week institutes.
They were taught by regionalacademic leaders (both outstanding
teachers andPh.D. scientists and mathematicians) who wereavailable
during the school year to assist teachersin their classes, to work
on curriculum teams, andto provide local workshops. The
SystemicInitiative’s model was successful beyondanyone’s
expectations. The Department’s lesscostly model involved
short-term, usuallythree-day, workshops with limited
classroomfollow-up. The teaching staff rotated, and manywere
imported for a few days work. Furthermore,the selection process for
the center was competi-tive, resulting in antagonism between the
unitsfunded (districts, colleges, county offices) andthose passed
over.
It was obvious that the two regional unitsneeded to merge, and
merger has occurred. Themergers have been slow and fraught with
diffi-culties; for example, how to retain the sustainedprofessional
development model within themerged regional center. However,
systemic re-form is about taking chances, about building
con-sensus, and about moving ahead with the results.In a large
complex state, regionalization isneeded, and it is better to have
one unit than
12
-
none, and two units are not sustainable. Further-more, the
merged center is fully assimilated intothe state’s educational
system.
Challenge Three: Systemic Initiatives haveunderestimated the
difficulty of getting mathema-ticians and scientists-at all
levels-to work to-gether, the difficulty of shifting university
facultyfrom teaching by lecture to inquiry, as well as
thedifficulty in communicating between campusesand across
disciplines. In Ohio, two separategroups of mathematicians
developed mathemat-ics by inquiry courses, because that process
wasmore efficient in terms of time than the collabo-ration
necessary to develop one course. Later,when middle school
mathematics and scienceteachers were both in institutes on the same
cam-puses (often in adjacent classrooms), both groupsactively
resisted working on integrated units orcomparing strategies across
disciplines.
Lesson Learned It takes time and effort toencourage
collaboration. Yet, without it, a reformis only pockets of change,
not systemic. There-fore, the Systemic Initiative instigated
collabora-tions with Ohio’s three Urban Systemic Initia-tives and
the Appalachian Rural Systemic Initia-tive. Further, it identified
collaborative relation-ships with Ohio’s Mathematics and Science
Coa-lition, the Parent Teachers Association, and manyregional and
local businesses and foundations.
Challenge Four: Although Ohio’s SystemicInitiative focused on
individual (or groups of)teachers, a school is a more viable unit
ofchange. Teachers need a support system for thereforms they are
initiating in their classrooms. Aschool focus also produces the
visibility to attractexternal market-driven resources that may
con-tinue the reform after the funding period.Changes in a school’s
science program or sciencedepartment-with documented improvement
inresults-is a phenomenon that may be quicklycommunicated to
parents and policymakers.
Lesson Learned As part of the assessment,described below, brief
site visits were made to 12schools, primarily in poor urban or
rural areas.Both the observational and questionnaire datagathered
suggested that there were greaterchanges in learning environments,
in teaching
practices, and in student outcomes in schools thathad a critical
mass of reform teachers (up to one-third of the mathematics/science
faculty), com-pared to schools with only a few teachers, or
anisolated teacher with the sustained professionaldevelopment. A
supportive group of teachers isespecially important because of the
high mobilityof principals.3
Systemic or standards-based reform requiresa critical and
self-sustaining mass of teachers in aschool. The solution was
twofold: first, districtswere requested to require or provide
incentivesso that all appropriate teachers would be in-volved in
local professional development; and,second, the intensity and depth
of the profes-sional experiences were moderated with cautionand
some trepidation. (See Challenge One.)
Challenge Five: All initiatives that are sys-temic in nature
have important research/develop-ment and dissemination/support
roles.Well-researched and validated professional de-velopment
packages, such as Physics by Inquiry(McDermott, Shafer, &
Rosenquist, 1996) havesustainability independent of the
instructors.Such packages can be assimilated very quicklyinto
existing delivery systems, such as a state’sregional centers. Where
such packages do notexist (or lack research validation), the
SystemicInitiative must take the professional developmentpackages
through carefully controlled field testsand refinement activities
to document their valueand sustainability through research
studies.
Lesson Learned There is neither time normoney to do it all. Ohio
has learned two lessons.First, find and use the expertise of
others, and,second, assess progress and outcomes in order torefine
and improve your efforts. As mentionedearlier, in the fourth year
of the reform, the Sys-temic Initiative undertook a major study to
de-scribe the landscape of science and mathematicseducation in the
state. The intent of the assess-ment was to tell the reform story
in terms ofchanges in learning environments, in teachingpractices,
and in student learning. Because of thefocus on equity, the schools
selected to gatherstudent achievement data were in poor urban
orpoor rural districts.
1 3
-
What Are the Outcomes?
Four years into the reform, a comprehensiveassessment of
learning environments; teachingpractice; principal, teacher, and
student attitudesas well as student learning was undertaken.
Thestudy consisted of two levels and involved thecollection of both
quantitative and qualitativedata. Level one consisted of a random
sample of126 schools, drawn from all the schools in thestate that
had at least one teacher who had com-pleted the initiative’s
professional training. Atlevel one, principals and all teachers who
taughteither science or mathematics (grades 5 through9) completed
questionnaires concerning class-room instruction, administrative
support, andparent influence as well as issues of schoolchange.
Level two consisted of brief site visits to12 (from the original
random sample of 126) se-lected schools. In those schools, students
andparents also completed questionnaires, studentscompleted
achievement tests, and principals,teachers, and randomly selected
students and par-ents were interviewed. At each site visit school,
ateacher who had had the sustained professionaldevelopment (reform
teacher) was matched witha teacher who had not had that
experience(nonreform teacher). In addition, a randomly se-lected
class of the reform teacher was matchedwith a comparable class of
the matchednonreform teacher. Changes were identified bycomparing
the responses of these matched groupsof teachers and students.
Using 1990 and 1992 public release itemsfrom the National
Assessment of EducationalProgress (NAEP), science and
mathematicsachievement measures were developed by teamsof faculty,
teachers, and regional leaders. Testitems focused on process, not
product, becausethe reform’s goals were to increase
conceptualunderstanding as well as skills needed to inter-pret and
use scientific and mathematical informa-tion. In Miller’s (1996)
discussion of barriers tosystemic reform, he notes that there is
often adisconnect between the practical paradigm of re-form
(focused on process) and the technical para-digm of education
(focused on product). This dis-connect was avoided by developing
new achieve-ment measures. When possible the questionnaires
for principals, teachers, students, and parentscontained the
same questions, phrased appropri-ately. That strategy allowed
comparison of re-sponses across groups. For example, did
bothteachers and students respond that manipulativeswere used at
least once a week? The results indi-cate that students responded
similarly to theirteachers concerning instruction in reform
andnonreform classes. That is, students in reformclasses
significantly more often talked with eachother about the subject,
had to support theirclaims, and were encouraged to ask
questions.Interestingly, significantly more students andteachers in
reform classes reported that theirprincipal had learned to accept
classroom noise.
When the ways in which students learn wereexamined, interesting
and significant differenceswere found between reform and
nonreformclasses. Students in classes taught by reformteachers
significantly more often wrote abouthow they solve problems, solved
problems insmall groups, and used hands-on manipulatives.Those
strategies are recommended both by theNational Science Education
Standards (NSE$National Research Council, 1996) and by the
re-search literature concerning strategies to improvethe
participation, attitudes, and achievement lev-els of girls and
minority students. One of thesix-week content courses incorporated
the com-puter as a learning tool; another heavily usedgraphing
calculators. Further, both the NCTMstandards and NSES argue for the
incorporationof technology into science and mathematics les-sons
(National Council of Teachers of Mathemat-ics, 1989; NRC, 1996).
However, neither practicewith, nor information about, the efficacy
of tech-nology in promoting learning affected the use ofcalculators
or computers. The lack of appropriateequipment and software remains
a major chal-lenge to implementing the reform agenda.
Briefly, there were significant differences inscience
achievement in favor of reform classes,as measured on the Discovery
Inquiry Tests. It isimportant to note that it was a low-stakes
test(grades were not affected) and that it focused ona student’s
ability to interpret information and onconceptual, not factual,
understanding. Becauseof the reform’s focus on equity, the results
alsowere examined for any gender, race, and/or
14
-
group (reform versus nonreform) differences.Those analyses
revealed interesting patterns ofachievement; for example, minority
students (inthis case, African American) in reform classesscored
significantly higher than their peers innonreform classes. In fact,
African American sev-enth and eighth graders in reform classes
scoredas well as white students in nonreform classes.Because the
data were collected from reform andnonreform students taking the
same type of class(e.g., general science, introductory life
science)with a “matched” teacher in the same school,economic
differences were not a major factor.
Further, in science classes taught by teachersinvolved in the
systemic reform, both AfricanAmerican and white females scored
higher thanthe males in their racial group. In reform classes,white
females, shown by other research to be thegroup most socialized
away from science(Campbell, 1991; Campbell & Connoly,
1987;Kahle & Damnjanovic, 1996), scored higher thanwhite males
or African American females ormales. When science test scores were
subdividedinto physical science and life science items, fe-males in
classes taught by the reform teachersscored higher than males on
the physical scienceitems. These fmdings contrast with those of
otherstudies. Using a large national data set (NELS:88), gender
differences have been found inachievement in physical science, but
not in biol-ogy (Burkham, Lee, & Smerdon, 1995). Theseresults
may be the first time that a genderachievement gap in physical
science, favoringgirls, has been reported. Clearly, the type
ofteaching observed and recorded in responses tostudent and teacher
questionnaires-more use ofmanipulatives, more time to talk about
science,more opportunities to write about science, in-creased use
of cooperative learning groups-hasaffected achievement,
particularly for studentswho have been underrepresented in
science.
A Further Challenge
Once the data were collected and analyzed,the challenge was to
distribute the findingswidely in an accessible way. Over 10,000
copiesof a small, easy-to-read publication, the PocketPanorama
(Kahle & Rogg, 1996), have been dis-
tributed across the state and nation. Becausemost state
departments of education do not haveeither the time or the
expertise to performlarge-scale research and dissemination
activities,documentation, validation, and dissemination ofchange
and of best practice provide unique andimportant roles for Systemic
Initiatives. Such ac-tivities are critical for the public’s
understandingand acceptance of systemic reform. Indeed, re-search
and dissemination may be the key rolesfor externally funded reform
initiatives within astate. The final lesson learned is that
research orassessment without dissemination benefits onlythose who
are already involved in the reform.Dissemination of findings in
practical andeasy-to-use ways informs others of theinitiative’s
success and invites them to becomeactive participants in it.
Further, assessment,coupled with wide-spread dissemination,
pro-vides the basis for successful reform strategies tobecome
sustained through the existing educa-tional system.
All parts of a reform must be addressed andwork together if the
results are to be systemic.The challenges in one state led to
alterations inits reform strategies; those alterations, in turn,led
to wider participation and acceptance of thereform. Although the
changes described in theassessment cannot be directly attributed to
theprofessional development and support strategiesthat were part of
the reform, the findings suggestthat improved learning is
associated with im-proved practice that is initiated through
sustainedprofessional development.
Notes1. Ohio was one of the first ten states to receive Na-
tional Science Foundation funding for a StatewideSystemic
Initiative. Further, Ohio’s three cities thatwere eligible for
Urban Systemic Initiative funds,Cincinnati, Cleveland, and
Columbus, have beenawarded grants. Ohio also has five counties in
theAppalachian Rural Systemic Initiative.
2. NSF Systemic Initiative awards are for up to fiveyears,
although contracts are renewed annually.Three of the first cohort
of states were terminatedduring the five-year cycle.
3. In Ohio, over 50% of principals are in that positionin a
particular school for four years or less.
15
-
ReferencesBurkham, D. T., Lee, V. E., & Smerdon, B. A.
(1995).
Gender and science learning early in high school:Subject mat ter
and laboratory experiences . Unpub-lished paper, University of
Michigan at Ann Arbor.
Campbell, J. R. (199 1). The roots of gender inequityin
technical areas. Journal of Research in ScienceTeaching, 28,25
l-264.
Campbell, J. R., & Comroly, C. (1987). Decipheringthe
effects of socialization. Journal of EducationalEquity and
Leadership, 7(3), 208-222 .
Kahle, J. B., & Rogg, S. R. (1996). Apocketpan-orama of the
landscape study, 1995. Oxford, OH:Miami University.
Kahle, J. B., & Damnjanovic, A. (1996, April). Esjrectsof
inquiry teaching on the achievement levels of
urban middle school science s tudents by race andsex. Paper
presented at the annual meeting of theAmerican Association for the
Advancement of Sci-ence, Baltimore, MD.
McDermott, L. C., Shafer, P. S., & Rosenquist, M. L.(1996).
Physics by inquiry. New York: Wiley.
Miller, K. W. (1996). Paradigmatic school philoso-phies as
barriers to school reform. Science Educa-tor 5(l), 1-6.
National Council of Teachers of Mathematics. (1989).Curriculum
and evaluat ion s tandards for schoolmathematics. Reston, VA:
Author.
National Research Council. (1996). NationaZ scienceeducation
standards. Washington, DC: NationalAcademy Press.
Uri TreismanProfessor of MathematicsDirector; Charles A. Dana
Center for Mathematics and Science Education and the Texas
SSIUniversity of Texas at Austin
My purpose in these remarks is to commentnot only on what we,
the Dana Center/SSI, havelearned, but also on what we are trying to
learnabout systemic reform as it is practiced in Texasand other
large states. I have organized my re-marks around the first four
drivers of systemicreform as enshrined in the NSF’s assessment
andmonitoring system for its systemic initiatives. Inmy view, these
drivers do in fact capture the es-sential dimensions of the Dana
Center/SSI’swork and provide quite a useful framework fordescribing
the complex of interlocking projectsand initiatives that we have
found necessary formoving the system forward in ways that
respectour fundamental commitment to equity.
I. Policy
The extent to which state or local educationalpolicies affect
practice is determined by the na-ture and sharpness of the teeth
associated withthem in the relevant accountability system. Inthis
sense, managing the policy side of systemicreform can be thought of
as a kind of educationalorthodontics-requiring lots of steady
pressure atjust the right place constantly applied. In the
work in question, generating and maintaining therequired steady
pressure requires the develop-ment and maintenance of relationships
with indi-viduals and groups whose interest in mathematicsand
science is quite minor. But more on this later,in my discussion of
public engagement.
It is my observation that in many states thepolicy focus of the
mathematics and science edu-cation communities has been almost
exclusivelyon shaping particular policy documents, i.e., onsetting
the content of state curriculum frame-works. Too little attention
has been paid to themechanisms through which these documentshave
their influence-an essential issue in sys-temic reform.
In Texas, for example, the processes for suc-cessfully
developing state curriculum f&me-works and for getting them
adopted by a conser-vative State Board of Education were
remarkablysimilar for mathematics and science. Unusualand sometimes
unnatural coalitions had to bebuilt; a broad perspective had to be
inculcated inindividuals who are both skilled in and fond
offighting over arcane and often politically mean-ingless issues.
Incentives had to be created forvarious leaders to support
positions that differed
16
-
in minor ways from those of the professional or-ganizations that
give them authority and specialinfluence. Most challenging, of
course, was pro-ductively engaging the opponents of standards-based
reform, who often argued vociferouslyabout the fine details of the
proposed curriculumframeworks but did so in ways that were
actuallyintended to shake confidence in public education.
Yet, these now-adopted mathematics and sci-ence frameworks have
very different effects onpractice at the school and district
levels. By law,each constitutes the legal base of any state
ex-aminations in its subject area. But, herein lies thecrux. While
mathematics is tested at many gradelevels, science is tested only
at grade eight.Moreover, state ratings of school performance,which
are widely published and attended to by-among others-the real
estate industry, are influ-enced heavily by mathematics scores but
not atall by science scores. The effect is profound.School
administrators spend heavily to help theirstudents learn
mathematics; they spend no morethan is absolutely necessary on
science unless itis a special interest of theirs, their spouse’s,
or ofan all-too-rare group of their teachers who areboth science
people and effective negotiators.
This unpleasant reality has consequences forthe kinds of
networks one builds, the state-levelprofessional development one
offers, and so on.In mathematics, one can count on felt need
todrive action. In science, one dependson argu-ments about the
general good, the future, and soon-arguments that, in the absence
of a clearthreat to the nation’s security, are very, very hardto
make.
Whereas, in mathematics, the Texas SSI cansupport (and find
support for) the implementa-tion on a large scale of high quality
curriculasuch as those developed with NSF support, inscience, the
primary challenge is to organize po-litical support for the
inclusion of science in theaccountability system. In practice, one
confrontsa Catch-22 situation. The legislature will onlymove to
include science in the accountabilitysystem if it believes that the
school system canmeet the challenges it will generate. Few
legisla-tors want their constituents’ schools to face yetanother
challenge they cannot overcome. Thesesame legislators will only
support increases in
funding for science if they feel an overwhelmingdemand from
their constituents. This demand isnot there because it is not
generated by the ac-countability system. And so on.
In such an environment, the natural but verylong-term strategy
is capacity building-creatingand supporting the kinds of statewide
profes-sional development networks, collaboratives,etc., that will
generate reasonable confidence thathigher standards can indeed be
met. Until then,science standards will serve as a banner and agoal.
Mathematics standards will be drivers oflocal action.
Another and even more important questioncenters on equity and
the allocation of scarce re-sources in the educational system. It
is widelyassumed in our community-indeed it is themantra of
systemic reform-that standards-basededucational policy is a friend
of, if not a prereq-uisite for, equity. Roughly, the most common
ar-gument for this position is that explicit standardsdemystifl the
system by making public exactlywhat children must know and be able
to do ineach curricular domain. In being public, stan-dards enable
the transformation of a system puta-tively based on ability to one
that is manifestlybased on effort. The needed catalyst (which,
ofcourse, is rarely present) is adequate resources sothat the
playing field is approximately level forall students, no matter
their family’s financial oreducational resources.
In Texas there is a particular and, in my view,all-important
feature of the state’s educationalaccountability system that may
turn out to be anecessary and sufficient condition for greater
eq-uity to be an outcome of standards-based reform.Specifically, in
Texas, schools and districts aregiven one of four ratings ranging
from “low per-forming” to “exemplary,” based mostly on stu-dent
performance on state examinations (TexasAssessment of Academic
Skills) whose content isdirectly defined by the state curriculum
fiame-works. School data are reported for all studentsand
separately for four subgroups: AfricanAmerican, Hispanic, White,
and EconomicallyDisadvantaged students. To achieve a given rat-ing,
the scores of every subgroup must exceed aparticular cut-off score,
which rises over time ona predetermined schedule. Sanctions for low
per-
1 7
-
formance include public hearings and possibletakeover by the
state. In short, school and districtratings depend on the success
of everybody’schildren.
The effects of this rating system in Texashave been
extraordinary and are increasinglywell known. Ten years ago, Texas
student perfor-mance mirrored that of the South-it was abys-mal.
Today, the average scale scores of each ofthe above mentioned
populations on NAEP math-ematics tests are near or at the top of
nationalrankings. Indeed, on NAEP Grade 4 Mathematicsscores, Texas
is tied for sixth place-onlyslightly behind Maine, Minnesota,
Connecticut,Wisconsin, and North Dakota. Texas fourth-grade scores
now exceed those of such tradition-ally high performing states as
New Jersey, Mon-tana, Michigan, Colorado, Vermont, and
Utah.Amazing. The mechanism of action is clear andeasily visible
from the ground floor where Ispend most of my days. Today, building
adminis-trators in Texas are far more likely than in thepast to
direct their resources to supporting highachievement among groups
of students who theybelieve might endanger their school
accountabil-ity ratings. The focus on mathematics (still
thedominant reason for low school ratings) startlesvisitors from
out of state. Indeed, this feature ofthe Texas accountability
system, in concert withthe recent changes in the Improving
America’sSchools Act, has set a new benchmark for thespeed at which
a large state’s performance datacan change.
In systems in which school ratings depend onaverage
(nondisaggregated) scores, the naturalstrategy for a building
administrator would likelybe to direct resources to those children
whom heor she believes can be most easily educated. Thisstrategy
would keep the performance gap amongethnic groups large and might
increase it. Indeed,Texas is one of the few states where
differencesin performance in mathematics among ethnicgroups is
decreasing-at least at the K-8 levels,where the accountability
system is strongest.
In short, there is no evidence that standards-based
accountability by itself is a friend of eq-uity. There must be
significant incentives for ad-dressing equity, but unless there are
explicit andsubstantive consequences for not educating to
high levels everybody’s children, unless the partsof the
accountability system that deal with equityhave teeth, it will
continue to be minorities whoare underserved by the educational
system.
II. Curriculum and Instruction
Perhaps the greatest surprise to emerge fromour work in the
Texas SSI has come from our re-search on effective school responses
to our high-stakes standards-based accountability system. Wehave
now studied quite carefully 26 high pov-erty/high minority
elementary schools whose stu-dents score well above the state
average at everygrade level on state examinations in both
math-ematics and reading. These are schools whoseperformance
profiles are similar to those of typi-cal schools in the wealthiest
suburbs of Texas.We have also studied the 13 Texas high schoolsin
Title I feeder patterns, i.e., in high povertycommunities, with the
highest average scores onthe state’s End-of-Course Algebra
examination.
What did we find? First, we found enormousvariations both in how
schools organized them-selves to succeed and in the particular
curriculumand instructional approaches they adopted. Yes,some of
the high-performing high schools useSaxon’s Algebra. Others use
books that wouldmake any NCTM leader happy. At the elemen-tary
level, we found as many examples of schoolsthat espoused direct
instruction/explicit phonics-based approaches as we did schools
that identi-fied themselves with “constructivist” approaches.The
eclectic pragmatism of these schools im-pressed us. The teachers
were clearly more com-mitted to nurturing students than to climbing
onbandwagons.
It is, of course, one thing to espouse or toidentify with a
philosophy and quite another toactually practi