Measuring - Institute for Student Achievement...measuring 21st-century skills and using data to fuel change has had fortunate results. Although the percentage of students receiving

MeasuringMike Schmoker

Data-driven decisionmaking is here to stayThroughout the lastdecade, educators havecome to embrace data as

an indispensable tool for schoolimprovement. Like our colleagues inindustry and medicine, teachers havelearned that data help us identify priori-ties for improvement. When schoolsformally measure and publicize theirweaknesses, addressing problem areasacquires new urgency

The marriage between the data-drivenmovement and No Child Left Behindinstigated many positive actions, espe-cially on behalf of low-performingsubgroups. As Tom Peters (1987) wrotetwo decades ago, "What gets measuredgets done." But data-based reform hasalso had less happy consequences. Inmany schools, it has morphed into anunintended obstacle to both effectiveinstruction and an intellectually rich,forward-looking education.

For data-driven instruction to trans-form schooling-which it can-it mustserve a master very different from rigidaccountability formulas. It must aim tohelp students from all backgroundsattain an authentic 2 1st-century educa-tion. A good model is the New YorkPerformance Standards Consortium, aconsortium of 28 high schools in NewYork State that uses data-driven decisionmaking in service of 2 1st-centurylearning, with both simplicity andsuccess. Advocates of data-driveninstruction can learn much from this

t,'

consortium. But first, let's look at what a2 1st-century education requires andwhether most schools are using data tohelp or hinder this kind of education.

What Is a 21st-Century Education?The most prominent advocates of 21st-century education all stress the impor-tance of learning essential content byway of authentic intellectual skills.These advocates' documents invariablycontain the terms critical thinking andproblem solving. They emphasize theability to argue, analyze others' argu-ments, conduct research, and acquiresuch "habits of mind" as the ability toinvent or synthesize information.Literacy-the ability to read, write, andmake effective presentations-is central(Berliner & Biddle, 1995; Kay &

Houlihan, 2006; Pearlman, 2006).College professors and business exec-

utives whom I've interviewed concurthat such skills are crucial to students'success as college freshmen, profes-sionals, and participants in a democracyYet these leaders uniformly lament thatK-12 schooling does not emphasizesuch an education. This raises the ques-tion, Does the focus on data nowpresent in public schools promote 2 1st-century teaching and learning?

When Data Interfereswith ImprovementTo answer this question, we must realizethat our current data-driven decisionmaking is to a great degree standardized-test-data-driven decision making. This isnot all bad. As W James Popham (2001)writes, standardized test items canoperate as proxies-imperfect but usefulindicators of legitimate learning. Indeed,when higher test scores are the result ofsound curriculum and effectiveteaching, we should applaud thesegains. There are many schools that usedata to drive a curriculum rich in 21st-century skills and that achieve bothhigh test scores and deep learning.Tempe Preparatory Academy inPhoenix, Arizona, and View ParkPreparatory in Los Angeles, California,(a majority-minority school with half itsstudents living below the poverty line)are two prime examples (Hernandez,Kaplan, & Schwartz, 2006).

But as those of us who spend signifi-cant time observing in schools haveseen (Schmoker, 2006), few schoolstruly focus on these key skills. Just as

70 EDUCATIONAL LEADERSHIP/DECEMBER 2008/JANUARY 2009

Noo

Schools must collect data

What Mattersthat serve a 21st-centuryagenda. A consortium ofNew York schools shows how.

the ability to write and think critically,evaluate, and discern facts from opin-ions. After all, these skills abound on

Sstate standards documents and corre-sponding assessments (Liben & Liben,2005). Authentic literacy learning didincrease in some schools. Standardizedtest data reflected this reality, and weshould celebrate that fact (DuFour,DuFour, Eaker, & Many, 2006; Reeves,2006; Schmoker, 2006).

Consortium schools

have a process to

ensure that their

focus on rubric datadrives improvement.

education leaders were beginning to winhearts and minds to the importance ofauthentic intellectual tasks and assess-ments, our attention was turned-toomuch, perhaps-to the numbers them-selves. And we made an unpleasantdiscovery: Schools and even wholestates could make steady gains on stan-dardized tests without offering students

intellectually challenging tasks.Steadily improving scores didn't alert

leaders to this paradox. In literacy, forinstance, many of us assumed that data-based accountability would spur ageneral increase in reading and writingassignments relevant to 21st-centuryconcerns in all disciplines. We believedthat such assignments would promote

But we can't discount evidence thatfixating on data hindered instructionalimprovement in many schools. TeachersI've talked to in dozens of states admitthat the quality of what they teach andhow they teach hasn't changed thatmuch since data grabbed the focus.Test-prep activities-not authenticteaching and learning-were respon-sible for much of the increase in testscores.

This explains why achievement gainson state tests are often at odds with stag-nant performance on the NationalAssessment of Educational Progress(Cavanaugh, 2007). It explains whyhigher passing rates on standardized

ASSOCIATION FOR SUPERVISION AND CURRICULUM DEVELOPMENT 71

tests have had little effect on the highproportion of students who enroll inremedial college courses (Fitzhugh,2007; Kollars, 2008).

In the last few years, at the invitationof school administrations, I haveobserved in several schools with goodreputations. I found that in most class-rooms, ill-devised lessons, aimlessgroup activities, and busywork predom-inated. Worksheets were everywhere,and movies blared through school hall-

ways. There was almost no emphasis oncritical thinking, problem solving,reading, discussion, or writing. Ironi-cally, faculties showed little interest inimproving instruction because each ofthese schools enjoyed exceptional testscores, and some had received theirrespective state's highest rating foracademic achievement. The data itselfcreated a ceiling on instructionalimprovement.

A More Forward-LookingApproachFortunately, forward-looking institu-tions like the New York PerformanceStandards Consortium have begunusing data to support instruction forauthentic learning. Because consortiumschools operate on a waiver, studentsonly have to take one of the state' fiveRegents exams (the English exam).Regents exams determine whether NewYork high school students receive creditfor core courses and graduate. ThomasSobol, then New York State's commis-sioner of education, granted the waiverin 1991 because after observing the

success of exemplary schools, includingthose in the consortium, he believedsuch schools could be even more inno-vative if they were relieved of theRegents requirements. With test-scorepressure eased, these schools are free toconcentrate on measuring the kinds ofskills the world beyond the classroomdemands.

Instead of test prep, students andteachers focus on work that culminatesin four or more final projects in core

academic areas: (1) a literary analysis,(2) a science experiment and relatedresearch project, (3) an extended mathe-matics problem-solving project, and (4)a research paper in social studiesdemonstrating the use of argument andevidence.

All projects require students to read,think critically, write, discuss, doresearch, construct an argument, andpublicly present their knowledge. Theintellectual richness of these projects isevident in sample titles, such as "WhyDo They Have to Die? A ComparativeAnalysis of the Protagonists' Deaths inDr Jekyll and Mr Hyde, Metamorphosis,and Of Mice and Men" and "Finding theParabolic Path of a Comet as It MovesThrough the Solar System." A set ofrubrics accompanies each project-related task. The entire process isinformed by Wiggins and McTighe's(2005) notion of "backward design."

The consortium encourages anyschool to liberally borrow from itspractices. The consortium's Web site(http://performanceassessment.org/performance/index.html) features links

to rubrics used in consortium schools,guidance in implementing performanceassessment, and pacing charts and plan-ning documents (under the Forms andCharts link).

Rubrics listing criteria for satisfacto-rily accomplishing tasks like analyzing aliterary work and performing scientificresearch were developed by consortiumfaculty, college professors, and localprofessionals-who also identifiedexemplar "anchor" projects. This grouphelps evaluate whether each studentswork meets graduation requirements.

Consortium schools analyze mainlytwo types of data to inform instruction:

m Information on how many studentsare on track to successfully complete themajor projects required for graduationand how many need guidance.

* Data on students' individualperformance on key rubrics.

Rubric data informs teachers' profes-sional conversations and professionaldevelopment-most of which occurs inhouse within teacher teams. Staff devel-opment sessions and team meetingsfocus on how well the school's methodof operating helps learners meetcommon criteria like the following,taken from consortium rubrics:

m For a science experiment: Hypoth-esis reflects a synthesis of primary back-ground research. Creates graphs andcharts that reflect the use of basic alge-braic functions and multiple statisticaltests.

* For a social studies paper: Includesdetailed evidence drawn mainly fromprimary sources. Evaluation of substan-tial number of opposing/varied sources.

How Data DriveConsortium TeachingConsortium schools have a process toensure that their focus on rubric datadrives instructional improvement. Everysummer, a group of representativeteachers, college professors, and otherprofessionals conduct "moderationstudies" of 150 sample projects repre-

72 EDUCATIONAL LEADERSHIP/DECEMBER 2008/JANUARY 2009

For data-driven instruction to transform

schooling-which it can-it must serve

a master that is very different from

rigid accountability formulas.

senting each school. This group deter-mines whether these projects as a wholemeet the rigorous standards of theanchor projects and pinpoints whereschools meet or fall short of criteria. Thegroup gives feedback to individualschools so that they can adjust theirteaching.

This process revealed to UrbanAcademy, for example, that its students'written work needed to contain more

Data review goes on continuallyduring the school year. Teacher teamsmeet weekly to evaluate student workand compare it to both the anchorpapers and graduation requirements.Academic departments at each schoolconduct interim assessments three timesa year to ensure that the quality ofstudent work is consistent with estab-lished criteria. Unlike typical end-of-year reviews of test scores, these

T0

analysis and evaluation. According tothe school's codirector, Ann Cook,"Some papers contained too muchmaterial that was merely summative,but not sufficiently analytical." DanielDrmacich, principal of consortiummember School Without Walls, whichhas participated in moderation studiesfor two years, has found that thisfocused review gives teachers an extrapush and helpful direction in makingstudent work come up to higher levels:

Teachers all know that it is not only theirstudents' work [that] reviewers arereviewing, but also their own. There's asubtle feeling of tension among teachersas we do the moderation studies becausethey feel pressure to make sure their kidsare creating quality work. Almost all staffhave interpreted this undercurrent aspositive in their quest for helpingstudents grow and develop.

analyses provide critical feedback onareas in which students are-andaren't-performing well enough to ulti-mately succeed on their graduationprojects.

During such a team meeting, severalteachers at Drmacich's school learnedhow a colleague was smoothly inte-grating the final academic projectsrequired for consortium schools into herclass's overall academic units, ratherthan adding these projects on as some-thing extra. Inspired by her planning,this teacher's colleagues began inte-grating the social studies research paperand literary analysis projects intononcore classes, including courses in artand health.

Consortiumwide, this process ofmeasuring 21st-century skills and using

data to fuel change has had fortunateresults. Although the percentage ofstudents receiving free or reduced-pricelunch in consortium schools (more than60 percent) is higher than average forNew York City schools (Foote, 2007),the schools' dropout rate is 9.9 percent,compared with 19.3 percent in NewYork schools overall. Ninety-one percentof consortium students are acceptedinto college compared with 63 percentin the city as a whole. According toMartha Foote, the consortium's researchdirector, graduates report that once theyget to college they are more competentin writing and revising than their class-mates are (Olson, 2006).

Expanding the ModelFollowing the consortium model, wecould easily create a system throughwhich schools continue to administerstandardized tests but subordinate thefocus on test scores to assessments of2 1st-century skills modeled on theconsortium's rubrics or other effectivesystems. There is now consensusamong educators that students are ableto read, write, argue, and evaluate fromthe earliest grades. Schools could assesscompetence in such skills through whatGrant Wiggins (1998) has called"educative assessments" in the finalgrades of elementary and middleschool, not only in core courses, butalso in art, music, and languages. Forother grades, we could design perform-ance-based assessments that showstudents have mastered relevant skills,and we could nurture these skillsthroughout the year.

Rhode Island has launched an assess-ment program similar to the New Yorkconsortium's, demonstrating that statescan convert their creaking, 20th-centurysystems into systems promoting whatstudents need now We must harnessthe power of data to make school moreinteresting and relevant-and to preparestudents to succeed as workers, collegestudents, and informed citizens. ID

ASSOCIATION FOR SUPERVISION AND CURRICULUM DEVELOPMENT 73

ANNUA COVNTO & EXOITOFERUR 2-ARC ,20 SAN DIGO CA

T argeted professional development will bestructured around the core areas of Breaking

Ranks, including collaborative leadership; PLCs and

personalization; and curriculum, instruction, and

assessment.

Special strands will focus on:

" Assistant principals

"* Middle level educators

"* Urban educators

"* Web 2.0 technologies

Save with team discounts. Bring your leadership

team and get everyone on board with the latest in

educational trends and technology!

$75 discount off registration when you preregister

by February 6, 2009.NATIONALASSOCIATION

SECONDARY SCHOOL

PR INC IPAL S

ReferencesBerliner, D. C., & Biddle, B. J. (1995).

Myths, fraud, and the attack on America'spublic schools. Cambridge, MA: Perseus.

Cavanaugh, S. (2007). State tests, NAEP,often a mismatch. Education Week,26(41), 1.

DuFour, R., DuFour, R., Eaker, R., & Many,T. (2006). Learning by doing: A handbookfor professional learning communities atwork. Bloomington, IN: Solution Tree.

Fitzhugh, W (2007, March 29). Edupunditmyopia. EdNews.org. Available: www.ednews.org/articles/95 10/1/Edupunditmyopia/page 1.html

Foote, M. (2007). Keeping accountabilitysystems accountable. Phi Delta Kappan,88(5), 359-363.

Hernandez, A., Kaplan, M. A., & Schwartz,R. (2006). For the sake of argument.Educational Leadership, 64(2), 48-52.

Kay, K., & Houlihan, T. (2006). Redefining"rigor" for a new century. Education Week,25(37), 31, 33.

Kollars, D. (2008, May 12). Despite highschool algebra focus, more students needremedial college math. Sacbee.com (Sacra-mento Bee online).

Liben, D., & Liben, M. (2005). Learning toread in order to learn. Phi Delta Kappan,86(5), 401-406.

Olson, L. (2006). An alternative approach togauging readiness. Education Week,25(33), 28.

Pearlman, P (2006). New skills for a newcentury. Edutopia, 2(4) 50-53.

Peters, T. (1987). Thriving on chaos. NewYork: Knopf.

Popham, W J. (2001). Teaching to the test.Educational Leadership, 58(6), 16-20.

Reeves, D. (2006). The learning leader: Howto focus school improvement for betterresults. Alexandria, VA: ASCD.

Schmoker, M. (2006). Results NOW. How wecan achieve unprecedented improvements inteaching and learning. Alexandria, VA:ASCD.

Wiggins, G. (1998). Educative assessment.San Francisco: Jossey-Bass.

Wiggins, G., & McTighe, J. (2005). Under-standing by design (2nd ed.). Alexandria,VA: ASCD.

Mike Schmoker is the author of ResultsNOW How We Can Achieve Unprece-dented Improvements in Teaching andLearning (ASCD, 2006); 928-522-0006;[email protected].

74 EDUCATIONAL LEADERSHIP/DECEMBER 2008/JANUARY 2009

COPYRIGHT INFORMATION

TITLE: Measuring What MattersSOURCE: Educ Leadership 66 no4 D 2008/Ja 2009

The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited.