Page 1
FormativeEvaluationofNewHampshire'sPerformanceAssessment
ofCompetencyEducation(PACE)
GeneWilhoit,CenterforInnovationinEducationArthurThacker,HumRRO
PaulLeather,NewHampshireDept.ofEducationSusanLyons,CenterforAssessment
(AccesstheEvaluationHere)
Page 2
FormativeEvaluationofNewHampshire’sPerformanceAssessmentofCompetency
Education(PACE)PaulLeather,DeputyCommissioner
NewHampshireDepartmentofEducation
2
Page 3
NHPACE-- PerformanceAssessmentofCompetencyEducation
• PACEisalearningsystemdesignedtocapitalizeonthelatestadvancesinunderstandinghowpeoplelearn.Thegoalistostructurelearningopportunitiesthatallowstudentstograpplewithgainingmeaningfulknowledgeandskillsatadepthofunderstandingthatcantheycantransfertonewreal-worldsituations.
CurrentStatus:• 3yearsofwaiverimplementation,and• ESSAplanningtoresultinWaiverrequest,subjecttoSec.1204implementationbySecretaryDeVos
• OurgoalistoscalePACEacrossourstateofNewHampshireandtoinspireotherstatestoacceptasimilarchallenge.
6/28/17 3
Page 4
PACE– Whatarethebasics?
• First-in-the-nationUSEDapprovedalternativeassessmentandaccountabilitysystempilot
• Reducedlevelofstandardizedtestingalongwithlocallydevelopedcommonperformanceassessments
• SmarterBalanced/SATgivenonceinelementaryschool,onceinmiddleschoolandonceinhighschool(threegradesinsteadofseven)
• Inallotheryears,administrationofcommonandlocalperformanceassessmentsdevelopedbythedistrictsandvalidatedatthestatelevel
Page 5
EngagingtheUSDepartmentofEducation
March27,20121st MeetingwithArneDuncan
Sept.23,20142nd MeetingwithArne
March3,2015
Approval!
5
PD DesignTheoryof
ActionImplementation
NHProcess
Page 6
WhathaschangedwithPACE?
NCLB/Waiver• Alldistrictsheldaccountableatoncetostatesystem
• Statelevelassessment– lasttwoinNHarenationallydeveloped
• AnnualDeterminationsbasedonStateAssessmentssolely
• Thereoftenaretwosystems– stateandlocal
• Educatordevelopmenttiedtoschoolimprovementafterassessmentresultsarereleased
PACE• DistrictsmustmeetguardrailsbeforejoiningPACE
• PACEAssessmentsdevelopedlocally–SystemisStateANDLocal
• Annualdeterminationsbasedonmultiplemeasures
• LocalcommunitiesandeducatorsOWNthesinglesystemofaccountability
• IntensiveEducatorDevelopmentoccursPRIORtojoiningPACE,andcontinuesasPACEinvolvementevolves
6
Page 7
AssessmentContinuum
Extended Performance
Tasks(SCALE,
EPIC, ILN)
Narrow Assessment Assessments of Deeper Learning
CCSS Assessments
(SBAC &PARCC)
Performance Based Items
& Tasks(MARS, BAM)
Student-Designed Projects
(Envision, NY Performance
Standards Consortium, Singapore, IB)
Traditional Tests
Descriptions
Standardized, multiple-
choice tests of routine skills
Systems of standardized performance
items and tasks (1 day to 1 week) that measure key concepts in
thought-provoking items
that require extended
problem solving
Examples
Standardized tests with m-c & open-ended items + short
(1-2 day) performance tasks of some applied skills
Performance tasks that require
students to formulate and carry out their own inquiries,
analyze & present
findings, and (sometimes)
revise in response to feedback
Longer, deeper investigations,(2-3
months) & exhibitions, including
graduation portfolios,
requiring students to initiate, design, conduct, analyze,
revise, and present their work
in multiple modalities LindaDarling-Hammond
Page 8
WhatisPACE?– WaterTowerProposal!• TheProblem:Yourtown’spopulationispredictedtoincreaseoverthenext3years.Asoneofthetownplanners,youareaskedtoaddressthisissueintermsofthetown’swatersupply.Inordertomeetthefutureneedsofthetown,youneedtomakeaproposaltoaddawatertowersomewhereontownpropertythatwillbecapableofholding45,000±2,000cubicfeetofwater.Thetownislookingforawatertowertocontainthemostamountofwaterwhileusingtheleastamountofconstructionmaterial.
• StudentTask:Yourjobistoprepareaproposalthatcanbesubmittedtothetownplanningcommittee.Usingyourcalculationsofsurfaceareaandvolumeforthetwodesigns,describeandanalyzethecharacteristicsthatleadyoutoafinalrecommendation.
6/28/17 8
GeometryPACECommonTask
Page 9
PACE:Local,StateandCommonAssessments
Grade ELA MATH SCIENCE
K-2 Local PBA LocalPBA LocalPBA
3 SmarterBalanced CommonPACEPBA Local PBA
4 CommonPACEPBA SmarterBalanced CommonPACEPBA
5 CommonPACEPBA CommonPACEPBA LocalPBA
6 CommonPACEPBA CommonPACEPBA LocalPBA
7 CommonPACEPBA CommonPACEPBA LocalPBA
8 SmarterBalanced SmarterBalanced CommonPACEPBA
9 CommonPACEPBA CommonPACEPBA CommonPACEPBA
10 CommonPACEPBA CommonPACEPBA CommonPACEPBA
11 SAT SAT CommonPACEPBA
12 Local PBA/CapstoneExhibitions
Local PBA/CapstoneExhibitions
LocalPBA/CapstoneExhibitions
Page 10
WhoisPACE-- WhichDistricts/Schools?
6/28/17 10
Tier1--Year1(2014-15)
AdditionalTier1--Year2(2015-16)
Tiers2and3–Year3(2016-17)
• Rochester• SanbornRegional
• Epping• Souhegan HS
• Concord• Monroe• Pittsfield• SeacoastCharter
• Allenstown• FallMountain• Plymouth• SAU23No
Haverhill• SAU58Groveton• Manchester• Rollinsford• Ashland• SAU39Amherst
andMontVernon• VLACS
AdditionalTier1--Year3(2016-17)
• SAU35WhiteMountains
(asnegotiatedwithUSED)
Page 11
OrganizationtoPACEScaling
AdoptedfromNHDOEgraphic
Tier1ImplementingDistricts
• DataCollection• CalibratingandScoring• BodyofWorkProtocols
Tier2PreparingDistricts
• PerformanceAssessmentDevelopment
• ImplementationProcess
Tier3PlanningDistricts
• DevelopCBE• Competencies,Instruction, AssessmentandGrading
Page 12
PACESystemDesignisbasedonHighPerformingProfessionalLearningCommunities• Whereeducatorsworktogethertocontinuously:
• Reviewcompetencyexpectations, alignedwithcollegeandcareeroutcomes
• Evolveinstructionalpractice tosupportstudentlearningofcompetencies
• Developstrategiestoenhancepersonalizedlearningwherestudentagencyisaprimarystudentoutcome
• ImprovingtheAssessmentsystem tomeasurestudentachievementandgrowthrelatedtocompetencies
• Educatorsreviewstudentworkonperformancetasksregularly
6/28/17 12
Page 13
Headquarters: 66 Canal Center Plaza, Suite 700, Alexandria, VA 22314-1578 | Phone: 703.549.3611 | www.humrro.org
Formative Evaluation of New Hampshire’s Performance Assessment of Competency Education (PACE)
PresentationforNCSA
Presenters: ArthurThacker,HumRROSunnyBecker,HumRRO
June29,2017
Page 14
PresentationOverview
• WhatisHumRRO?• BriefReminderofSelectPACECharacteristics• EvaluationDesign• KeyEvaluationFindings• EvaluationRecommendations• TheStoryofPACE
14
Page 15
HumanResourcesResearchOrganization(HumRRO)• Independent,non-profitU.S.basedorganization• Approximately100employees
• Professionalstaffmostlyeducationresearchers,I-Opsychologists,statisticians,andmeasurementexperts
• Expertise:• Programevaluationsatlocal,state,andnationallevels
EvaluationsofschoolimprovementinitiativesLargescale,longrangeevaluationsofstateandnationaleducationprogramsQualityassuranceforNationalAssessmentofEducationalProgress(NAEP)Independent,real-timereplicationofassessmentresultsasaqualitycontrolcheckforseveralstates
15
Page 16
SelectPACECharacteristics
• Competencyeducation• Performanceassessmentsintegratedintostudents’day-to-daywork• Tier1,2,and3districts• Commonandlocaltasks• Taskbank
16
Page 17
EvaluationDesign(1of2)
• April2016– February2017• Formativeevaluation• TheoryofAction• Datasources
• observationsofmajorPACEevents(e.g.,taskdevelopmentandscoringsessions);• interviewswithstudents,parents,teachers,principals,anddistrictleaders;• classroomobservations;• surveysofteachers;• reviewofmaterialsprovidedtoandbyTier1districts;and• analysesofscoredata.
• AccumulatedevidenceregardingeachclaimintheTheoryofActionoverthecourseoftheevaluation.
17
Page 18
PA
18
PACETheoryofAction
Page 19
KeyEvaluationFindings(1of3)
• ReportincludesdetailedfindingsoneachclaimwithintheTheoryofAction
• Buy-in• Educatorsdeterminewhatisassessed,howitisassessed,andhowthetasksarescored
• Assessmentsintegratedintoinstruction• ReplacesSmarterBalancedassessmentsandrequiresdeepknowledge
• Collaboration• Cross-districtactivities(taskdevelopmentsessions,professionaldevelopment,scoringsessions,standard-setting,etc.)
• LibGuide (onlinerepository)
19
Page 20
KeyEvaluationFindings(2of3)
• Teaching&Learning• Increaseddepthofknowledge• “Testingtowhatistaught”vs.“teachingtothetest”• Learningwhiletesting
• Context• Previousexperiencewithcompetency-basededucation• Districtsize• Substantialworkforteachers
20
Page 21
KeyEvaluationFindings(3of3)
• Continuousprocessimprovement• CurriculumCoordinator• Contentleads• Buddydistrict• Saturdaytaskdevelopmentsession• ProvidingconsultantstosupportPACEcoordinator
21
Page 22
DataAnalyses
22
47.3
53.555.5
44.2
63.3
57.2
48.9
44.6
52.0
44.046.0
51.0
57.0
48.0 47.0
52.0
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
Grade3 Grade5 Grade6 Grade7
Percen
tageofStude
ntsM
eetin
gorExceedingAchievemen
tLevelBen
chmarks(3
or4
)
Mathematics
PACE2014-2015 PACE2015-2016 SB2014-2015 SB2015-2016
Page 23
DataAnalyses
23
43.6
49.7
40.1
48.651.3
55.3
43.1 42.4
56.0
63.0
57.0
63.0
57.0
63.0
59.062.0
0
10
20
30
40
50
60
70
Grade4 Grade5 Grade6 Grade7
Percen
tageofStude
ntsM
eetin
gorExceedingAchievemen
tLevelBen
chmarks(3
or4
)
ELA
PACE2014-2015 PACE2015-2016 SB2014-2015 SB2015-2016
Page 24
DataAnalyses
Math2015 Science2016
Science2016 .487
ELA2016 .317 .637
24
Table 1. Correlation Table Grade 3-4
Science2015 ELA2015 ELA2016
ELA2015 .555
ELA2016 .459 .630
Math2016 .440 .603 .735
Table 2. Correlation Table Grade 4-5
Page 25
DataAnalyses
Math2015 ELA2015 ELA2016
ELA2015 .635
ELA2016 .520 .619
Math2016 .625 .590 .648
25
Table 3. Correlation Table Grade 5-6
Math2015 ELA2015 ELA2016
ELA2015 .470
ELA2016 .482 .586
Math2016 .558 .513 .531
Table 4. Correlation Table Grade 6-7
Page 26
DataAnalyses
Math2015 ELA2015 Science2016 ELA2016
ELA2015 .517
Science2016 .523 .448
ELA2016 .557 .483 .574
Math2016 .488 .477 .590 .541
26
Table 5. Correlation Table Grade 7-8
Page 27
Recommendations(1of2)
OngoingMonitoring1. MonitorandSupportDistrictEngagement
2. EvaluateEffectivenessofCollaborationMethods
Additionaltraining/supports3. ConsiderAdditionalTraining/SupportsforTeachersNotDirectlyInvolvedinCommonTask
Development
4. InfuseEquityandAccommodationsTrainingintoPACEActivities
27
Page 28
Recommendations(2of2)
Routinizeprogramfeatures6. RoutinizeTimelyReviewsofLocalPerformanceTasks
9. ConsiderSystematicallyRecyclingTasks
10. BeginTrackingPerformancefromYeartoYear
TargetedStudies5. InvestigatetheImpactofReading/WritingRequirementsonAccessibility
7. PlanforFutureResearchontheImpactofPACEonTeachingandLearning
8. EvaluatetheBenefitofTimeinProgramonOutcomes
28
Page 29
TheStoryofPACE
• Integratedcompetency-basededucationsystemwithperformanceeventsareadramaticshiftfromtraditionalschooling
• Substantialprogresstowardthisgoal• Sustainability• Scalability• Carefulexpansion,districtbydistrict
29
Page 30
ProgrammaticResponsestoPACEFormativeEvaluation
SusanLyons,Ph.D.CenterforAssessment
CCSSO’sNationalConferenceonStudentAssessment
June29,2017
Page 31
PACEisaboutclassroomimpact
Lyons•Page31
ComplexPerformanceAssessments
•Buildingeducatorcapacity
•Enhancingstudentengagement
TransformingInstruction
•Increasingrigor•Developingstudentagency
CollegeandCareer
Readiness
•Preparingstudentstoengagemeaningfullyintheirpost-secondaryplans
Page 32
IncreasingCollegeandCareerReadiness
Page 33
Supporting All TeachersvConsider Additional
Training/Supports for Teachers Not Directly Involved in Common Task Development
Teachers routinely reported that the process of developing the common tasks greatly improved their own task development process and their approach to assessment. As the program expands, it will be important to maintain that benefit for all participants.
ü 16-17: Hiring of PACE Teacher Leaders within districts to better transmit institutional knowledge to all teachers.
ü 16-17: Investment in online intranet for all PACE teachers to share key documents and resources.
ü 17-18: Expanding high quality performance assessment development training within Tiers 2.
ü 17-18: Developing robust set of common resources for competency-based education and assessment across all three tiers of PACE.
Page 34
Quality Local AssessmentvRoutinize Timely
Reviews of Local Performance Tasks
It is important to ensure that the tasks and rubrics are of sufficient quality to be used to generate student scores and annual determinations. Teachers report that their skill level in developing these tasks improves with each year of PACE participation…. Instituting a system of regular task review will help ensure that happens.
ü 16-17: The NHDOE is now reviewing one major assessment per competency for each PACE course in all of the participating districts.
ü 16-17: Contract with Stanford to review all local performance assessments.
ü Summer ’17: NH DOE will provide feedback to districts related to their assessments systems and targeted supports for those districts in need of additional guidance.
Page 35
Evaluating ImpactvPlan for Future
Research on the Impact of PACE on Teaching and Learning
The positive impacts of PACE on teaching and learning should continue to be externally verified beyond this evaluation. This may be part of a future research agenda when it becomes possible to evaluate the predictive strength of PACE results on college and career performance.
ü On-going: Annual evaluation of student performance on standardized assessments. 2016 yielded some early indications of success.
ü 16-17: Providing access to PACE data to internal and external researchers
ü 17-18: Seeking funding from Hewlett to more deeply understand the connection between learning and engagement in complex performance assessments.
ü 17-18: Begin to longitudinally track trends in career and college readiness (e.g., persistence in college).
Page 36
Formoreinformation:
CenterforAssessmentwww.nciea.org
[email protected]