EMENY- The impact of using Numeracy Ninjas on the … elementary numeracy skills such as: number bonds to 10, adding near doubles with compensating and understanding when it is beneficial

1NumeracyNinjas

TheimpactofusingNumeracyNinjasonthenumeracyfluencyofKeyStage3studentsinalargecomprehensivesecondaryschool

WilliamEmeny

CurriculumLeaderforMathematicsandLeadingPractitioner

WyvernCollege,FairOak,Hampshire,UKSignificantcontributionsbyCatherineBell,ClairHosie,CharlotteKay,HelenLo,LaurenRossandFrancesWalkerWyvernCollege,FairOak,Hampshire,UK10June2016ExecutiveSummaryAfterathree-yearprofessionallearningjourneywecreatedNumeracyNinjas,aKS3-focussednumeracyactivitytosupportstudentsbecominghighly-fluentinelementarynumeracyskillsandskillsthatarethemostimportantfoundationsforstudyingGCSEMathematics.Afterlaunchinginsummer2015,NumeracyNinjasisnowestimatedtobeinuseinover1800schoolsworldwide.InordertohelpschoolsconsideringusingNumeracyNinjasanswerthequestion“whatimpactwillithaveonmystudents’outcomes?”,weconductedayear-longstudytoevaluateitsimpactonthenumeracyfluencyofourYear7and8studentsatWyvernCollege.Thestudyinvolvedover470studentsatWyvernCollegeusingNumeracyNinjas.Wefound:

1. Numeracy Ninjas increased the average student’s gain in fluency with mentalcalculationstrategiesbyatleastanadditional70%whencomparedtothegainsmadebyacontrolcohort1.

1Measuredusinganin-housedevelopeddiagnosticnumeracyassessment

2NumeracyNinjas

2. NumeracyNinjasincreasedtheaveragestudent’sgaininfluencywithessentialskillsrequiredtostudyGCSEMathematics2byatleastanadditional70%whencomparedtothegainsmadebyacontrolcohort1.

3. Numeracy Ninjas had a statistically significant positive impact on the attainmentoutcomesofourKS2level4studentsintermsoftheirfluencywithmentalcalculationstrategies,timestablesandtheessentialskillsrequiredtostudyGCSEMathematics1,2Error!Bookmarknotdefined..

4. TheimpactonKS2level2and3studentswaslikelypositive,butwecouldnotprovestatisticalsignificanceatα=0.05duetorelativelysmallsamplesizes.

5. TheimpactonKS2level5and6student’snumeracyfluencyandtheirfluencywiththeessential skills required to studyGCSEMathematicswere inconclusive.Webelievetheremaystillbesomebenefit inusingNumeracyNinjaswith thesestudents,butdonesomorediagnosticallyandonalowerfrequencybasisthanwithKS2level4(andbelow)students.

6. Bysamplingstudents’performanceonNumeracyNinjasonaweeklybasiswebelievethatwecanuseNumeracyNinjasevenmoreeffectivelyinthefuture.AllKS2level4(and below) students will use Numeracy Ninjas every lesson until they regularlyachieveNinjaScoresof30.Afterapproximatelyweek10teacherswillbeencouragedtouseNumeracyNinjasmorediagnostically,usingAssessmentforLearningstrategies,toidentifywhichtopicsstudentsstillcannotanswerandexplicitlyteachthemhowtoanswerthequestionscorrectly.ThediagnosticapproachwillbeusedfromthestartwithKS2level5(andabove)studentsrunningNumeracyNinjasonaweeklyortwiceweeklybasisuntiltheyregularlyachieveNinjaScoresof30.

2See‘PriorLearningDependenciesNetwork’sectionofthisreportfordetails

3NumeracyNinjas

NumeracyNinjas-threeyearsinthemakingNumeracyNinjaswascreatedfromtheaccumulationofthreeyearsofprofessionallearningfocusingonhowcognitivescienceresearchfindings,particularlythoserelatingtomemory,could be used to improve students' fluency with mathematical concepts that underpinsuccessfulGCSEMathematicsstudy.Theprofessionallearningjourneystartedbyresearchingandidentifyingthespecificmentalnumeracystrategiesandelementarymathematicalconceptsthataremostbeneficialtohavestored in long-term memory. Through this essential prior knowledge being accessibleautomatically,andwithout significantdemandonworkingmemory, these skills, factsandconceptsthendonotbecomecognitive-resource-consumingbarrierstolearningthehigher-levelmathematicalconceptswhichrelyuponthem.Oncewehadidentifiedtheskillsinwhichwesoughtourstudentstohaveahighdegreeoffluency,wethenproceededtodevelopadiagnosticnumeracyassessmenttounderstandourstudents’numeracyfluencyinyear7and8atWyvernCollege.Fromtheresultinganalysis,weobservedwhatweconsideredtobeasignificantnumberof issuesrelatingtoa lackoffluencywithelementarynumeracyskills,evenwithourKS2highestattainingstudents.Subsequently, we studied the literature relating to memory, specifically retention andtransferoflearning,tounderstandhowresearchfindingscouldinformusinourdevelopmentofahigh-impactKeyStage3numeracyinterventionresource.Theaimwasforthistoresultinsignificantandsustainedfluencygains(withinthecontextofthenumeracygapsidentifiedinthediagnostictesting)andultimatelytosupportenhancedGCSEMathematicsoutcomesfortheparticipatingstudents.

PriorlearningdependenciesnetworkIn2013,wevisitedtheMathematicsDepartmentatKingSolomonAcademy,LondonledbyBrunoReddy.Thedepartmenthad(andstillcontinuestohave)astrongreputationfor itsdevelopment and delivery of a 'depth-before-breadth', mastery-based mathematicscurriculumandtheuseofcognitiveresearchfindingstoinformthepedagogicalpracticesoftheteachersattheschool.Duringmyvisit,wemetKrisBoulton,whointroducedustothenotionsofworkingmemory,retrievalandstoragestrength,thespacingeffectandalsothebenefitsofinterleavingpractice.In subsequent reading around the concept ofworkingmemory and aboutCognitive LoadTheory (Sweller, 1988), we realised the importance of this idea within the domain of

4NumeracyNinjas

mathematicsteaching.Specifically,ifstudents'workingmemoriesareoverloadedbyalackoffluencywiththeprior-learningskillsunderpinningthecurrenttopicofstudy,thecognitiveresourcestheyhavelefttodevotetoprocessingthelearningofthistopicarereducedandthe learning is sub-optimal.Anecdotally, if you are trying to teach a lessonon factorisingalgebraicexpressionsandstudentsneedtodevoteconsiderableworkingmemoryresourcestoidentifyingthehighest-common-factorofthetermcoefficients,theresourcestheyhaveleft to think about the concept of algebraic factorisation are reduced. DanielWillingham(2009) talksaboutmemorybeing ‘the residueof thought’. If students spendyour lessonsthinkingabouttheprior-learningskillsthatarethefoundationsofyourlessonobjective,theywillnotbelearningtheintendedobjective.Followingon from this reading, itoccurred tous thatmathematics teachersmayneed toconsidersomelearningobjectivesasmoreimportantthanothersintermsoftheirstudents’fluency.Weaskedourselvesthequestion,‘whataretheKeyStage3conceptsthatformtheessentialprior-learningskills forthemostnumberoftopics inGCSEMathematics?’Orputanother way, ‘which topics are the biggest barriers-to-learning in GCSE Mathematics ifstudentslackfluencyinthem?’In 2014 we published an analysis of the GCSE Mathematics curriculum looking at therelationships between the syllabus learning objectives in terms of the prior learningdependencies (Emeny, 2014).We organised the syllabus into 164 differentmathematicalskills,representingeachoneasanode.Whereyouneededtobeabletoperformoneoftheskills inordertodoanother,wejoinedthemwithalink.Forexample,the‘findingfactors’nodewaslinkedtothe‘simplifyingfractions’nodebecauseyouneedtobeabletodotheformerinordertodothelatter.Weidentified935priorlearninglinksandusedtheonlinefreeware,Gephi,tovisualisetheresultingcurriculummap.Wescaledthenodesizebythenumber of out-going links; i.e. the largest nodes represent the concepts that were therequiredpriorlearningforthemostnumberoftopics.

5NumeracyNinjas

Figure 1- GCSE Mathematics Curriculum showing the prior-learning relationships between topics. Red nodes representnumbertopics.Purpletopicsrepresentshape,spaceandmeasuretopics.Turquoisenodesrepresentalgebratopics.Lightgreennodesrepresentdatahandlingandprobabilitytopics.

WehypothesisedthatifwecouldsupportourYear7and8studentsindevelopinghighlevelsoffluencywiththelargestnodaltopicsinthecurriculummap,thatinsubsequentstudyontheGCSEMathematicscourseinYears9,10and11,thesetopicswouldnotformworking-memory-consumingbarriers-to-learningthehigherlevelconceptsresultinginbetterlearningofthoseconcepts.Justasastudentwhohadtophoneticallysoundoutwordswouldunlikelybeabletoappreciatethe linguistic flairofShakespeare,weconsideredthatstudentswhohaveinefficientornostrategiesforelementarymentalcalculationmayhavelimitsimposedontheirappreciationandlearningofGCSEMathematicsconcepts.

6NumeracyNinjas

Inspiredbythisanalysis,andseeingtheconsistencyofprioritiesidentifiedbyitandthoseinBrunoReddy’smasterycurriculum,weredesignedourKeyStage3curriculumtoensurewetaughtthe‘largenode’topicsinincreaseddepth.

DiagnosticnumeracytestingTo understand extent of the need for this curriculum approach amongst our students atWyvern College and to be able to assess the impact of the changes we had made, wedevelopedourowndiagnosticnumeracyassessment.TheassessmenthasbeensatbyallYear7and8studentsatWyvernCollegeforthelasttwoyearsandmeasuresstudents’fluencywith25mentalnumeracystrategies,asampleof30ofthetimestablesupto10×10and29ofthelargestnodaltopics(whichwecalled‘keyskills’).The mental numeracy strategies were identified from the Department for Educationpublication,TeachingChildrentoCalculateMentally(DepartmentforEducation,2010)andincludedelementarynumeracyskillssuchas:numberbondsto10,addingneardoubleswithcompensating and understandingwhen it is beneficial to count from the smallest to thelargestnumberinasubtraction.Thetestisadministeredoverasinglelesson,takingapproximately50minutestocomplete.StudentswatchaseriesofquestionsonaPowerpointpresentationandhaveatimelimittochoose from5multiple-choiceanswers.Eachanswerhasanassociated letterA-Eandthestudents colour in the corresponding circle on aQuick Key3 optical reader answer sheet.Whilstthetimelimitsvarydependinguponthequestion,theyarerelativelyshorttoensurethatonlystudentswhocanidentifythecorrectanswersusinganefficientstrategywillbeabletorecordtheiranswerintime.Theintentionoftheassessmentisnotsimplytofindoutifstudentscananswerquestionsonthedifferentstrategies,buttofindoutiftheycandosoefficientlywithoutplacingsignificantsustainedloadontheirworkingmemoriesinordertoidentifythecorrectsolutions.

3QuickKeyisawebapplicationthatenablesteacherstoperformmultiple-choicequestioningefficiently.Studentsshadeintheiranswersontheopticalreadersheetandtheteacherthenscansthesheetviaasmartphoneappthatmarkstheassessment.

7NumeracyNinjas

Figure2-ExampleofaQuickKeyanswersheetusedbystudentswhensittingthediagnosticnumeracyassessment

Figure3-MentalnumeracystrategiesresultsfortheYear7cohortatWyvernCollegein2014-2015.ThepercentagesrepresenttheproportionofeachKeyStage2priorattainmentgroupthatgotthecorrespondingnumeracyskillcorrectontheassessment

8NumeracyNinjas

Theresultsfromthediagnostictestingshowedustheprevailingassessmentcultureatthetime,tosummarisestudents’proficiencybyasinglenationalcurriculumlevel,wasmisleading(Emeny,2015).Many level5and6studentshad level2numeracygaps. Instudentsbeingawardeda level5 itcouldnotbeassumedthattheyhadfluencywillall thebelow-level-5topics.Whilsttheirlevel2numeracygapshadnotlimitedtheminreachinglevel5upuntilthatpoint,wouldtheyinthefuturepreventorslowtheprogressthosestudentscouldmake?Ourstudents’lackoffluencywithsomeofthemostelementarynumeracyskillssurprisedus.For example, when given the question, 2 + 359, only 57% of our KS2 level 4.0 studentsansweredthequestioncorrectlyhavingbeengiven10secondstodoso.Reversinganadditiontomakeiteasier,suchabasicstrategy,wasnotafluentskillforasignificantnumberofourstudentsforwhomtheirKS2priorattainmentwouldnotbeconsidered‘low’.Weinferredthatthediagnosticnumeracyassessmentresultsshowedthatallstudents,notjusttheKeyStage2low-attaining,neededaregularactivityintheirKeyStage3mathematicslessonsthatwouldidentifyandrectifyanylackoffluencywithelementarynumeracyskills.

DesirabledifficultiesWeconductedaliteraturereviewtounderstandtheconceptoffluencybetter.Ifweweretocreatea fluency-building intervention,weneeded tounderstand thedefinitionof fluencywithinthecontextofmathematicseducationandthenstrategiesthatimpactonitindepth.However, before we read around the notion of fluency, we learned of the important,recurring theme that appears in educational research on learning; the much discusseddistinction between learning and performance. Whilst the jargon associated with theconceptshaschangedoverthepastcentury,theimportantideahasnot.Learningmusthaveelementsofretentionovertimeandcontextualtransferability,whereasperformanceisjusthowalearnercanperformataninstantaneouspointintime.Performancecanbemeasured,learningmustbeinferred.Performanceisoftenapoorproxyforlearning.Thingssuchasrecencyofretrieval,mimicryand priming can enhance performance, but we should not necessarily infer from thatperformancethattheknowledgewillbesustainedovertimeand/orthatthestudentwillbeabletoapplyittoothercontexts.Wecannotbesurethatlearningtookplace.The New Theory of Disuse (Bjork & Bjork, 1992) gave us a conceptual framework tounderstandhowtodesignlearningactivities(andsequencesofactivities)thatarefocussedon building retention and transferability of learning over time rather than short-termperformance.We learnedthat thenotionsofperformanceand learningcouldbemapped

9NumeracyNinjas

ontocorrespondingnotionsofretrievalandstoragestrengthrelatedtomemory.Ratherthanseeingforgettingasdetrimental,welearnedaboutitsimportanceinenhancinglearning.Bylimiting gains in retrieval strength by using desirable difficulties such as spacing andinterleaving (Rohrer & Taylor, The shuffling of mathematics practice problems improveslearning,2007),(Rohrer&Pashler,Increasingretentionwithoutincreasingstudytime,2007),(Rohrer,Theeffectsofspacingandmixingpracticeproblems,2009)ithasbeenshownthatyoucanenhancegainsinstoragestrength(learning).Wewereparticularlygratefulfor,andinspiredby,anoutstandingcontribution(Cepeda,Vul,Rohrer, Wixted, & Pashler, 2008) that suggested the optimum spacing interval isapproximately3weeks(ifthetestisarelativelylongtimeaway).We also learned about the benefits of the testing effect (J, Rawson, Marsh, Nathan, &Willingham,2013)andhowregularretrievalpracticecouldenhancelong-termretentionoflearning.Subsequently,wefeltinamoreinformedpositiontodefinefluencywithinthecontextwithwhich we were working as learning that is stored in long-term memory with both highretrievalandstoragestrength.Inthiscase,beingfluentwiththenumeracyandprior-learningskillsrequiredtostudyGCSEMathematicsmeantlearnersneededtobeabletoretrieveandprocesstherequirednecessaryinformationwithlowcognitiveloadandtobeabletoapplyittoabroadrangeofdifferentcontexts.Thiskindoflearningcanonlybeachievedovertimeandwerealisedastructured,regular,plannedapproachwasneededtogainthebenefitsofthedesirabledifficulties.We summarised our learning (Emeny, Forgetting is necessary for learning, desirabledifficultiesandtheneedtodissociate learningandperformance,2015)andthenbegantothinkabouthowwecouldincorporatetheseideasintoalearningactivitythatcouldusethemto tackle the numeracy needs our students had thatwere identified from the diagnostictesting.NumeracyNinjaswasborn.

10NumeracyNinjas

WhatistheNumeracyNinjasinitiative?NumeracyNinjasisrunbyteachers,typicallyinYear7mathematicslessons.EachstudenthastheirownNinjaSkillBookwhichfeaturesquizactivitiescontaining30questions.Thereare10questionsonmentalstrategies,10onthetimestablesand10onthekeyskills.

Figure4-ExampleofapagefromaNinjaSkillBook

Studentshave5minutestoworkonthequestions.AttheendofthesessiontheteachershowsaPowerpointpresentationcontainingtheanswersandthestudentsmarktheirwork.TheyrecordtheirNinjaScore(outof30).ThiscorrespondstoaNinjaBeltcolour.StudentsaspiretoachieveaNinjaScoreof30andacquirethecovetedBlackBelt.

11NumeracyNinjas

Figure5-Studentsaimtoachieve'BlackBeltStatus'withaNinjaScoreof30

Afterthestudentshavemarkedtheirworkitissuggestedthattheteacheraskstheclassiftheywouldliketodiscussanyofthequestiontheygotwrongorcouldn’tdo.Thisismeanttobea‘littleandoften’approach-gettingbetterquestion-by-question,day-by-day.AfullNinjassessionismeanttotakenomorethan10minutesatthestartofalesson.Thisispossibleifefficientroutinesareestablished.Thetopicsofthe10mentalstrategiesand10keyskillsquestionsrotateonaweeklybasis.Thereare30ofeachrespectivelywhichrepeatevery3weekstosupportthebuildingtheofretentionofourstudents’learning.Thus,weincorporatedintothedesignofNumeracyNinjasourlearningaboutthebenefitsofspacingandinterleaving.TheNinjaSkillBooks,ananswersPowerpointpresentationandotherpromotionalmaterialswerepublishedontheNumeracyNinjaswebsite,www.numeracyninjas.org,inAugust2015,freeforanyschooltodownloadandusefornon-profit-makingeducationalpurposes.Additionalpromotionalmaterialsuchaspencils,badges,stickersandrulersarealsoavailablefromanonlineshoponthewebsite.

12NumeracyNinjas

Impactstudydesign

CohortsOurcurrentYear8cohortcompletedthediagnostictestingwhentheywereinYear7andsoweused this cohort as a control groupasNumeracyNinjaswasn’t createduntil they thesummerbeforetheystartedYear8.WhilstinYear7,13ofthelowestattainingstudentstookpart in an intervention calledNumeracyAmbassadors. Thiswasa tutor timeactivity theyparticipatedin3timesaweekwheretheycompletednumeracy-focussedactivitiessupportedbyaYear10student(NumeracyAmbassador).ThewholecohortisreferredtoasC1NAfortheremainderofthisreport.Thesubsequentanalysisisbaseduponassessmentdataof233studentsinthiscohort.WhentheC1NAcohortreachedYear8theyallparticipatedinNumeracyNinjas.Everylessonstartedwith the 5-minuteNumeracyNinjas session in their Skill Books. By approximatelyChristmastime(week15)classeswhosemajorityofstudentswereregularlyattainingaNinjaScoreof30(fullmarks)droppedtousingNumeracyNinjastwiceperweekfortheremainderoftheyear.NumeracyAmbassadorsdidnotrunforthiscohortwhentheywereinYear8.Thiscohort(whentheywereinYear8)isreferredtoasC1NNfortheremainderofthisreport.Thesubsequentanalysisisbaseduponassessmentdataof248studentsinthiscohort.TheYear7cohortthatstartedatWyvernCollegeafterNumeracyNinjaswascreateduseditinitiallyinalllessons.AspertheC1NNcohort,highperformingclassesdroppedtousingittwiceperweekfromapproximatelyweek15onwards.ThiscohortalsoparticipatedintheNumeracyAmbassadorsinitiative.ThiscohortisreferredtoasC2NANNfortheremainderofthisreport.Thesubsequentanalysisisbaseduponassessmentdataof241studentsinthiscohort.

DatacollectionAllcohortstookthein-housediagnosticnumeracyassessmentatthestartandtowardstheendofeachacademicyear.Themonthsinwhichtheseweretakenarestatedonsubsequentfiguresinthisreport.TheNinjaScores(outof30)ofallstudentsinallclasseswasrecordedonaweeklybasisonasharedGoogleSpreadsheet.Theintentionofthiswastosupplyuswithacontinuousproxyfor the impact Numeracy Ninjas was having on students’ numeracy fluency as the yearprogressed and in between thepoints atwhich thediagnostic numeracy assessmentwastaken.ItalsoallowedustoidentifyourGrandMasterseachterm,thehighest-performingand

13NumeracyNinjas

most-improved student. The Grand Masters received certificates and prizes including aNumeracyNinjaspencil,blazerbadgeandgoldGrandMasterstickerfortheirexercisebook.

14NumeracyNinjas

Resultsanddiscussion

DiagnosticnumeracyassessmentresultsFigure6showsthemeanscoresfortheC1NA,C1NNandC2NANNcohortsonthediagnosticnumeracy assessment as a whole and then broken down into the mental strategies,timetablesandkeyskillssections.

Figure6-Meancohortscoresofdiagnosticnumeracyassessment.Errorbarsshowthe95%confidenceintervals

The cohortswhousedNumeracyNinjas, C1NN andC2NANN, outperformed the controlcohort, C1NA,whoonly usedNumeracyAmbassadors. TheNumeracyNinjas-led gains influencyweregreaterinallthreeareasofthediagnostictest:mentalstrategies,timestablesandkeyskills.ThefluencygainsweregreatestfortheC2NANNcohort.Thisisunderstandable,giventhattheC1NNcohortstartedfromapositionofhavingmadesomeprogressduringYear7theyearbefore.Significancetestingwascarriedoutusingtwo-tailedstudentttestswithα=0.05.HypothesesweretestedbasedoncomparingthemeanscoresoftheC2NANNcohortversustheC1NAcohort.Thefollowinghypotheseswerefoundtobestatisticallysignificant:

1. Numeracy Ninjas increased the average student’s result in themental strategiessectionofthediagnosticnumeracyassessmentbyatleastanadditional70%whencomparedtothegainsmadebytheC1NAcontrolcohort.

2. NumeracyNinjasincreasedtheaveragestudent’sresultinthekeyskillssectionofthediagnosticnumeracyassessmentbyatleastanadditional70%whencomparedtothegainsmadebytheC1NAcontrolcohort.

6.74.2 5.9

10.010.4 9.76.5

15.114.010.6 10.0

21.2

0

10

20

30

40

50

60

70

80

90

100

Wholediagnostictest Mentalstrategies Timestables Keyskills

Cohortm

eanpercentagesc

ore

Meancohortscoresonthediagnosticnumeracyassessment

C1NASep2014C1NAJun2015C1NAChangeC1NNJun2015C1NNMay2016C1NNChangeC2NANNSep2015C2NANNMay2016C2NANNChange

15NumeracyNinjas

ThegainstheC1NNcohortmadewiththeirtimestableswerenotfoundtobestatisticallysignificant.WebelievethiswasduetoalargeproportionoftheKS2level5and6students‘toppingout’onthetimestablessectionafteralreadyinastrongpositioninJune2015.Therewasonlyasmallamountofprogresstobemadebeforegainingfullmarksonthetimestablessection.ThediagnosticnumeracyassessmentresultswerethenviewedintermsofdifferingKS2priorattainmentlevelgroupstounderstandtherelativegainseachgroupmadeusingNumeracyNinjas.

Figure 7- Key Stage 2 level 2 students'mean scores on the diagnostic numeracy assessment. Error bars show the 95%confidenceintervals

Figure7showsthefluencygainsmadebytheKS2level2studentsintheC2NANNcohort.Statisticalsignificancecannotbetestedastherewerenolevel2studentsintheC1NAcohort.Whilstconclusionsshouldbeapproachedcautiouslywithsuchasmallsamplesizeandspreadintheresults,itwasdisappointingthattheapparentfluencygainsinmentalstrategiesandtimestables were not matched by similar gains in the key skills.We cautiously pose thefollowingquestionsforfuturediscussionandresearch:

1. DotheseKS2level2studentshaveotherlearningdifficultiesinacquiringfluencywiththe key skills that are unaffected by improvements in fluency with the mentalstrategiesandtimestables?

2. TowhatdegreearetheimprovementstheseKS2level2studentshavemadebasedonunderstandingratherthanfactmemorisation?

11.414.2

19.2

0.80.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

Wholediagnostictest

Mentalstrategies Timestables Keyskills

Meanpercentagescore

KS2level2students'scoresonthediagnosticnumeracyassessment

C2NANNSep2015C2NANNMay2016C2NANNChange

16NumeracyNinjas

3. Dotheynotreachthekeyskillsquestionsduringa5-minuteNinjasessionbecausetheyarestillworkingontheirfluencywiththeothertwoareas.I.e.aretheyrunningoutoftimebeforereachingthekeyskillssection?

Figure8showsthefluencygainsmadebytheKS2level3studentsintheC1NAandC2NANNcohorts.

Figure 8- Key Stage 2 level 3 students'mean scores on the diagnostic numeracy assessment. Error bars show the 95%confidenceintervals

Due to the relatively small sample sizesof13and20 in theC2NANNandC1NAcohortsrespectively, it was not possible to show that the additional fluency gains made by thestudentswhousedNumeracyNinjas,asshowninFigure8,werestatisticallysignificant.ThegainsinkeyskillsbytheC2NANNcohortwereclosetopassingthesignificancetest(p=0.07).ThepvaluesfortheadditionalgainsmadebytheC2NANNcohortinthementalstrategiesandtimestablessectionswere0.94and0.33respectively.WhilstthereisnotsufficientevidenceyettoverifythestatisticalsignificanceoftheimpactNumeracyNinjashashadonourKS2level3students, it iscleartheresultsaresuggestingtherelikelyisapositiveadditionalimpactandmoredatashouldbecollectedinthefuturetosupportfurtherstatisticalsignificancetesting.Figure9showsthefluencygainsmadebytheKS2level4studentsintheC1NAandC2NANNcohorts.

0.02.2 2.0

-7.6

16.6

11.0

18.7 20.0

-10.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Wholediagnostictest Mentalstrategies Timestables Keyskills

Meanpercentagescore

KS2level3students'scoresonthediagnosticnumeracyassessment

C1NASep2014C1NAJun2015C1NAChangeC2NANNSep2015C2NANNMay2016C2NANNChange

17NumeracyNinjas

Figure 9- Key Stage 2 level 4 students'mean scores on the diagnostic numeracy assessment. Error bars show the 95%confidenceintervals

KS2level4studentswerethepriorattainmentgroupwhogainedthemostfromNumeracyNinjas.IncomparisonwiththefluencygainsmadebytheKS2level4C1NAcontrolcohort,thegainsoftheKS2level4studentsintheC2NANNcohortinmentalstrategies,timestablesandkeyskillswereallstatisticallysignificantwiththefollowinghypothesespassingatwo-tailedstudentttestwithα=0.05:

1. NumeracyNinjas increased the average KS2 level 4 student’s result in thementalstrategiessectionofthediagnosticnumeracyassessmentbyat leastanadditional69%whencomparedtothegainsmadebytheC1NAcontrolcohort.

2. NumeracyNinjasincreasedtheaverageKS2level4student’sresultinthetimestablessectionofthediagnosticnumeracyassessmentbyatleastanadditional470%whencomparedtothegainsmadebytheC1NAcontrolcohort.

3. NumeracyNinjasincreasedtheaverageKS2level4student’sresultinthekeyskillssectionofthediagnosticnumeracyassessmentbyatleastanadditional140%whencomparedtothegainsmadebytheC1NAcontrolcohort.

CriticstocomparingtheKS2level4students’resultsinprogresstermswouldarguethatthelevel4studentsintheC2NANNcohortwereweakerthanthelevel4studentsintheC1NAcohortinallthreeareasofthediagnosticnumeracyassessmentatthestartoftheacademicyear.Therefore, itcouldbesuggestedthat iswasunfairtoconsidertheproportionoftheprogresstheC2NANNlevel4studentsmadeonthediagnosticnumeracyassessmentovertheyearthatraisedthemuptotheproficiencythattheC1NAlevel4studentshadatthestart of the year. I.e. perhaps this proportion of the progress that the C2 NANN level 4studentsmadewouldhavebeenmadewithoutNumeracyNinjas?Whilstwebelievethistobe unlikely, if this line of argument is followed itwould therefore also be appropriate to

2.7 3.00.8

4.5

14.611.4 11.8

20.5

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Wholediagnostictest Mentalstrategies Timestables Keyskills

Meanpercentagescore

KS2level4students'scoresonthediagnosticnumeracyassessment

C1NASep2014C1NAJun2015C1NAChangeC2NANNSep2015C2NANNMay2016C2NANNChange

18NumeracyNinjas

comparetheresultsoftheKS2level4studentsalsoinabsoluteattainmentterms,ratherthanjustbyprogress.By inspectingFigure9-KeyStage2 level4students'meanscoresonthediagnosticnumeracyassessment.Errorbarsshowthe95%confidenceintervalsitisclearthatdespite the C2 NANN level 4 students starting from a weaker position, they ended theacademic year with a higher attainment in all three areas of the diagnostic numeracyassessmentthantheC1NAlevel4students.Evenifthecautiousviewaboutdifferencesinstartingpointsistaken,itwasfoundthattheoverall attainment of the KS2 level 4 students in the C2 NANN cohort was statisticallysignificantlyhigherthantheKS2level4studentsintheC1NAcohortinallthreeareasofthediagnosticnumeracyassessmentattheendoftheacademicyear.These results strongly suggest that NumeracyNinjas can have a statistically significant,transformativeimpactonthenumeracyfluency(andthefluencyofpre-GCSEMathematicsskills)ofKS2level4students.

Figure10showsthefluencygainsmadebytheKS2level5studentsintheC1NAandC2NANNcohorts.

Figure 10- Key Stage 2 level 5 students'mean scores on the diagnostic numeracy assessment. Error bars show the 95%confidenceintervals

Fluency gains by the KS2 level 5 students using Numeracy Ninjas were not statisticallysignificantlydifferent to theKS2 level5 students in theC1NAcontrol cohort.Webelievetheseresultsmaysuggest:

12.1

6.310.7

18.9

13.19.5

6.7

23.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Wholediagnostictest Mentalstrategies Timestables Keyskills

Meanpercentagescore

KS2level5students'scoresonthediagnosticnumeracyassessment

C1NASep2014C1NAJun2015C1NAChangeC2NANNSep2015C2NANNMay2016C2NANNChange

19NumeracyNinjas

1. There couldbediminishing returns gainedbyKS2 level 5 students in termsof themental strategies and timestables; they are already reasonably proficient in theseareasandthereforehavelesstogain.

2. IfNumeracyNinjas isnotused,thesestudentsappeartomakestrongergainsthanKS2level4(andbelow)studentsintermsoftheirmentalstrategies,timestablesandkeyskills.ThesestudentsseemnottorequireNumeracyNinjastosupportthefluencygains to suchanextent asKS2 level 4 students; theymayacquire similar gainsbyfollowing conventional lessons without the need for the additional retrievalopportunitiesgivenbyNumeracyNinjas.

AstheKS2level5studentsintheC2NANNcohortlikelyperformedbetterthantheC1NAlevel 5 students at the start of the academic year (see the 95% confidence intervals) thepreviousargumenttocompareattainmentratherthanprogress(asperwiththeKS2level4students)wouldnotbe valid and sowasnot analysed.A criticwould argue, they startedhighersotheyendedhigher.Nonetheless,theC2NANNlevel5studentsdidendtheyearwithhigherattainment inall threeareas. FurtherexperimentsareneededbeforewecanreasonablyclaimNumeracyNinjas leadstohigherattainmentonthediagnosticnumeracyassessmentforKS2level5students.However, we believe that KS2 level 5 students may still benefit from participating inNumeracyNinjas;theresultsareinconclusive,butthequestionremainsopen.Intermsoftheimpacton theiraverage fluencygainsacrossmental strategies, timestablesandkey skills,these may be modest, but an occasional use of Numeracy Ninjas could support theidentificationandsubsequentclosingofanyspecificbasicnumeracyskillgapsthatKS2level5studentshave.Nonetheless,webelievethefrequencywithwhichNumeracyNinjasisusedwith these students should be considerably much less than the KS2 level 4 (and below)students.Figure11showsthefluencygainsmadebytheKS2level6studentsintheC1NAcohort.

20NumeracyNinjas

Figure 11- Key Stage 2 level 6 students'mean scores on the diagnostic numeracy assessment. Error bars show the 95%confidenceintervals

TherewerenostudentsintheC2NANNcohortidentifiedasKS2level6andthereforetheimpactofNumeracyNinjasonthesestudentscannotbeassessed.However,itisclearfromtheC1NAcontrolcohort’sresultsshowninFigure11thattheconclusionsfortheKS2level5studentsarealsoreasonableforKS2level6students;thereisonlyasmallmarginofpossibleimprovementinthementalstrategiesandtimestableswhichtheyaremakingfromfollowingtheir normal curriculum. They are alsomaking substantial fluency gains on the key skillswithoutusingNumeracyNinjas.

NinjascoresovertimeFigure12andFigure13showtheweeklymeanNinjaScorebyKS2priorattainmentgroupfortheC2NANNandC1NAcohortsrespectively.

12.3

4.8

14.317.6

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Wholediagnostictest

Mentalstrategies Timestables Keyskills

Meanpercentagescore

KS2level6students'scoresonthediagnosticnumeracyassessment

C1NASep2014C1NAJun2015C1NAChange

21NumeracyNinjas

Figure12-C2NANNcohortweeklymeanNinjaScorebyKeyStage2priorattainmentgroup

Figure13-C1NAcohortweeklymeanNinjaScorebyKS2priorattainmentgroup

The Ninja Scores in both cohorts improved across all prior attainment groups up untilapproximately week 11 when they start to plateau, broadly remaining constant for theremainderoftheyear.However,inbothcohortstheNinjaScoresoftheKS2level3students(andtheKS2level4studentsintheC2NANNcohort)riseagainaftertheinitialplateau.Thissuggeststhattheplateauwasnotcausedbyadefinitivelearningblock;thesestudentsstillhadthecapacitytoimprove.TheplateauandsubsequentimprovementraisesanumberofinterestingquestionsaboutthemosteffectivewaytorunNumeracyNinjasintheclassroom.Whathappenedaroundweek11(Christmastime)thatledtoaplateauinimprovementforallpriorattainmentgroups?Wehypothesise thatapossibleexplanationcouldbe that theavailablebenefitsof the testing

10

12

14

16

18

20

22

24

26

28

30

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

MeanNinjaScore

Week

C2NANNcohortweeklymeanNinjaScorebyKeyStage2priorattainmentgroup

Wholecohort KS2Level2 KS2Level3 KS2Level4 KS2Level5

10

12

14

16

18

20

22

24

26

28

30

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

MeanNinjaScore

Week

C1NNcohortweeklymeanNinjaScorebyKeyStage2priorattainmentgroup

Wholecohort KS2Level3 KS2Level4 KS2Level5 KS2Level6

22NumeracyNinjas

effect had been reaped and that with the remaining topics that students were gettingincorrect,thestudentswerenotnotgoingto improvejustbyrepeatedtestingandseeingcorrect answers; they needed teacher input to learn how to correctly answer theiroutstandingtopics.Someteachersdidcometothisconclusionthemselvesandtookactionwiththeirclassestoidentifywhichtopicsstudentscontinuallystruggledwithandactedtotalkstudentsthroughexamples.However,thiswasnotconsistentacrossthedepartment’spractice.Itcould,nonetheless,suggestapossibleexplanationforthesubsequentprogressoftheaforementionedpriorattainmentgroupsaftertheirinitialplateau.WebelievethatmonitoringtheC2NANNandC1NAcohorts’NinjaScoreshasbeenvaluableasitsuggestsaformatforimprovingourimplementationofNumeracyNinjasinsubsequentyears.Afteraninitialapproximate10-weekperiod,thefocusshouldswitchtodiagnosticusetoidentifytopicsthatremainoutstandingandteachingtailoredtospecificallyaddressthese.

23NumeracyNinjas

ConcludingremarksWebelievethattakingsucharigorousapproachtoanalysingtheimpactNumeracyNinjashashadonourKeyStage3students’numeracyfluencyatWyvernCollegehasgivenusconfidencethat it having a highly beneficial effect on the mathematics progress of our KS2 level 4students(andmostlikelyourKS2level3studentstoo).MoredatacollectionisneededbeforewecansaywithconfidencethatitimpactspositivelyforourKS2level2students.WewillalsoadjusttheimplementationofNumeracyNinjasnextyearinanumberofways:

1. AllKS2level4(andbelow)studentswilluseNumeracyNinjaseverylessonuntiltheyregularlyachieveNinjaScoresof30.

2. In approximately week 11 we will ask teachers to consciously focus on using itdiagnosticallytoidentifyoutstandinggapsstudentshaveandtoteachthemhowtoapproach thesequestions.As ever, a tool is only ever effective if there is ahighlyskilledcraftspersonusingit…Inthiscontextanexpertmathematicsteacherusinghighqualityassessmentforlearning.

3. Level5and6studentsmayuseNumeracyNinjasonaweeklybasisfromthestartoftheyearwithteachersbeingproactiveaboutdiagnosingandfixinganynumeracygapsfromdayone.OncetheyaresatisfiedthesehavebeensecuredtheywillstopusingNumeracyNinjasanduseotherspaced-learningbasedstarterswithahigherpitch.WebelievethattheplateauintheNinjaScores,ifitwasattributabletoreachingalimitwiththetestingeffectbystudentsnotbeingableto infercorrectapproachesfromanswers without explicit instruction, may suggest that any higher-pitch spaced-learningactivitiesusedsubsequentlyshouldbebasedontopicspreviouslystudiedinclass,ratherthanunstudiedtopics.

WhilsttheseconclusionshavebeenreachedbasedonourownexperiencesandevaluationofrunningNumeracyNinjasinourowndepartmentthisyear,wearefullyawarethatduetotheflexiblenatureofNumeracyNinjas,thatotherdepartmentsmayhavehadsuccessbyusingitinotherformats.Wefullyencouragethesharingofyourexperiences,preferablybackedupbydatasuchthatyouhaveconfidenceinanysuggestionsyouputforward.IftheNumeracyNinjascommunitycanlearnfromeachotherallourstudentswillbebetteroffforit.Asever,thejourneyisjustbeginningandweputforwardthefollowingsuggestions,questionsandchallengesthattheNumeracyNinjascommunitymayliketoaddresscollaborativelyinthefuture:

24NumeracyNinjas

1. What if thediagnostic testingwereadaptedso that itwas time-based rather thanbinary‘correct/incorrect’?Ifyoucreatedadiagnostictestthatrecordedthetimeittook the student to answer then compare it against whole cohort, national orinternationalaveragetimesmightitgiveusabetterinsight?Wouldweinterprettheevaluationdatadifferently?

2. Woulddevelopingsomeinstructionalresources,forexampleaseriesofvideolessons,that students could access to learn how to approach questions they have not yetmasteredonNumeracyNinjasavoidanyplateauinNinjaScores(before30!)?

3. Couldstudentscollaboratewithclasspeerstolearnhowtoapproachquestionstheyhavenotyetmastered?Couldyouidentifyanduse‘Ninjaexperts’tohelpyousupportotherstudents?

4. IsitpossibletodesignarobuststudytoassesstheimpactthatNumeracyNinjashason students’ GCSE Mathematics outcomes? Are there too many confoundingvariables?WhataboutlookingattheimpactonprogressandattainmentinjustYear7or8?

5. Howcouldwemeasurethe‘transferability’oftheimprovementinskillswemeasuredinthediagnosticnumeracyassessment?Canstudentsapplytheirfluencywiththeseskillsindifferentcontexts?Howcanwetell?

6. Could you collect enough reliable data to test if Numeracy Ninjas’ impact on thenumeracyfluencyofKS2level2studentsisstatisticallysignificant?

25NumeracyNinjas

BibliographyBjork,R.A.,&Bjork,E.L.(1992).ANewTheoryofDisuseandanOldTheoryofStimulus

Fluctuation.InA.Healy,&R.Shiffrin,LearningProcessestoCognitiveProcesses:EssaysinHonorofWilliamK.Estes(Vol.2,pp.35-67).Hillsday,NJ.

Cepeda,N.J.,Vul,E.,Rohrer,D.,Wixted,J.T.,&Pashler,H.(2008).Spacingeffectsinlearning:Atemporalridgelineofoptimalretention.PsychologicalScience(11),1095-1102.

DepartmentforEducation.(2010).TheNationalStrategies-Primary-TeachingChildrentoCalculateMentally.DepartmentforEducation.

Emeny,W.(2014,January5th).You'veneverseentheGCSEMathscurriculumlikethisbefore.RetrievedfromGreatMathsTeachingIdeas:http://www.greatmathsteachingideas.com/2014/01/05/youve-never-seen-the-gcse-maths-curriculum-like-this-before/

Emeny,W.(2015,January20).Forgettingisnecessaryforlearning,desirabledifficultiesandtheneedtodissociatelearningandperformance.RetrievedMay29,2016,fromGreatMathsTeachingIdeas:http://www.greatmathsteachingideas.com/2015/01/20/forgetting-is-necessary-for-learning-desirable-difficulties-and-the-need-to-dissociate-learning-and-performance/

Emeny,W.(2015,July3).Theproblemwithlevels-gapsinbasicnumeracyskillsidentifiedbyrigorousdiagnostictesting.RetrievedfromGreatMathsTeachingIdeas:http://www.greatmathsteachingideas.com/2015/07/03/the-problem-with-levels/

J,D.,Rawson,K.A.,Marsh,E.J.,Nathan,M.J.,&Willingham,D.T.(2013).Improvingstudents'learningwitheffectivelearningtechniques:Promisingdirectionsfromcognitiveandeducationalpsychology.PsychologicalScienceinthePublicInterest,4-58.

Rohrer,D.(2009).Theeffectsofspacingandmixingpracticeproblems.JournalforResearchinMathematicsEducation(40),4-17.

Rohrer,D.,&Pashler,H.(2007).Increasingretentionwithoutincreasingstudytime.CurrentDirectionsinPsychologicalScience(16),183-186.

Rohrer,D.,&Taylor,K.(2007).Theshufflingofmathematicspracticeproblemsimproveslearning.InstructionalScience(35),481-498.

Sweller,J.(1988,Jun).Cognitiveloadduringproblemsolving:Effectsonlearning.CognitiveScience,257-285.

Willingham,D.T.(2009).Whydon'tstudentslikeschool?:Acognitivescientistanswersquestionsabouthowthemindworksandwhatitmeansfortheclassroom.SanFrancisco:Wiley,John&Sons,Incorporated.