Test Planning and Execution in a Mobile Game Development Project using SCRUM

The Magazine for Agile Developers and Agile Testers

© iStockphoto.com/GarySandyWales

April 2011

issue 6www.agilerecord.com freedigitalversion madeinGermany ISSN2191-1320

26 www.agilerecord.com

Theinformationtechnologyindustryincreasinglyrealizestheim-portanceofconducting,inacarefulandefficientmanner,verifi-cationandvalidationactivities,whichincludessoftwaretesting.Herethetestingindividuals,alongwiththerestoftheteam,worktoassurethatthedevelopedsoftwaremeetsallclients’needsandisofahighqualitystandard.Toachievethisgoal,aneffec-tivetestplanisindispensable.Fromthebeginningoftheproject,atestengineershouldbepresent,becausethisallowsustoplanaheadandtofindandfixdefectsassoonaspossibleinthede-velopmentlifecycle.Afterall,asmentionedintestingliterature,thesoonerdefectsare found, the lowerwillbe thecosts tofixthemandthehigheristheprobabilitythattheircorrectionwon’tcausenewbugs(GlenfordJ.Myers,“Theartofsoftwaretesting”).

Thisarticle,describeshowtestingactivitieswereperformed inamobilegamedevelopmentprojectusingSCRUMastheman-agementprocess.Itwilldescribeindetailthetestingstrategiesused, alongwith the best practices identified and the learnedlessons.Themaingoalof thearticle is toassistother testen-gineerswhoarestartingingamedevelopmentprojects,sothattheycaneasilyand rapidlyadapt to thedifferencescomparedtostandardsoftwaredevelopmentprojects.Thiswillalsoallowthemtocontributetothecreationofnewandevenmoreeffec-tivetestingtechniques.

The projectAllthetechniquesandlearnedlessonsdescribedinthisarticlewereexperiencedduringaprojectdevelopedatC.E.S.A.R. (Re-cife’sCenterofAdvancedStudiesandSystems)fromDecember2007toMarch2008.

Sincetheprojectincluded,amongstothers,elementslikeshortduration, a small team, frequent client involvement, and con-stantrequirementchanges,itwasdecidedtoapplySCRUMandanagiledevelopmentmethodologytohelpmanageallactivitiesduringtheproject.InaccordancewithSCRUM,alltaskstobede-velopedwerelistedasbacklogitems(BLI).Thesewereelectedto

bedevelopedinshortdevelopmentcycles(“sprints”),wherebyattheendofeachsprintanewversionoftheproductwasreleasedtotheclient.Inourproject,eachsprintlasted10workdays,andtheitemstobedevelopedwerechosenbytheteamduringsprintplanningmeetings.

Duringthesprintplanningmeeting,allteammembershadtopri-oritizeallBLIs,inordertohelpdecidewhichtaskshadtobede-velopedduringthefollowingsprint.Fromatestengineer’spointofview,theapproachwastoalwaystrytoanticipatethefeaturesthatappearedtobecriticalforsystembehaviorandformeetingtheclient’sneeds.Prioritizationcouldbeassignedduetocom-plexityorimportance;theintentionwastopreventbugsrelatingtothesefeaturesfrombeingfoundlateintheprojectlifecycle.

Testing strategyPlanning – First sprintTesting activities started before the end of the project’s firstsprintwiththearrivalofatestengineer.Asafirsttask,acom-plete analysis of the Basic GameDesign Specification (BGDS)wasmade.Thisdocumentsummarizesallbasicgamefeatures.AfterevaluatingtheBGDSandallclient’srequestsandneeds,asimpleteststrategywasdefinedwhichinvolveddocumentingmanualtestcasesassoonasthesprintstartedusingasimplepriorityschemebasedonthecomplexityoftheselectedstoryandtheimplementationorder.Atthisinitialstagewealsoidentifiedandsolvedalltestenvironmentneeds,likeavailablehardware,SIMcards,bug trackingsoftware,etc. Finally, a setof generaltestcasesmadeavailablebytheclientwerealsoevaluatedpriortostartingtestexecution.

Test case designOneoftheinitiallydefinedconstraintstotheprojectwasthatalldocumentedBLIsshouldbecoveredbytestcases,andtheBGDSwereconsideredasthetestoracle.Basedonprojectcharacteris-ticslikeshortduration,scarceBLIdocumentation,andfrequentchanges,wedecidedtodesigntestcasesinamoregeneralway

© iStockphoto.com

/uriy2007

Test Planning and Execution in a Mo-bile Game Development Project using SCRUMby José Carréra Alvares Neto

27www.agilerecord.com

withafocusontestingthegame’sbasicfunctionalitiesforeachBLI.Thisresultedinasetoftestcasessimilartothoseusedfor“sanity”tests.Thiswayweexpectedtomaximizethetimespentontestexecutionandtoavoidspendingexcessivetimeondocu-mentation.

Test executionThe testspecificationconsistedofaspreadsheetwithasetoftestcasessentbythecustomerandagroupofspecificscenar-iosdesignedspecifically for thegamebythetestengineers.,AMANTISbugtracker(http://www.mantisbt.org/)wasusedasthedefectmanagementtool.

Duringtheexecutionphase,thefollowingactivitieswereplannedtobeexecutedineachsprint:Themainfocuswastheexecutionoftheexploratorytestsasfeatureswerereleasedthroughoutthesprint,alongwiththeexecutionoftheclientsetoftestcasesandthegame’sspecificgroupoftestcases.Theuseofexploratorytestingisgenerallyencouragedforprojectswithcharacteristicssimilar to ours, and when executed by experienced test engi-neers,such”exploratory testscanbemuchmoreefficient thanthetestsperformedfollowingscriptedtestcases”(JamesBach).

Duringthecourseofeachsprint,an importanttaskperformedby the testengineerwas toeffectivelymonitor theprogressofalldevelopers’activitiesonthecurrentsprint.Thiswasdoneinordertodefinethebesttimetorequestthereleaseofintermedi-ateversionsofthegameforcomponenttestingandalsotoavoiddefectsorchangerequestsbeingraisedforfeaturesthathadnotbeenfullyreleasedbythedevelopmentteam.Thismonitoringofactivitieswasmadeeasierby theSCRUMmethodologywhere,duringthedailymeetings,wecouldeasilyfollowprojectactivitiesthroughtheburndowngraph.

Aspreviouslymentioned, thedecision toprioritize theexplora-tory testswasmadedue to theproject’smain characteristics,suchasalackofaextensivedocumentationatthebeginningoftheproject,andfrequentchangesofclient’sneedsandrequire-ments.

Formal testsThecompletesetoftestcasesconsistedoftheclient’sstandardtestcasestogetherwithgame-specifictestcaseswhichcametoatotalofaround15Otests.However,notallspecifictestswereexecuted in the initial sprints, sincemostof the featureswerenot yetdeveloped.A complete test execution cyclewould takeanaverageofthreedays.Duringtherestofthesprintothertest-ingactivitiesliketestdesignandmaintenancewereperformedalongwithexploratorytestingandchangerequestvalidation.

The client’s set of test casesmainly focused on assuring thatthecompany’sstandardswerebeingfollowedbyourteam.Thisconcernedfeatures likekeymapping,performance, interactionanduserinterface.Takingthisintoconsideration,itwasmanda-torytorunthesetestsforeachsprintinordertoassurethatthedevelopedgamesuitedallclients’demandsand,aboveall,thattheywouldn’tinterferewiththemobilephone’sbasicfeatures.

At theendofeachsprintan intermediateversionof thegamewasreleasedto theclient,whocouldanalyze thedeliveryandprovidefeedbacktoourteam.Thisusuallyinvolvedaspectslikegameplay,gamedesignanddefects.Throughananalysisofthisfeedbackwecouldfigureoutwhichareasweremorerelevanttoourclient,adjustourteststrategyaccordingly,andthenfocusonthemisseddefectsinthenextsprint.

Exploratory testsExploratory tests, which were chosen as ourmain test execu-tionstrategyduetotheprojectprofile,beganearlyintheinitialsprints.Inatraditionalapproach,informaltestcharterswerepre-paredfocusingonaspecificareaorBLItobetested.Duringthecourseoftheproject,astheGameDesignSpecificationbecamemoremature,westartedusingtwonewapproachesforrunningtheexploratorytests,whichwillbedescribedbelow.

Test case basedIn thisapproach theactual testcaseswereusedas the focusareaforeachexploratorytest,wherebythestepsofthetestcasewereruninanunusualway.Thetesterisencouragedtodivergeasmuchaspossiblefromthespecifiedteststeps,andtotryandthinkofalternativepathsthatcouldbetakeninsteadoftheonesuggestedbythetestcase.Themainideaistousetheexistingtestcasesjustasareferenceinordertocoverallthefeaturesof theapplicationandto leave it to the testengineer toevalu-ate relevanceof the features to the system.By doing this,weencouragethetestengineertothinkcreativelytofindnewways,orwaysnotpreviouslyforeseen,tobreakthesoftware.Thisap-proachachievedverygood resultsandcertainly increased thetotalnumberofrelevantdefectsfound.

Ifthisapproachistoachieveahighdegreeofsuccess,itisveryimportant for the testengineer toknowallexisting featuresofthesystem,themarketandtheclient’sexpectations.Heneedstofullyconcentrateonhisworkinordertonoticedetailsthatmayhaveescapedbefore.Itisalsoimportantthattheengineercanworkinacomfortableenvironmentduringtestexecutionwithoutbeingconstantly interrupted,thusallowinganefficientanalysisoftheexistingtestcasesandunexploredpossibilities.

GDS scanningA different approach, which we used in the later sprints, wasbasedontheexecutionoftheexploratoryteststhroughacom-pletescanoftheGDS.Thisapproachcouldn’tbeappliedfullyintheinitialsprints,becauseonlytheBGDSwasavailable,whichdidn’tcontainenough information toallowamoredetailedex-ecution.

Forthistechniquetobeappliedsuccessfully,thetestengineerneedstohavealreadyreadandunderstoodthedocument,,andthere shouldbenoopenquestions. Themain ideaof this ap-proach is tomake sure that every description included in theGDS iscorrectlycoded in thegame.Bysimultaneouslyanalyz-ingthedocumentandexploringthegame,itbecomesvisibleifanyimportantscenarioisn’twelldescribedinthetext.Withthisapproachwecancombinethebenefitsofstatictestingofdocu-


mentationwiththeadvantagesofexploratorytestingoftheap-plication.Anygapsbetweengameanddocumentationcaneasilybefoundandthegamedesignercanclarifythetypeofanybugsfound.

Registered defectsDuring the course of the project, 122 change requests (CRs)wereregistered,whichweresimplyclassifiedforthisarticleintofourcategories,accordingtothetypeofdefect,asfollows:func-tional, user interface, art, and sound effects. Later on in thisarticle isdescribedhoweachoftheseareaspresenteduniqueimportantcharacteristics.Inadditiontothesedescriptions,twographswillindicatetheamountofCRspertypeandtheseverityoftheregistereddefects.

Theseverityofeachbugwasclassifiedasfollows:(i)minor(fordefectsthatdonotblockthegame’scorrectbehavior,e.g.,thephonedoesn’tvibratewhenanewlevelisunlocked);(ii)normal(for defects that affect important elements of the game, butdo not block the game’s execution, e.g., a special effect lastsshorterthanthevaluespecifiedintheGDS);(iii)major(forde-fectsthatdirectlyaffectgameplayorusersatisfaction,thathaveadirectimpactonthegame’sleveldesign,andthatprohibittheplayerfromproceeding,e.g.,scenarioswherethegamefreezes);(iv)critical(fordefectssimilartomajordefects,butwithanevenhigherimpactonthegame’scorrectoperation,e.g.,levelisnotunlockedaftercompletingtasksorphonedoesn’treceivecallswhilegameisstarted).

FunctionalTheCRsfromthisgrouparerelatedtoinconsistenciesregardingthegame’sdesigned rulesand logic.This typeofdefect,eventhosewith lower severity, have a direct impact on the game’ssuccess, because theyusually get in theway of a smoothun-derstandingofthegame’sobjectives.Theymayevenblocktheplayer from overcoming the challenges presented, turning thegameintoanimpossiblemission.

Scenarios like score limits, lousy player and excellent playerapproach, and other possible features that involve testing thegame’sfeaturesandlimits,wereteststhatusuallydetectedthiskindofdefect.ThesescenarioswerenotalwayswelldescribedintheGDS,anddevelopersdidn’ttakethemintoconsiderationorunittestthemproperly.

User interfaceInterface defects were connected to failures during display oftexts, openingof pop-upwindows, screen limits, etc. These is-suescouldbeeasilynoticedbyanyplayer,andwoulddefinitelygivetheideaofapoorlydevelopedgame,withoutcarefordetails.

Thesedefects,althougheasilydetected, frequentlyescape thedevelopingteamandeventheinitialtestcycles.Thiscanhappenbecausedevelopersusuallyruntestsusingasimulatorandnotarealdevice.Itispartofthetestengineer’sjobtoassurethatallgamescreensandtextsarecheckedforthesupporteddevices.

ArtAllchangerequestsofthe“art”categoryarerelatedtofeatureslikeimagerendering,lightening,andanyotheraspectsrelatedtotheelementsproducedbytheartteam(althoughsomeofthemmayalsobeperformedbythedevelopmentteam).

Thistypeofdefectvarieswidelyfromhugeperceptiblefailures,whichcanbeeasilynoticed,tospecificscenariosthatarecausedonlybyadefinedsequenceofactions.Thiskindofdefectcanbefoundnotonlybyfocusingonthisaspectduringtesting,butalsowhilerunninganyothertypeoftest.Allthatisneededisahighlyalerttesterwhopaysattentiontodetails.

Itishighlyimportantforthetestengineer,especiallyifheisnotexperiencedwiththiskindofdefect,tointeractwiththeartteamtoclarifythequestionsrelatedtopossibledefects,andindoingsobeginning tounderstand the features, their limitsand theirsolutions.

Sound effectsWealsoassigneddefects toa soundeffect category,becauseit turnedouttobeakeyareawhere initiallywedidnotexpecttofindarelevantamountofbugs.However,testingshowedthatthis assumptionwaswrong.Several defectswere foundwhichdemandedconsiderableworkfromthedevelopmentteamtogetthemfixed.Oneaspectobserved,theconcurrenteventsexecu-tion,causedcomplicationsinthegame’sgeneralbehavior.Thisconcernsscenariossuchasexecutingasoundwhileanotheroneisalreadyplaying,user-initiatedpausesof thegame,disablingandenablingthesound. Inaddition,severeperformanceprob-lemscouldresultfromsomeofthegame’ssoundevents.

Throughout the course of the project, this kind of the defect,whichinitiallywasunderestimated,gainedhigherpriorityandat-tention.Wefoundthatinthisareawehadahigherprobabilityofcausingotherdefectswhilefixingone.

Atfirst,testexecutionforthisaspectwasimpactedbythequalityoftheavailablehardware,whichpresentedabadsoundquality.Lateron,withthearrivalofnewhardware,testscouldbeeasilyexecutedandshowedbetterresults.Thereforeitisimportantforthetestingteamtoensuretheavailabilityofthecorrecthardwareatthebeginningoftheproject.

Figure 1 – Amount of defects registered by type

29www.agilerecord.com

Best practicesBelowwedescribesomeofthepracticesthatwereappliedinourprojectandthatpresentedgoodresults.Thesecouldthereforeevenbeappliedtoprojectsindifferentareas.

BLI defect trackingEvery time a new change request was submitted during thecourseoftheproject,alongwithitsshortdescription,atagwasaddedtoidentifytowhichBLIitwasrelated.Atthestart,thiswasonlyusedtohelptheConfigurationControlBoard(CCB)withtheCRassignment,butlateronitaidedtheteaminevaluatingwhichBLIspresentedmoredefectsofhigherseverity.

Onthebasisofthisinformationwecouldplantestexecutionfo-cusingontwoaspects:(i)validateifBLIswithfewornodefectsweresufficiently tested; (ii)analyzeBLIs thatshowedagreateramountofdefects,theircharacteristicsandwhichtestscenarioscouldbere-testedoraddedtoallowthediscoveryofnewbugs.Greater emphasis was applied to the second aspect, as de-scribedbyMyers“Theprobabilityoftheexistenceofmoreerrorsinasectionofaprogramisproportionaltothenumberoferrorsalreadyfoundonthatsection”.

Thisapproachprovedtobeefficientasnewerrorswerefound.Someadjustmentsweremadetomakethistaskfaster.Forex-ample,insteadofaddingtheBLIidentificationinthedescriptionheader,weaddedanewtextfieldtothedefectmanagementtoolsowecouldidentifytheBLIandlateroneasily identifythede-fectsfound.

Greater level of detail in defect descriptionOneof themost importantactivitiesof the testengineer is re-cordingthedefects,foundinadefectmanagementtool.Never-theless, somepeopledidn’t perform this taskasexpected. Is-suesweresometimesnotcompletelydescribed,makingitharderformanagersanddeveloperstounderstandtheerror.Lateronthisgeneratedseveralinterruptionsforthetestengineerinordertoclarifytheissuedescriptionor,evenworse,todiscardthede-fect.Therefore,itisveryimportantforthedefectstobereportedinadetailedanddidacticmanner,makingeveryone’sjobeasier,includingothertestersthatmightbe involved inretestingaftertheissuegetsfixed.

Toassistinthistask,onerealbenefitistoaddarecordedvideooftheissueor,ifthatisnotpossible,atleastascreenshot.Iftheissuecan’tbereproducedonthesimulator,usearegularcameratocaptureit.Justrememberthatthisisnotarule.It’suptoyoutoevaluatewhetheraCRshouldbe improvedbyaddingsomeextraresource(afterall,notallissueshaveavisualfeedback).

Defect management toolAnotherhighlyimportantassettoassistthetestengineer’sworkis thedefectmanagement tool. Inourproject theopensourcetoolMANTIS was used. It provides the engineer with effectivecontrol of the registered defects, allows trends analysis and,mostofall,enhancesteamcommunication.

Taking notesKeepinganelectronicnotepadalwaysopenorevenapieceofpaperandpenonyourdeskcanreallyhelpduringtestactivities.Withthetimepressureandtightdeadlinesalwayspresent,itisnotalwayspossibletotestallthescenariosthatweforeseedur-ingtesting.Thismayhappenbecauseoftheneedofkeepingfo-cusedonthecurrenttestcycleorothertasksbeingperformed.Ifnotwrittendownsomewhere,theseotheractivitiesorevenhintsobservedduringteammeetingsmaygetlostandnevertested.

Makingahabitofnote-taking forany relevant information thatmighthelpinfutureactivities(forexample,anewtestscenarioorsystemcharacteristic,someuserfeedback,etc.),willhelpthetest engineer to avoid forgetting interesting investigations thatcouldbeperformedandassistinfindingnewbugs.

LEASSONS LEARNEDThroughouttheproject,manynewexperienceswerefacedandalotwaslearnedbyworkinginaprojectwithverydifferentcharac-teristicstothosefoundinregularsoftwaredevelopmentprojects.

Test case designAfter gaining some experience in test case design for games,weobservedthatteststhatwererelatedtofunctionalitiesdidn’tneed tobe repeatedondifferentgamescenariosbecause theerrorfoundappliedtothewholeapplication(e.g.pressingaspe-cifickeydoesn’tproduce theexpectedbehavior).On theotherhand,whenconsideringtheuserinterfaceandart,thistypeoftestneededtoberepeatedforeachscenariooftheapplication,sincesomeerrorscanbefoundonlyinspecificscenarios.

Using cheat codesAsthegameevolved,acoupleofcheatcodes(codingthatpro-videdspecialgameadvantagesthatwouldnotbeavailableonthe final version, such as “invincibility”),were created tomakesome testseasier toexecute. forbothdevelopersand testers.However,weneedtobeextremelycarefulwhendecidingtousethistypeofassistance.Iftheyaremisused,itcanleadtohiddenbugsor,conversely,bugsthatonlyappearduetotheexistenceofthecheatcode.Anexamplethathappenedtouswasacheatcodethatunlockedagamelevel,whichblockeddevelopersfromreproducingabugreportedbytesters.

Figure 2 – Severity of reported defects


Ifcorrectlyused,cheatcodescanincreasetheteam’sperform-anceandevenhelpanticipatingbugs.

Team communicationIt isalso important for the testing teamtodefinespecific timeintervalsthroughouttheprojectwherereportsfromtestexecu-tioncycleswillbesentfortherestoftheteam.Thishelpsustomakeourworkvisibleandunderstandablefortheentireteam.Although SCRUM already makes tasks communication easieramongteammembersbyapplyingthedailymeetingsandburn-downgraphs,westillneedamoreformaltypeofcommunication.Arecommendedmomentforthesereportsisattheendofeachsprint.

Activities followed up using the burndown graphAstestengineerswecommonlyneedtoassurethatweareus-ing the latest available versions for testing. During the courseofasprint,thetimeforrequestingnewpartialversionscanbedeterminedthroughthedailymeetingsandtheburndowngraph,wherewecanbeawareofthestageofeachBLIandsetupwiththedevelopersthenumberoffeaturesavailablefortesting.Thetestengineerneeds tokeepconstantly in touchwith the teamleader toassurethat these intermediateversionsget releasedforcomponentandexploratorytesting.

BLIs changes closely followedGenerally on Agile projects, but especially on ours, a greatamountofchangehappenedtotheBLIswhichdirectlyimpactsonthepreviouslydesignedtestcases.Therefore,itishighlyrel-evanttostaytunedtochangescausedbyclient’sfeedbacks,us-abilitytestsandreports,meetingsandalsotechnicallimitations.This follow-up ismadeeasierwhen theGameDesignerworkscloselywiththerestoftheteamandkeepseveryoneinformedwhenchangesoccur.Thisway,anyquestionsrelatedtoanyfea-tureofthegamecanbeeasilydiscussedandclarifiedwiththeGameDesigner.

Informally reported defects don’t get fixedJustlikeatesterforgetstotestscenariosthathedoesn’tdocu-ment,byinformallyreportingadefecttoateammember,(devel-oper,GDorartist),thereisnoguaranteeofafix.Nomatterwhattheseverityofthereportis,theinformalcommunicationcreatesahighriskthatitdoesn’tgetfixedor,iffixed,thatitmaynotbevalidated.

Retest with different devicesThroughouttheprojectseveralerrorswerefoundwhichwerere-latedtothegame’sinteractionwithspecificdevicesorexternalapplicationssuchasphonecallsorSMS.Thiskindoferrorcaus-esconsiderableeffort for thedevelopment teamto investigatetheissueandtofindoutthecause.However,thiskindoferrorcan’tbesolvedbyourteamsinceitisnotrelatedtoourgame.Therefore,itisagoodapproachtoretestwithdifferentdevicesordifferentgameswithsimilarcharacteristics.Thisextrainforma-tionwon’teliminatetheCR,butdeveloperscanstartanalyzingwiththisaspectinmind.

Thisapproachcanalsobeappliedtodifferenttypeofscenarios,such as performance, sound effects, rendering and other fea-tureswhichcanbecomparedwithsimilargames.

CONCLUSIONNomatterwhat stagewe are at in the development life cycleortheexperiencelevelofthedevelopingteam,therearemanycauses forsoftwarebugs.Mostof themarenot related to thecodeitself,buttootherproblems,suchasincompleteorunclearrequirements,hardwareissuesandintegration.Therefore,con-sideringthepracticesandlessonslearnedfromthisproject,weareconvincedthatsoftwarequalityisahighpriorityformodernsoftware products, likemobile games, and that achieving thisshallbeacommongoalfortheentireteamwithacleardivisionof responsibilities.Making sure that there are no conflicts be-tween developers, testers and other teammembers regardingquality, everyonemust work together to deliver a high qualityproduct.Onlybyplacingpriorityonqualitywewillbeabletode-liverproductsthatfullymeetourclients’needsandexpectations.

José CarréraMSc, has been test engi-neer at C.E.S.A.R. (Recife’s Center for Advanced Stu-dies and Systems) since 2006 and Professor of Computer Science at FA-TEC (Faculdade de Tec-nologia de Pernambuco), Brazil, since 2010. He ob-tained his master degree

in software engineering (2009), graduated in computer science (2007), and is a Certified Tester - Foundation Level (CTFL) by the ISTQB (2009). His main research in-terests include quality assurance, agile methodologies, software engineering and performance testing.

> About the author

Test Planning and Execution in a Mobile Game Development Project using SCRUM

Documents