ASSESSING THE EFFECT OF DATA TRANSFORMATIONS ON TEST SUITE COMPILATION Panagiotis Stratis, Vanya Yaneva, Ajitha Rajan 12 October 2018 ESEM'18, Oulu, Finland
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ASSESSINGTHEEFFECTOFDATATRANSFORMATIONSONTESTSUITE
COMPILATIONPanagiotisStratis,VanyaYaneva,AjithaRajan
12October2018ESEM'18,Oulu,Finland
SOFTWAREISEVERYWHERE
SAFETYANDCORRECTNESSARECRUCIALTESTINGISCRITICAL
TESTINGCANBEEXTREMELYTIMECONSUMING.
FUNCTIONALTESTING
Testingframeworks:
GoogleTest(C++)JUnit(Java)Mocha(JavaScript)
...and100sothers
HOWDOWEIMPLEMENTTESTING?
Repeatthecycle:
everycommiteverymergeTest-drivendevelopment(TDD)
everyday(overnightbuilds)Continuousintegration(CI)
Testingtakesasignificantportionofthedevelopmenttime.
WHENDOWETEST?
Allthetime!
1. Write/generatetests-✔Automatedtestgeneration,paralleltestgeneration
2. Build(compile)-?
3. Runtests-✔Testsuiteminimisation,testcaseprioritisation,paralleltestexecution
SPEEDINGUPTESTINGEXISTINGRESEARCH
REDUCINGCOMPILATIONTIMEISIMPORTANT
Largetestsuitestakealongtimetocompile."...Comparabletorunningtime."[CodeplaySoftware]Weneedtocompilenotonlythesystemcode,butalsothetestcode.
Compileroptimisationsincreasecompilationtime.-O1,-O2,-O3
Testcodeneedstobecompiledoften.
CONTRIBUTIONS
Codetransformationstargetingtestcode,resultinginshortercompilationtimes.
Empiricalevaluationusing15programsfromEEMBC&SPECand1largeindustryprogram.
Speedupincompilationtime:1.3xto69x.
ContainsmanyfunctioncallstotheFUT-oneforeachtestinput.
Highcompilationoverheadforfunctioninliningandinstructionselection.
Thisnumbergrowsasmoretestsareadded.
OURAPPROACH-EXAMPLE
ContainsmanyfunctioncallstotheFUT-oneforeachtestinput.
Highcompilationoverheadforfunctioninliningandinstructionselection.
Thisnumbergrowsasmoretestsareadded.
Hypothesis:ReducingthenumberofcallstotheFUT,atcompiletime,willreducecompilationtime.
OURAPPROACH-EXAMPLE
OURAPPROACHDATATRANSFORMATIONOFTHETESTS
EVALUATION-RESEARCHQUESTIONS
RQ1:CompilationspeedupDoesthetransformationspeedupcompilationtime?
RQ2:ScalabilityDoesthetransformationallowustocompilelargertestsuites?
RQ3:ExecutiontimeandcorrectnessoftestingDoesthetransformationimpacttheexecutiontimeandcorrectnessoftesting?
EVALUATION-SUBJECTS
EEMBCindustry-standardembeddedsystemsapplicationsused5telecomprograms&5automotiveprograms
SPECwell-knownbenchmarkssuiteofcomputeintensiveapplicationsused5programs,includingbzip2,libquantum
Tests:10KrandomlygeneratedtestinputsCompilers:gcc,clang
ComputeCPP,CodeplaySoftwareimplementationoftheSYCLheterogeneousprogrammingmodel
Tests:2testsuites-imageTS&bufferTSeachhasapprox.10Ktests,producedbyCodeplaydevelopers
Compilers:2customcompilers,basedonClangoneforhost&onefordevice
EVALUATION-SUBJECTS
Maxspeedup1.5x(avg.1.3x)
Maxspeedup1.8x(avg.1.4x)
RQ1:SPEEDUP-EEMBC
Maxspeedup15x(avg.7.9x)
Maxspeedup20.2x(avg.12x)
RQ1:SPEEDUP-SPEC
SPEEDUPVARIATIONACROSSPROGRAMS
-ftime-report:showstimespentcompilingindividualfilesandfunctions.
Comparedtimetocompiletestcodevswholecode.
SPEEDUPVARIATIONACROSSPROGRAMS
SPEEDUPVARIATIONACROSSPROGRAMS
Averagecompilationtimeoftestcodeaspercentageoftotalcompilationtime.
SPEEDUPVARIATIONACROSSPROGRAMS
Averagecompilationtimeoftestcodeaspercentageoftotalcompilationtime.
Usingmodulardesignwithpre-compiledlibrariesgreatlyimprovesspeedup.
SPEEDUPANALYSISForEEMBCandSPEC,
Analysedassemblycodegeneratedbythecompiler.Analysedtimespentbyeachcompilerpass,using-ftime-report.
SPEEDUPANALYSIS
Beforetransformation:
SeparatecallstotheFUTemittedforeachtest.
47%oftestcodecompilationspentininstructionselectionfunctioninliningcombineredundantinstructions
Aftertransformation:
1calltotheFUTemitted.
Only13.6%ofcompilationtimespentinthosepasses.
ForEEMBCandSPEC,
Analysedassemblycodegeneratedbythecompiler.Analysedtimespentbyeachcompilerpass,using-ftime-report.
Maxspeedup69.5x(avg.42x)
Orig.time[s] Newtime[s]
bufferTS 257 4
imageTS 434 29
RQ1.SPEEDUP-COMPUTECPP
RQ2.SCALABILITY
Beforetransformation:
Clangandgcccrashwhen1milliontestsarereached.
Aftertransformation:
Clangandgccsuccessfullycompile10milliontests.
ForEEMBCandSPEC,
wegenerated10millionrandomtestinputs.
forhighestoptimisationlevel-O3.
RQ3.EXECUTIONTIMEANDCORRECTNESS
ForEEMBC,SPECandComputeCpp,
ExecutiontimeOurtransformationdoesnotslowdowntheexecutionofthetestcode.
CorrectnessOurtransformationpreservesthecorrectnessofthetestexecution.
TAKEAWAYTestcodecanaddsignificantcompilationoverhead,contributingtototaltestingtime.
Itcanbereducedwithcodetransformationsonthetestcode.
Thisisparticularlyeffectivewhenusingmodulardesignandpre-compiledlibraries.
Related Documents
