Fuzzing - Columbia Universitysuman/secure_sw_devel/fuzzing.pdf · Blackbox fuzzing • Given a program simply feed random inputs and see whether it exhibits incorrect behavior (e.g.,

Fuzzing

SumanJana

*Acknowledgements:DawnSong,KostyaSerebryany,PeterCollingbourne

Techniquesforbugfinding

Automa'ctestcasegenera'on

LowercoverageLowerfalseposi0vesHigherfalsenega0ves

Fuzzing DynamicsymbolicexecuFon

StaFcanalysis ProgramverificaFon

HighercoverageHigherfalseposi0vesLowerfalsenega0ves

Blackboxfuzzing

Testprogram

Randominput

Milleretal.‘89

Blackboxfuzzing

•  Givenaprogramsimplyfeedrandominputsandseewhetheritexhibitsincorrectbehavior(e.g.,crashes)

•  Advantage:easy,lowprogrammercost•  Disadvantage:inefficient

–  InputsoUenrequirestructures,randominputsarelikelytobemalformed

–  InputsthattriggeranincorrectbehaviorisaaverysmallfracFon,probablyofgeVngluckyisverylow

Fuzzing

•  AutomaFcallygeneratetestcases•  Manyslightlyanomaloustestcasesareinputintoatarget

•  ApplicaFonismonitoredforerrors•  Inputsaregenerallyeitherfilebased(.pdf,.png,.wav,etc.)ornetworkbased(hWp,SNMP,etc.)

Inputgenerator

Monitor

TestapplicaFon

ProblemdetecFon

•  Seeifprogramcrashed–  Typeofcrashcantellalot(SEGVvs.assertfail)

•  Runprogramunderdynamicmemoryerrordetector(valgrind/purify/AddressSaniFzer)–  Catchmorebugs,butmoreexpensiveperrun.

•  Seeifprogramlocksup•  Rollyourowndynamiccheckere.g.valgrindskins

Regressionvs.FuzzingRegrssion Fuzzing

DefiniFon Runprogramonmanynormalinputs,lookforbadness

Runprogramonmanyabnormalinputs,lookforbadness

Goals Preventnormalusersfromencounteringerrors(e.g.,asserFonfailuresarebad)

PreventaWackersfromencounteringexploitableerrors(e.g.,asserFonfailuresareoUenok)

Enhancement1:MutaFon-Basedfuzzing

•  Takeawell-formedinput,randomlyperturb(flippingbit,etc.)

•  LiWleornoknowledgeofthestructureoftheinputsisassumed

•  AnomaliesareaddedtoexisFngvalidinputs– AnomaliesmaybecompletelyrandomorfollowsomeheurisFcs(e.g.,removeNULL,shiUcharacterforward)

•  Examples:ZZUF,Taof,GPF,ProxyFuzz,FileFuzz,Filep,etc.

Seedinput Mutatedinput Runtestprogram

?

Example:fuzzingaPDFviewer

•  Googlefor.pdf(about1billionresults)•  Crawlpagestobuildacorpus•  Usefuzzingtool(orscript)

–  CollectseedPDFfiles–  Mutatethatfile–  Feedittotheprogram–  Recordifitcrashed(andinputthatcrashedit)

MutaFon-basedfuzzing

•  Supereasytosetupandautomate•  LiWleornofileformatknowledgeisrequired•  LimitedbyiniFalcorpus•  Mayfailforprotocolswithchecksums,thosewhichdependonchallenge

EnhancementII:GeneraFon-BasedFuzzing

•  TestcasesaregeneratedfromsomedescripFonoftheinputformat:RFC,documentaFon,etc.–Usingspecifiedprotocols/fileformatinfo–E.g.,SPIKEbyImmunity

•  Anomaliesareaddedtoeachpossiblespotintheinputs

•  KnowledgeofprotocolshouldgivebeWerresultsthanrandomfuzzing

Inputspec Generatedinputs Runtestprogram

?RFC

EnhancementII:GeneraFon-BasedFuzzing

SamplePNGspec

MutaFon-basedvs.GeneraFon-based

•  MutaFon-basedfuzzer– Pros:Easytosetupandautomate,liWletonoknowledgeofinputformatrequired

– Cons:LimitedbyiniFalcorpus,mayfallforprotocolswithchecksumsandotherhardchecks

•  GeneraFon-basedfuzzers– Pros:Completeness,candealwithcomplexdependncies(e.g,checksum)

– Cons:wriFnggeneratorsishard,performancedependsonthequalityofthespec

Howmuchfuzzingisenough?

•  MutaFon-based-fuzzersmaygenerateaninfinitenumberoftestcases.Whenhasthefuzzerrunlongenough?

•  GeneraFon-basedfuzzersmaygenerateafinitenumberoftestcases.Whathappenswhenthey’reallrunandnobugsarefound?

Codecoverage

•  SomeoftheanswerstothesequesFonslieincodecoverage

•  Codecoverageisametricthatcanbeusedtodeterminehowmuchcodehasbeenexecuted.

•  Datacanbeobtainedusingavarietyofprofilingtools.e.g.gcov,lcov

Linecoverage

•  Line/blockcoverage:Measureshowmanylinesofsourcecodehavebeenexecuted.

•  Forthecodeontheright,howmanytestcases(valuesofpair(a,b))neededforfull(100%)linecoverage?

if( a > 2 ) a = 2; if( b >2 ) b = 2;

Branchcoverage

•  Branchcoverage:Measureshowmanybranchesincodehavebeentaken(condiFonaljmps)

•  Forthecodeontheright,howmanytestcasesneededforfullbranchcoverage?

if( a > 2 ) a = 2; if( b >2 ) b = 2;

Pathcoverage

•  Pathcoverage:Measureshowmanypathshavebeentaken

•  Forthecodeontheright,howmanytestcasesneededforfullpathcoverage?

if( a > 2 ) a = 2; if( b >2 ) b = 2;

BenefitsofCodecoverage

•  CananswerthefollowingquesFons–HowgoodisaniniFalfile?–AmIgeVngstucksomewhere? if (packet[0x10] < 7) { //hot path� } else { //cold path } – HowgoodisfuzzerXvs.fuzzerY– AmIgeVngbenefitsbyrunningmulFplefuzzers?

Problemsofcodecoverage

•  For:mySafeCopy(char *dst, char* src) { if(dst && src) strcpy(dst, src); }

•  Doesfulllinecoverageguaranteefindingthebug?

•  Doesfullbranchcoverageguaranteefindingthebug?

EnhancementIII:Coverage-guidedgray-boxfuzzing

•  SpecialtypeofmutaFon-basedfuzzing–  Runmutatedinputsoninstrumentedprogramandmeasurecodecoverage

– Searchformutantsthatresultincoverageincrease

– OUenusegeneFcalgorithms,i.e.,tryrandommutaFonsontestcorpusandonlyaddmutantstothecorpusifcoverageincreases

– Examples:AFL,libfuzzer

AmericanFuzzyLop(AFL)

Inputqueue

Seedinputs

Nextinput

MutaFon

Executeagainst

instrumentedtarget

branch/edge

coverageincreased?

Addmutanttothequeue

Periodicallycullsthequeuewithout

affecFngtotalcoverage

AFL

•  Instrumentthebinaryatcompile-Fme•  Regularmode:instrumentassembly•  RecentaddiFon:LLVMcompilerinstrumentaFonmode•  Provide64KcountersrepresenFngalledgesintheapp•  Hashtablekeepstrackof#ofexecuFonofedges

–  8bitsperedge(#ofexecuFons:1,2,3,4-7,8-15,16-31,32-127,128+)

–  Imprecise(edgesmaycollide)butveryefficient•  AFL-fuzzisthedriverprocess,thetargetapprunsas

separateprocess(es)

Data-flow-guidedfuzzing

•  Interceptthedataflow,analyzetheinputsofcomparisons–  Incursextraoverhead

•  Modifythetestinputs,observetheeffectoncomparisons

•  PrototypeimplementaFonsinlibFuzzerandgo-fuzz

Fuzzingchallenges

•  Howtoseedafuzzer?– Seedinputsmustcoverdifferentbranches– Removeduplicateseedscoveringthesamebranches

– SmallseedsarebeWer(Why?)

•  Somebranchesmightbeveryhardtogetpastasthe#ofinputsstaFsfyingthecondiFonsareverysmall– Manually/automaFcallytransform/removethosebranches

Hardtofuzzcode

void test (int n) { if (n==0x12345678) crash(); }

needs2^32or4billionaWemptsIntheworstcase

Makeiteasiertofuzz

void test (int n) { int dummy = 0; char *p = (char *)&n; if (p[3]==0x12) dummy++; if (p[2]==0x34) dummy++; if (p[1]==0x56) dummy++; if (p[0]==0x56) dummy++; if (dummy==4) crash(); }

needsaround2^10aWempts

Fuzzingrulesofthumb•  Input-formatknowledgeisveryhelpful•  GeneraFonaltendstobeatrandom,beWerspecsmakebeWerfuzzers

•  EachimplementaFonwillvary,differentfuzzersfinddifferentbugs– MorefuzzingwithisbeWer

•  Thelongeryourun,themorebugsyoumayfind–  ButitreachesaplateauandsaturatesaUerawhile

•  Bestresultscomefromguidingtheprocess•  NoFcewhereyouaregeVngstuck,useprofiling(gcov,lcov)!