Explore the State of Open Source Performance Testing in Continuous Delivery Pipelines with BlazeMeter Chief Scientist Andrey Pokhilko

World®’16

ExploretheStateofOpenSourcePerformanceTestinginContinuousDeliveryPipelineswithBlazeMeter ChiefScientistAndreyPokhilkoAndreyPokhilko - ChiefScientist,BlazeMeter - CATechnologies

DO5X54E

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

ContinuousIntegrationhasrapidlybecomethedefactostandardinsoftwaredevelopmentprocesses.Whilefunctionalandunittestinghaveprovedrelativelyeasytointegrateintotheseprocesses,performancetestinghastypicallyraisedmorechallenges.Inthistalk,AndreywillexaminekeyapproachestointegratingperformancetestingintoContinuousDeliveryworkflows,includinghowtoovercomethekeychallengesencounteredineachmethod.

Ultimately,atruecontinuousdeliveryprocessreliesupontheactualizationoffullyautomated,configurationdrivenandversion-basedworkflows.YouwillgetanunderstandingonhowtoevolveyourCIsystemtofacilitatethis.

AndreyPokhilkoCATechnologiesChiefScientist,BlazeMeter


ImportanceofCIandCD

§ Machinetimecostsnothing,humantimepriceless

§ De-factowinningpractice

§ MostadvancedteamsgowithCD

§ Alotofteamsarestillinprocessofadoptingit


Agenda

THINGSWEPUTINTOCI

BESTPRACTICE(FORNOW)

CHALLENGESOFTESTINGINCI

OVERCOMINGCHALLENGES

1

2

3

4

JENKINSFORPERFORMANCETESTING

SOMEUSEFULPLUGINS

5

6


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


TriggeringandPreparations

§ Whentodothejob

§ VCScheckout+dependenciescheckout

§ Buildingproject(compilingetc.)

§ Putresultingpackagesintorepos


QualityControl

1. Staticcodeanalysis

2. Unittests

3. Functionaltests

4. Performancetests

needs deployment


Deployment

§ ToQAenvironment(forfurthermanualtests)

§ Tostagingenvironment

§ Toproductionenvironment


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


OneWayRoad(Driverless)

VCSCheckout+

DependenciesBuild Unit

Tests

FunctionalTests

PerfromanceTests

Deployment

Filter Filter Filter Filter


VCS-DrivenPipeline

§ NaturalevolutionofCIsystems

§ Branchingandpullrequests

§ Jenkins2.0pipelines

§ TaurusToolaspartofthisapproach


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


Challenge#1:TestEnvironment

§ Applicationsarecomplex§ Lotsofdependencies§ Third-partysystems

Databases

Microservices

Third-parties


Challenge#2:TimeConsuming

§ Preparations§ Alotoffunctionaltests§ Performancetestsarenaturallylong


Challenge#3:DebuggingCIJobs

§ Evolvingtestcomplexity§ Debuggingandtroubleshooting§ Buildhistoryisavalue


Challenge#4:ResultsAnalysis

§ Richreportingneeded§ JenkinsUIhasitslimits§ AutomateddecisionmakingforCD§ Collaborationfornon-techpeople

wut???


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


Overcome:TestEnvironment

§ Containershelp(youhavetousecontainers:)§ Youdon’tneedtobeasrealistic

≈


Overcome:TimeConsuming

§ Preparewhatyoucanupfront- inthenight§ Reusewhatyoucan§ Shorttestscanrevealalotofthings§ Don’ttrytoputspikeandendurancetestsintoCI§ Parallelizetests,Jenkins2.0orTaurushelps


Overcome:DebuggingandImproving

§ TaurusTool,thehandheldpieceofCI


Overcome:DebuggingandImproving

LoadTestJob

LocalDebugging


Overcome:Reporting&ResultConsumption

§ Purpose-builtserviceswithintegrations§ Overview&StatusinJenkins


Overcome:DecisionMaking

§ Deploytostagingisgreatwin§ Havepass/failcriteria,includingAPMinfo§ It’soktohavedeployasseparatejob(forUAT,toavoid

disruptions)


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


WhyUseJenkinsforTesting?

§ Multi-stepprocess§ Longprocess§ Repetitiveprocess


StartSimple

LoadTest1

LoadTest2

LoadTest3

Center of Excellence

Developer

Operations

QA

Maintain Jobs


BecomeContinuous

§ Periodicjobshavemanyapplications§ Havevalidfailurecriteria(thresholds)§ Usenon-blockingdownstreamjobs


PursueContinuousDelivery

FastFeedbackDeveloper/

DevOps

Build+xUnit

FunctionalTests

LoadTests

Deploy Staging

SlowerFeedback


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


PlotPlugin


PerformancePublisherPlugin


BlazeMeterPlugin


TaurusTool

§ Whynotplugin,butcommand-linetool§ UniversalplugfortestsinsideJenkins§ TaurusdesignedtoworkwithotherJenkinsplugins

gettaurus.org


FinalSummary

WehavetotestinsideCI

Therearewaystostartitsimple

Therearetools&pluginstohelp

Jenkins2.0pipelinesFTW!


Questions?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T57SCaseStudy:HowIMovedTheNFLfromWaterfalltoAgilein4Months 11/16/2016at03:45pm

DO5X55SCaseStudy:WalmartBrasil—SeasonalPeakTrafficReadiness 11/16/2016at04:30am

DO5X56SNext-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagement 11/17/2016at04:30pm


MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P


Thankyou.

Stayconnectedatcommunities.ca.com

@CAWORLD#CAWORLD ©2016CA.AllRIGHTSRESERVED.40 @CAWORLD#CAWORLD

DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu

Explore the State of Open Source Performance Testing in Continuous Delivery Pipelines with BlazeMeter Chief Scientist Andrey Pokhilko

Technology