- 1. BT3 ConcurrentSession 11/8/201210:15AM "Massive Continuous
Integration and Light-speed Iterations" Presented by: Jesse Dowdle
AtTask, Inc. Broughttoyouby:
340CorporateWay,Suite300,OrangePark,FL32073
[email protected]
2. Jesse Dowdle AtTask, Inc. Senior manager of development for
AtTask Inc. Jesse Dowdle directs the engineering efforts of nearly
a dozen agile teams in the U.S. and overseas. Successful
engineering in a fast-paced industry requires constant research in
emerging technologies. Jesse drives thought leadership on
continuous delivery, development best practices, and process
evolution to help AtTask ship software at lightning speed. Happiest
when hes building great software, Jesse sees technology, process,
and people as equally important prerequisites for success. A native
of Utah, where he enjoys fast cars in the summer and snowboarding
in winter, Jesse tweets at @dowdlemj and blogs. 3.
[email protected] Continuous Integration and Light-speed
IterationsJesseDowdle, Director of Development@dowdlemj
linkedin.com/in/jessedowdle www.continuousdeliverist.com 4.
SaltLake City, UtahSoftware Projectas a
ServiceManagementCollaborativeSaltWork ManagementLake City,
UtahSoftware Projectas a ServiceManagementCollaborativeWork
Management 5. SaltLake City, UtahSoftware Projectas a
ServiceManagementCollaborativeWork ManagementAgile at AtTask 5
years Wall to wall 11 Dev Teams 6. How we did
ScrumSprintSprintSprintHow we did Scrum 4 WEEKS SprintSprintSprint
7. How we did Scrum 2 days Demo Retrospective Plan4 WEEKS
SprintSprintSprintHow we did Scrum 2 days Demo Retrospective Plan4
WEEKS SprintSprintPublic ReleaseSprintPublic ReleasePublic Release
8. How we did Scrum 2 days Demo Retrospective Plan4 WEEKS
SprintInternalInternalSprintInternalPublic
ReleaseInternalInternalSprintInternalPublic
ReleaseInternalInternalInternalPublic ReleaseInternalPublic
ReleaseHow we did Scrum 2 days Demo Retrospective Plan4 WEEKS
SprintInternalInternalSprintInternalPublic
ReleaseInternalInternalSprintInternalPublic ReleaseInternalInternal
9. Then Scripted scenarios Slow, brittle Monthly Releases 3-5 days
acceptance Manual administrationNow Tight unit-like UI tests Fast,
Robust Daily releases 30-45 minutes Sign-off and go 10. What is
Continuous Integration? Continuous Integration is a software
development practice where members of a team integrate their work
frequently, usually each person integrates at least daily - leading
to multiple integrations per day. Each integration is veried by an
automated build (including test) to detect integration errors as
quickly as possible. Many teams nd that this approach leads to
signicantly reduced integration problems and allows a team to
develop cohesive software more rapidly. -- Martin FowlerComponents
Version Build TestControl
SystemServerRunnerTestSuiteNoticationMechanism 11. Basic
Integration Version Build TestControl triggers new actionServer
retrieves latest version, compiles, starts Application StackRunner
executes tests against Application StackResults Buildare collected
and sent by noticationServer is cleaned and restored.Basic
Integration 12. BASIC INTEGRATIONBUILD SERVER VERSION CONTROLTEST
RUNNERNOTIFICATION SYSTEMBASIC INTEGRATION 13. BUILD SERVER
APPLICATION STACKVERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC
INTEGRATIONBUILD SERVER APPLICATION STACKVERSION CONTROLTEST
RUNNERNOTIFICATION SYSTEMBASIC INTEGRATION 14. BUILD SERVER
APPLICATION STACKVERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC
INTEGRATIONBUILD SERVER VERSION CONTROLTEST RUNNERNOTIFICATION
SYSTEMBASIC INTEGRATION 15. Massive Test Automation 800Unit
tests3300API tests2000UI tests55,000hours per
monthReleaseAcceptance + CITrue Continuous Integration Howoften do
I integrate?Whattests tell me Im integrated?Whatmust I know to
release? 16. Pipeline Priorities Performance(Cycle Time)Scalability
Visibility AccountabilityATTASKS PIPE BUILD -- DEPLOY -- TEST --
DESTROY 17. CommitATTASKS PIPE BUILD -- DEPLOY -- TEST --
DESTROYCommit Build Installer Build TestsATTASKS PIPE BUILD --
DEPLOY -- TEST -- DESTROY 18. Selenium Grid Grid + AtTask Commit
Build Installer Build TestsATTASKS PIPE BUILD -- DEPLOY -- TEST --
DESTROYSelenium Grid Grid + AtTask CommitInstall Build Installer
Build TestsATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROY 19.
Selenium Grid Grid + AtTask CommitInstall Build Installer Build
TestsUnitIntegration 1/43/42/44/4ATTASKS PIPE BUILD -- DEPLOY --
TEST -- DESTROYSelenium GridSelenium IEGrid + AtTask
CommitInstallSelenium FFBuild Installer Build TestsOther
testsUnitIntegration 1/4 2/4ATTASKS PIPE BUILD -- DEPLOY -- TEST --
DESTROY3/4 4/4 20. Selenium GridSelenium IEGrid + AtTask
CommitInstallSelenium FFBuild Installer Build TestsOther
testsUnitIntegration 1/43/42/44/4ATTASKS PIPE BUILD -- DEPLOY --
TEST -- DESTROYHOW TO MAKE A PAPER AIRPLANE OR, QUEUEING THEORY AS
EXPRESSED BY A FOLDED PROJECTILETear Down 21. Selenium GridSelenium
IEGrid + AtTask CommitInstallSelenium FFTear DownBuild Installer
Build TestsOther testsUnitIntegration 1/43/42/44/4SERIAL VS
PARALLELSelenium GridSelenium IEGrid + AtTask CommitInstallSelenium
FFBuild Installer Build TestsOther testsUnitIntegration 1/4
2/4SERIAL VS PARALLEL3/4 4/4Tear Down 22. CYCLE TIMECYCLE TIME 23.
CYCLE TIMECI Vendors CruiseControl Jenkins (Hudson) TeamCity Go
Team Foundation Server 24. Cloud Vendors Amazon Web Services
OpenStack SauceLabsJENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVEN
25. JENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVENJENKINS DYNAMIC
SLAVE ALLOCATION 26. JENKINS DYNAMIC SLAVE ALLOCATIONSLAVEJENKINS
DYNAMIC SLAVE ALLOCATION 27. SLAVE SLAVE SLAVE SLAVEJENKINS DYNAMIC
SLAVE ALLOCATIONJENKINS DYNAMIC SLAVE ALLOCATION 28. Scalability
Taking Tothe Test Suite to massive scalethe cloud with Amazon Cloud
FormationsDynamicSelenium GridsDynamicAtTask EnvironmentsDivide and
Conquer Moduleseparation will allow for scaleTestSuites should
support shardingRuntests with different dependencies in parallel
29. Divide and Conquer Selenium GridSelenium IEGrid + AtTask
CommitSelenium FFInstall Build Installer Build TestsOther
testsUnitIntegration 1/43/42/44/4Divide and Conquer UnitIntegration
1/43/42/44/4Tear Down 30. PARALLEL BY DEPENDENCIESIn
MemoryDatabaseEmailPARALLEL BY DEPENDENCIESUI OnlyCustomer Data 31.
In Memory Database Email UI Only Customer DataPARALLEL BY
DEPENDENCIESFrom 3 days to 30 minutes with Selenium 2 32. SELENIUM
1 (1800 TESTS)SOMETIMES THE TESTS NEED TO CHANGESELENIUM 1 (1800
TESTS) 4 HOURSSLOWSOMETIMES THE TESTS NEED TO CHANGE 33.
TRANSITIONING TO SELENIUM 2TRANSITIONING TO SELENIUM 2 34.
TRANSITIONING TO SELENIUM 2SELENIUM 1 (1800 TESTS) 4
HOURSTRANSITIONING TO SELENIUM 2 35. SELENIUM 1 (1800 TESTS) 4
HOURSSELENIUM 2 (750 TESTS) 30 MINUTESTRANSITIONING TO SELENIUM
2TRANSITIONING TO SELENIUM 2WHERE DID THE TESTS GO? 36. 2.1
MINUTESTRANSITIONING TO SELENIUM 22.1 MINUTESTRANSITIONING TO
SELENIUM 234 SECONDS 37. 2.1 MINUTES34 SECONDS WITH
GRID...TRANSITIONING TO SELENIUM 22.1 MINUTES34 SECONDS WITH
GRID...8 SECONDSTRANSITIONING TO SELENIUM 2 38. 2.1 MINUTES34
SECONDS WITH GRID...8 SECONDS2.4 SECONDSTRANSITIONING TO SELENIUM
2JSON PARTLY CLOUDY WITH A CHANCE OF RAIN 39. JSON PARTLY CLOUDY
WITH A CHANCE OF RAINScalability Selenium GridSelenium IEGrid +
AtTask CommitInstallSelenium FFBuild Installer Build TestsOther
testsUnitIntegration 1/43/42/44/4Tear Down 40. Scalability Selenium
GridGrid + AtTaskCommit Build InstallerBuild TestsSLAVEITS RAINING
GRIDS 41. SLAVEITS RAINING GRIDSSLAVESELENIUM GRIDSELENIUM GRIDITS
RAINING GRIDS 42. SLAVESELENIUM GRIDSELENIUM GRIDITS RAINING
GRIDSSLAVESELENIUM GRIDSELENIUM GRIDITS RAINING GRIDS 43.
SLAVESELENIUM GRIDSLAVESLAVE SELENIUM GRIDSLAVESLAVEITS RAINING
GRIDSVisibility Testresults in JenkinsDescription TheresView
Allpluginmy commit!pluginthe jobs on one line, oh the
humanity!Screen-shotting TestUI TestsAge, Project Health 44.
Visibility Testresults in JenkinsDescription TheresView Allpluginmy
commit!pluginthe jobs on one line, oh the humanity!Screen-shotting
TestUI TestsAge, Project HealthVisibility Testresults in
JenkinsDescription TheresView Allpluginmy commit!pluginthe jobs on
one line, oh the humanity!Screen-shotting TestUI TestsAge, Project
Health 45. Visibility Testresults in JenkinsDescription TheresView
Allpluginmy commit!pluginthe jobs on one line, oh the
humanity!Screen-shotting TestUI TestsAge, Project HealthVisibility
Testresults in JenkinsDescription TheresView Allpluginmy
commit!pluginthe jobs on one line, oh the humanity!Screen-shotting
TestUI TestsAge, Project Health 46. Accountability Claim/ Blame
test results using AtTaskAccountability Claim/ Blame test results
using AtTask 47. Accountability Claim/ Blame test results using
AtTaskCI as a Platform On-Demand Easydemo environmentsbutton for
version controlVisibility Otherinto production systemsquality
measurementsStaticAnalysisCodeCoverage 48. Stafng for Massive CI
Getthe right skill setWatchyour integration pointsTechnological
OrganizationalDevOpsAdoption Process/ Tool alignmentInformation
Integration ScrumRadiators with existing
toolsboardBacklogIntegrationwith Release Management 49. CI
EVOLUTION ITS A CONTINUOUS INVESTMENTWhat does it cost?
3FTETwohuman, one machine(s)Keeping Efcientcosts down machine
sizesRock-solid Elasticitytear down 50. Return on investment 5days
(40 hours) to certify one release...10engineers$40per
hour$20,00030minutes to certify one release...10engineers$40per
hour$250To release 12 times in a year... $240,000 TO CERTIFY$3000
TO TEST $160,000 FOR TWO FTE $77,000 FOR CITo release 24 times in a
year... $480,000 TO CERTIFY$6000 TO TEST $160,000 FOR TWO FTE
$77,000 FOR CITo release 100 times in a year... $2,000,000 TO
CERTIFY$25000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI 51. 100
deployments annually$2,000,000100 deployments
annually$265,000$2,000,000 52. 100 deployments
annually$265,000Where to go from here
ContinuousDeliveryApplicationLifecycle
ManagementContinuousIntegration of InfrastructureOpenSource 53.
Tools Manifest AtTask EC2-contributions...
wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+PluginCloudFormation
PipelineView -Description
GIT--wiki.jenkins-ci.org/display/JENKINS/AWS+Cloudformation+Plugingithub.com/JoelJ/PipelineDashboardSetter
-github.com/JoelJ/DescriptionSetter/wiki.jenkins-ci.org/display/JENKINS/Git+PluginAtTaskfor
Jenkins -Coming soon...Further Reading ContinuousIntegration --
Martin Fowler -
http://www.martinfowler.com/articles/continuousIntegration.html
ExtremeProgramming Explained -- Kent Beck -
http://amzn.to/c0VvTBContinuousDeployment in 5 Easy Steps -- Eric
Ries -
http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html
FourPrinciples of Low Risk Software Releases -- Jez Humble -
http://www.informit.com/articles/article.aspx?p=1833567
ContinuousDelivery -- Jez Humble & David Farley -
http://continuousdelivery.com/ TheLean Startup -- Eric Ries -
http://theleanstartup.com/ContinuousIntegration -- Paul Duvall -
http://amzn.to/9Zx1l1 54. Thank you! JesseDowdle, Sr Manager of
Development@dowdlemj linkedin.com/in/jessedowdle
www.continuousdeliverist.comDavidTolley, Sr QA Automation
Engineer@davidtolley linkedin.com/prole/view?id=81007074