Page 1
�
AW12 Concurrent�Session�11/13/2013�3:45�PM�
�����
"Test (and More) Patterns for Continuous Software Delivery"
���
Presented by:
Andy Singleton Assembla
���������
Brought�to�you�by:��
��
340�Corporate�Way,�Suite�300,�Orange�Park,�FL�32073�888Ͳ268Ͳ8770�ͼ�904Ͳ278Ͳ0524�ͼ�[email protected] �ͼ�www.sqe.com
Page 2
Andy Singleton Assembla
Andy Singleton is the founder of Assembla, a SaaS company providing tools for distributed agile development teams. Andy moved Assembla from bi-weekly releases to multiple daily releases. His forthcoming book Unblock! A Guide to the New Continuous Agile describes workflows for product management, task management, coding, and testing. Previously, Andy founded PowerSteering Software, a provider of enterprise project management software, and Cambridge Interactive, an eBusiness consultancy. He developed financial information systems for Reuters, Bloomberg, SNL Kagan, and Thomson. During his career, Andy has produced more than twenty major software and financial information products.
Page 3
Be#er%So(ware%East%From%Andy%Singleton,%h#p://andysingleton.com%
www.assembla.com%%
Page 4
In%this%session%! IntroducAon%to%CD,%benefits%of%CD,%and%ways%to%pitch%it%! MulAple%code%and%test%pa#erns%to%choose%from%! How%to%get%the%automated%tests%you%need%! Changes%in%the%role%of%QA%%
Page 5
Program% Launch%
Releases%
Plan% Program% Test% Doc% Deploy%
Skip%Automate%&%Blend% Lag% Automate%
Pull%
CI%&%CD%
End%up%with%
Waterfall%to%ConAnuous%
Measure%
Page 6
Assembla%in%2011%! “Scrumban”%with%iteraAve%releases,%but%conAnuous%planning%to%accommodate%a%distributed%team.%
! Releases%took%longer%as%system%got%bigger%and%there%was%more%to%test.%2%weeks%V>%3%weeks%
! Bugs%in%producAon.%%2%days%for%fixes.%%Stressful)! CompeAtors%achieved%faster%velocity%with%conAnuous%delivery%
! Made%a%study%of%conAnuous%methods%with%our%own%team,%customers,%and%tools.%
! Now%–%releasing%about%250%Ames%per%month.%%Fewer%bugs.%%Much%less%stress.%
Page 7
You%can%do%it%incrementally%1. Release%more%frequently%2. Improve%%%%%%%
Page 8
When%to%use%ConAnuous%Delivery?%! You%provide%an%online%service.%%CompeAAve%pressure%will%force%you%to%conAnuous:%Office%365%vs%Google%Docs.%
! You%provide%any%service%with%so(ware%inside%! Your%release%Ames%are%gedng%longer,%or%the%release%process%is%stressful%
! You%are%developing%a%new%product%with%lean%startup%and%MVP%techniques%
! You%have%a%big%project%with%a%lot%of%contributors%! This%applies%to%hardware,%also%
Page 9
Benefits%for%the%test%process%! Eliminate%stress%from%the%release%process.%%There%is%never%a%big%batch%of%stuff%to%test%under%pressure.%%
! Add%or%remove%test%layers%to%increase%quality%or%increase%speed%
! ConAnuous%integraAon%removes%the%stress%of%acquiring%and%fidng%components%from%many%teams%and%suppliers%
! Your%CEO%wonders%why%the%compeAAon%is%releasing%faster%and%moving%faster%
%
Page 10
ConAnuous%Delivery%Dial%
Dev
Business
Ops
DEV TEST UAT PROD
CONTINUOUS DELIVERY
RELEASE TRAIN
Self-Service
Controlled
INTEG STAGING
From%Steve%Brodie%and%Rohit%Jainendran%
Page 11
The%big%quesAon:%How%to%test?%! We%release%so(ware%in%batches%so%that%we%can%test%it.%%That%is%the%whole%reason%for%doing%it.%%We%test%so(ware%%“release%candidates”%to%make%sure%everything%works%together.%
! In%conAnuous%delivery,%we%might%get%as%li#le%at%10%minutes%to%test%a%release%candidate%
! If%we%can’t%test%the%complete%version,%how%do%we%guarantee%quality?%%Is%it%magic?%
Page 12
Test%Layering%–%A%Golden%Ticket%Monitor%your%released%so(ware:%Errors,%Usage%volume,%usage%pa#erns,%user%feedback%
QA%System%with%Human%test%consultants%
Code%review:%Both%a%manual%test,%and%a%place%to%ask%for%test%scripts.%
ConAnuous%integraAon:%Run%automated%tests%before%using%human%review%Ame%
Unit%tests%in%the%development%environment%
Switch%new%features%and%architecture%
Page 13
Go%Both%Ways%
Veloc i ty )
Quality)
Increase)Quality)
Increase)Velocity)
Page 14
Switch%and%Unveil%
Hidden%Programmer%sees%a%change%locally.%Change%is%tested%in%the%%main%version%but%not%seen.%
Test% Story%Owner%and%testers%see%the%change%on%test%systems.%
Beta%Insiders%see%it%and%use%it.%%Story%Owner%can%show%it%to%users%for%feedback%or%A/B%tesAng.%
UNVEIL!% The%big%event.%%Communicate%with%all%users.%%Measure%reacAon.%
One%code%version%
No%special%test%builds%
No%longVrunning%branches%
Page 16
Code%and%Test%Pa#erns%! Centralized%
! TradiAonal%process%with%no%branches%finds%and%fixes%problems%as%early%as%possible%
! Distributed%! Release%every%change%with%its%own%branch%and%test%
! Temporary%branches%! Efficient%review,%combines%benefits%of%centralized%and%distributed%
! MAXOS%! Use%centralized%conAnuous%integraAon%to%manage%a%massively%scalable%IT%system%
Page 17
Centralized%CI/CD%Contributor%Commits%–%“as%early%as%possible”%to%find%problems%
ConAnuous%IntegraAon%tests%
Fail%V%alarm%
Release%Candidate%Test%System% Release%
QA%TesAng%
Pass%
Page 18
Distributed%ConAnuous%Delivery%Contributor%Commits%
Branch%or%Fork%
Deployed%version%
Peer%review%merge%requests%
Other%contribuAons%merged%and%released%“as%late%as%possible”%
QA%Consults%Pass%Final%Auto%Test?%
Merge%back%Current%
Deploy%
Page 19
Distributed:%MulAple%Test%Systems%Contributor%1%
Contributor%2%
ProducAon%Revision%
Release%AnyAme%
CI%System%QA%Team%
Test%System%1%
Test%System%2%
Page 20
Assembla%spins%up%test%servers%
Page 21
Test%and%Review%on%Temp%Branches%
Contributor%Commits%
Auto%tests%
Mainline%
Review% Auto%tests% Review%
• Github%style%–%Pull%requests%or%merge%requests%going%to%CI%system%• Gerrit%style%–%Temporary%branches%for%each%contribuAon%• Assembla%git%supports%both%–%merge%requests,%and%temp%branches%%created%%
by%a%push%to%a%protected%branch.%
Test%before%spending%human%Ame%on%review%
Page 22
The%Services%Megatrend%Desktop Web App Cloud Services
App
DB
Service
Service
Service
Page 23
Matrix%of%Services%V%MAXOS%
PrioriAzed%Backlog%
Current%Work%
Each%team%releases%when%ready%
Service%team%IntegraAon%test%env%
ProducAon%service%
Service%team%IntegraAon%test%env%
ProducAon%service%
Service%team%IntegraAon%test%env%
ProducAon%service%
Feedback%on%speed,%errors,%usage,%and%requests%
Organize%by%tesAng%as%one%system%
Page 24
Coordinate%without%big%meeAngs%
PrioriAzed%Backlog%
Current%Work%
Service%team%IntegraAon%test%env%
Service%team%IntegraAon%test%env%
Service%team%IntegraAon%test%env%
ConAnuous%IntegraAon%between%%latest%dev%version%of%each%service%
• ConAnuous%integraAon%helps%teams%coordinate.%
• See%dependencies%between%“producers”%and%“consumers”%
• Errors%and%conflicts%show%related%team%contact%info%
• MeeAngs%and%changes%negoAated%between%two%teams,%not%many%
Page 26
Role:%Developer%! Developers%have%more%power%and%responsibility.%! Developers%have%more%responsibility%for%tesAng.%! Developers%(not%QA%or%PM)%decide%when%to%release.%This)is)a)strong)finding.%! IncenAves%are%correct.%%Developer%might%have%to%come%back%from%Friday%night%beers%to%fix%a%problem.%%This%provides%a%moAvaAon%to%make%good%decisions%and%automate%tesAng.%
! Features%can%be%released%but%hidden.%%Product%Managers%and%Marketers%will%unveil%when%they%are%ready.%Unblock!%
Page 27
Role:%QA%! QA%is%a%consultant.%! QA%gets%more%respect.%%Developers%have%to%ASK%for%service.%
! Developers%do%more%of%the%tesAng%work.%%They%should%organize%reviews%and%automated%tests%so%that%bugs%don’t%go%through%into%the%manual%test%process.%
! QA%has%more%Ame%to%invesAgate%usability%! QA%monitors%producAvity%and%quality%metrics%
Page 28
IncenAves%for%ConAnuous%Flow%You%don’t%need%culture%change.%%You%only%need%to%release%more%frequently%1. If%you%do%code)review,%you%can%get%automated.%No%
browbeaAng%and%cajoling%is%required.%%Developers%will%ask%for%tests%when%they%review.%
2. If%developers%decide%to%release,%they%will%take%more%responsibility%for%tesAng%and%automated%tests.%
3. If%you%release%more%frequently,%developers%will%quickly%learn%not%to%break%the%build%
4. If%PM’s%take%unveil)and)measurement%responsibility,%they%will%make%be#er%features%
Page 29
Our%Master%Plan%1. Release%more%frequently%2. Improve%%%%%%%
Page 30
www.conAnuousagile.com/unblock%