Test patterns for_continuous_delivery
Post on 10-May-2015
1096 Views
Preview:
DESCRIPTION
Transcript
Test Patterns forContinuous Delivery
Better Software EastFrom Andy Singleton, http://andysingleton.com
www.assembla.com
In this sessionIntroduction to CD, benefits of CD, and ways to pitch itMultiple code and test patterns to choose fromHow to get the automated tests you needChanges in the role of QA
Agile Projects
Program Launch
Releases
Plan Program Test Doc Deploy
SkipAutomate& Blend Lag Automate
Pull
CI & CD
End up with
Waterfall/Sprint to Continuous
Measure
Assembla in 2011“Scrumban” with iterative releases, but continuous planning
to accommodate a distributed team.Releases took longer as system got bigger and there was
more to test. 2 weeks -> 3 weeksBugs in production. 2 days for fixes. StressfulCompetitors achieved faster velocity with continuous delivery
Made a study of continuous methods with our own team, customers, and tools.
Now – releasing about 250 times per month. Fewer bugs. Much less stress.
You can do it incrementally1. Release more frequently2. Improve
When to use Continuous Delivery?You provide an online service. Competitive pressure will
force you to continuous: Office 365 vs Google Docs.You provide any service with software insideYour release times are getting longer, or the release
process is stressfulYou are developing a new product with lean startup and
MVP techniquesYou have a big project with a lot of contributors
This applies to hardware, also
Benefits for the test processEliminate stress from the release process. There is never
a big batch of stuff to test under pressure.
Add or remove test layers to increase quality or increase speed
Continuous integration removes the stress of acquiring and fitting components from many teams and suppliers
Your CEO wonders why the competition is releasing faster and moving faster
Continuous Delivery Dial
Dev
Business
Ops
DEV TEST UAT PROD
CONTINUOUS DELIVERY
RELEASE TRAIN
Self-Service
Controlled
INTEG STAGING
From Steve Brodie and Rohit Jainendran
The big question: How to test?We release software in batches so that we can test it.
That is the whole reason for doing it. We test software “release candidates” to make sure everything works together.
In continuous delivery, we might get as little at 10 minutes to test a release candidate
If we can’t test the complete version, how do we guarantee quality? Is it magic?
Test Layering – A Golden TicketMonitor your released software: Errors, Usage volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to ask for test scripts.
Continuous integration: Run automated tests before using human review time
Unit tests in the development environment
Switch new features and architectureStart h
ere to
add layers
Start here to
release a change
Go Both Ways
Veloc i ty
Qu
alit
y
Increase Quality
Increase Velocity
Switch and Unveil
HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.
Test Story Owner and testers see the change on test systems.
BetaInsiders see it and use it. Story Owner can show it to users for feedback or A/B testing.
UNVEIL! The big event. Communicate with all users. Measure reaction.
One code version
No special test builds
No long-running branches
Code and Test Patterns
Code and Test PatternsCentralized
Traditional process with no branches finds and fixes problems as early as possible
DistributedRelease every change with its own branch and test
Temporary branchesEfficient review, combines benefits of centralized and
distributedMAXOS
Use centralized continuous integration to manage a massively scalable IT system
Centralized CI/CDContributor Commits – “as early as possible” to find problems
ContinuousIntegration tests
Fail - alarm
Release CandidateTest System Release
QA Testing
Pass
Distributed Continuous DeliveryContributor Commits
Branchor Fork
Deployed version
Peer review merge requests
Other contributionsmerged and released“as late as possible”
QA ConsultsPass FinalAuto Test?
Merge backCurrent
Deploy
Distributed: Multiple Test SystemsContributor 1
Contributor 2
ProductionRevision
ReleaseAnytime
CI SystemQA Team
Test System 1
Test System 2
Assembla spins up test servers
Test and Review on Temp Branches
Contributor Commits
Auto tests
Mainline
Review Auto tests Review
• Github style – Pull requests or merge requests going to CI system• Gerrit style – Temporary branches for each contribution• Assembla git supports both – merge requests, and temp branches created
by a push to a protected branch.
Test before spending human time on review
The Services MegatrendDesktop Web App Cloud
Services
App
DB
Service
Service
Service
Matrix of Services - MAXOS
PrioritizedBacklog
CurrentWork
Each team releaseswhen ready
Service teamIntegration
test envProduction
service
Service teamIntegration
test envProduction
service
Service teamIntegration
test envProduction
service
Feedback on speed, errors, usage, and requests
Organize bytesting as one system
SAFe – Copyright Dean Leffingwell
See the diagram on the SAFe Site:http://scaledagileframework.com/
Looks like a good practice, but really,nobody in Silicon Valley would have allof those meetings
Coordinate without big meetings
PrioritizedBacklog
CurrentWork
Service teamIntegration
test env
Service teamIntegration
test env
Service teamIntegration
test env
Continuous Integration between latest dev version of each service
• Continuous integration helps teams coordinate.
• See dependencies between “producers” and “consumers”
• Errors and conflicts show related team contact info
• Meetings and changes negotiated between two teams, not many
Roles and Migration
Role: DeveloperDevelopers have more power and responsibility.Developers have more responsibility for testing.Developers (not QA or PM) decide when to release. This
is a strong finding.Incentives are correct. Developer might have to come back
from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.
Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!
Role: QAQA is a consultant.QA gets more respect. Developers have to ASK for
service.Developers do more of the testing work. They should
organize reviews and automated tests so that bugs don’t go through into the manual test process.
QA has more time to investigate usabilityQA monitors productivity and quality metrics
Incentives for Continuous FlowYou don’t need culture change. You only need to release more frequently1. If you do code review, you can get automated. No
browbeating and cajoling is required. Developers will ask for tests when they review.
2. If developers decide to release, they will take more responsibility for testing and automated tests.
3. If you release more frequently, developers will quickly learn not to break the build
4. If PM’s take unveil and measurement responsibility, they will make better features
Our Master Plan1. Release more frequently2. Improve
www.continuousagile.com/unblock
top related