Test patterns for_continuous_delivery

Test Patterns forContinuous Delivery

Better Software EastFrom Andy Singleton, http://andysingleton.com

www.assembla.com

In this sessionIntroduction to CD, benefits of CD, and ways to pitch itMultiple code and test patterns to choose fromHow to get the automated tests you needChanges in the role of QA

Agile Projects

Program Launch

Releases

Plan Program Test Doc Deploy

SkipAutomate& Blend Lag Automate

Pull

CI & CD

End up with

Waterfall/Sprint to Continuous

Measure

Assembla in 2011“Scrumban” with iterative releases, but continuous planning

to accommodate a distributed team.Releases took longer as system got bigger and there was

more to test. 2 weeks -> 3 weeksBugs in production. 2 days for fixes. StressfulCompetitors achieved faster velocity with continuous delivery

Made a study of continuous methods with our own team, customers, and tools.

Now – releasing about 250 times per month. Fewer bugs. Much less stress.

You can do it incrementally1. Release more frequently2. Improve

When to use Continuous Delivery?You provide an online service. Competitive pressure will

force you to continuous: Office 365 vs Google Docs.You provide any service with software insideYour release times are getting longer, or the release

process is stressfulYou are developing a new product with lean startup and

MVP techniquesYou have a big project with a lot of contributors

This applies to hardware, also

Benefits for the test processEliminate stress from the release process. There is never

a big batch of stuff to test under pressure.

Add or remove test layers to increase quality or increase speed

Continuous integration removes the stress of acquiring and fitting components from many teams and suppliers

Your CEO wonders why the competition is releasing faster and moving faster

Continuous Delivery Dial

Dev

Business

Ops

DEV TEST UAT PROD

CONTINUOUS DELIVERY

RELEASE TRAIN

Self-Service

Controlled

INTEG STAGING

From Steve Brodie and Rohit Jainendran

The big question: How to test?We release software in batches so that we can test it.

That is the whole reason for doing it. We test software “release candidates” to make sure everything works together.

In continuous delivery, we might get as little at 10 minutes to test a release candidate

If we can’t test the complete version, how do we guarantee quality? Is it magic?

Test Layering – A Golden TicketMonitor your released software: Errors, Usage volume, usage patterns, user feedback

QA System with Human test consultants

Code review: Both a manual test, and a place to ask for test scripts.

Continuous integration: Run automated tests before using human review time

Unit tests in the development environment

Switch new features and architectureStart h

ere to

add layers

Start here to

release a change

Go Both Ways

Veloc i ty

Qu

alit

y

Increase Quality

Increase Velocity

Switch and Unveil

HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.

Test Story Owner and testers see the change on test systems.

BetaInsiders see it and use it. Story Owner can show it to users for feedback or A/B testing.

UNVEIL! The big event. Communicate with all users. Measure reaction.

One code version

No special test builds

No long-running branches

Code and Test Patterns

Code and Test PatternsCentralized

Traditional process with no branches finds and fixes problems as early as possible

DistributedRelease every change with its own branch and test

Temporary branchesEfficient review, combines benefits of centralized and

distributedMAXOS

Use centralized continuous integration to manage a massively scalable IT system

Centralized CI/CDContributor Commits – “as early as possible” to find problems

ContinuousIntegration tests

Fail - alarm

Release CandidateTest System Release

QA Testing

Pass

Distributed Continuous DeliveryContributor Commits

Branchor Fork

Deployed version

Peer review merge requests

Other contributionsmerged and released“as late as possible”

QA ConsultsPass FinalAuto Test?

Merge backCurrent

Deploy

Distributed: Multiple Test SystemsContributor 1

Contributor 2

ProductionRevision

ReleaseAnytime

CI SystemQA Team

Test System 1

Test System 2

Assembla spins up test servers

Test and Review on Temp Branches

Contributor Commits

Auto tests

Mainline

Review Auto tests Review

• Github style – Pull requests or merge requests going to CI system• Gerrit style – Temporary branches for each contribution• Assembla git supports both – merge requests, and temp branches created

by a push to a protected branch.

Test before spending human time on review

The Services MegatrendDesktop Web App Cloud

Services

App

DB

Service

Service

Service

Matrix of Services - MAXOS

PrioritizedBacklog

CurrentWork

Each team releaseswhen ready

Service teamIntegration

test envProduction

service

Service teamIntegration

test envProduction

service

Service teamIntegration

test envProduction

service

Feedback on speed, errors, usage, and requests

Organize bytesting as one system

SAFe – Copyright Dean Leffingwell

See the diagram on the SAFe Site:http://scaledagileframework.com/

Looks like a good practice, but really,nobody in Silicon Valley would have allof those meetings

Coordinate without big meetings

PrioritizedBacklog

CurrentWork

Service teamIntegration

test env

Service teamIntegration

test env

Service teamIntegration

test env

Continuous Integration between latest dev version of each service

• Continuous integration helps teams coordinate.

• See dependencies between “producers” and “consumers”

• Errors and conflicts show related team contact info

• Meetings and changes negotiated between two teams, not many

Roles and Migration

Role: DeveloperDevelopers have more power and responsibility.Developers have more responsibility for testing.Developers (not QA or PM) decide when to release. This

is a strong finding.Incentives are correct. Developer might have to come back

from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.

Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!

Role: QAQA is a consultant.QA gets more respect. Developers have to ASK for

service.Developers do more of the testing work. They should

organize reviews and automated tests so that bugs don’t go through into the manual test process.

QA has more time to investigate usabilityQA monitors productivity and quality metrics

Incentives for Continuous FlowYou don’t need culture change. You only need to release more frequently1. If you do code review, you can get automated. No

browbeating and cajoling is required. Developers will ask for tests when they review.

2. If developers decide to release, they will take more responsibility for testing and automated tests.

3. If you release more frequently, developers will quickly learn not to break the build

4. If PM’s take unveil and measurement responsibility, they will make better features

Our Master Plan1. Release more frequently2. Improve

www.continuousagile.com/unblock