MAXOS - the new Scalable Continuous Agile

MAXOS Scalable Continuous Agile

with debatable points

From Andy Singleton, http://continuousagile.com

www.assembla.com

Microsoft Journey

• Scale: Releases were buggy at scale• Stress: Releases were stressful• Online: SaaS, devices and services, cloud first• Competition: The competition was releasing

more frequently and moving faster

• Now, about 80% of the way to “second generation agile”

Survey on Continuous Delivery

46% think their competitors

have adopted Continuous

Delivery

Ways to Scale

Scrum + SAFe• Add more hierarchy• Hold big meetings

and teleconferences• Block everyone into

one cadence• Coordinate big

releases

Top Tech CompaniesAutomate management,

as well as testing and deployment.

Communicate peer to peerUnblock! teams to move

as fast a possibleRelease early and often.

Separate release from launch

2 Ways to Be More Productive

1) Do the right thing

Users ignore at least 50% of the new stuff you try. If you can measure usage or value and figure out what to ignore in development, you can increase development productivity by 100% for zero extra cost.

Measurement is very important.

More frequent releases = more measurements

2) Use more machines

Better organization is not high on the list of ways to get more productive. Does not change much over centuries.

Are the methodology booths in the exhibit hall are being pushed out by vendors that make build, test, and deploy tools?

“Technical practices”

Productivity comes from more/bigger machines

Continuous Delivery Basics

• Code contribution patterns• Test layering• Automation• Developer responsibility• Feature switches

Code Contribution Patterns

Manage code if possible. People are hard to manage and can’t be automated. They want to contribute.• Centralized continuous delivery

– No branches, finds and fixes problems as early as possible• Distributed continuous delivery

– Release every change with its own branch and test• Temporary branches

– Combines benefits of centralized and distributed• MAXOS

– Use centralized continuous integration to manage a massively scalable IT system

Centralized CI/CDContributor Commits – “as early as possible” to find problems

ContinuousIntegration tests

Fail - alarm

Release CandidateTest System Release

QA Testing

Pass

Continuous delivery at Edmunds.com

Distributed Continuous DeliveryContributor Commits

Branchor Fork

Deployed version

Peer review merge requests

Other contributionsmerged and released“as late as possible”

QA Consults

Pass FinalAuto Test?

Merge backCurrent

Deploy

Distributed: Multiple Test Systems

Contributor 1

Contributor 2

ProductionRevision

ReleaseAnytime

CI SystemQA Team

Test System 1

Test System 2

Assembla spins up test servers

MAXOS Service Team

Test Layering

Monitor your released software: Errors, Usage volume, usage patterns, user feedback

QA System with Human test consultants

Code review: Both a manual test, and a place to ask for test scripts.

Continuous integration: Run automated tests before using human review time

Unit tests in the development environment

Switch new features and architectureStart h

ere to

add layers

Start here to

release a change

More frequent releases can increase quality

9 (sparse) Layers at Edmunds

Feature Switch and Unveil

HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.

Test Story Owner and testers see the change on test systems.

BetaInsiders see it and use it. Story Owner can show it to selected users for feedback or A/B testing.

UNVEIL! The big event. Communicate with all users. Measure reaction.

One code version

No special test builds

No long-running branches

Go Both Ways

Veloc i ty

Qu

alit

y

Increase Quality (more layers, longer beta)

Increase Velocity (less layers, faster unveil)

Role: Developer

• Developers have more power and responsibility.• Developers have more responsibility for testing.• Developers (not QA or PM) decide when to release.

This is a strong finding.• Incentives are correct. Developer might have to

come back from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.

• Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!

Programmers approve releases, not QA

Person, Dog, or Bot writes codeComputers will write code

Continuous Agile

Lean process

1. Release more frequently2. Improve

Role: Product Manager/Owner

• Batch -> Continuous

• Requirements -> User Experience

• Strategy -> Measurement– Usage measurements are so important, so

underutilized– Double your productivity

Product Owner -> Story Owner

Program Launch

Releases

Plan Program Test Doc Deploy

SkipAutomate& Blend Lag Automate

Pull

CI & CD

End up with

Separate Release from Launch

Measure

Matrix of ServicesBreaking the scale barrier

The Services MegatrendDesktop Web App Cloud

Services

App

DB

Service

Service

Service

Scale it like Google

• 15,000 developers, 5,000 projects, one current version of the code (2013). They can go from an idea to a release in 48 hours

• Vast Internet system divided into thousands of "services"

• Most programming done by teams of 3-4

• Centralized process with single version of the test system – run 100 million test cases daily

• Before developers release a change, they test it with the most recent version of all the other services. If a test script finds conflicts, it tells developers who to contact to resolve them

Matrix of Services - MAXOS

PrioritizedBacklog

CurrentWork

Each team releases

when ready

Hundredsof releases

per day

Service team Productionservice

Service team Productionservice

Service team Productionservice

Feedback on speed, errors, usage, and requests

Test as one system

Integrationtest env

Integrationtest env

Integrationtest env

Coordinate without big meetingsContinuous Integration between latest dev version of each service

• Continuous integration helps teams coordinate.

• See dependencies between “producers” and “consumers”

• Errors and conflicts show related team contact info

• Meetings and changes negotiated between two teams, not many

PrioritizedBacklog

CurrentWork

Service team

Service team

Service team

Integrationtest env

Integrationtest env

Integrationtest env

Machines can replace layers of management

Teams are largely self-managing

PrioritizedBacklog

CurrentWork

Service team Integrationtest env

Up to 50% of workfrom backlog

At least 50% of work is self-plannedProblems get fixed quickly

Productionservice

Productionservice

Productionservice

Feedback: quality, reliability, speed, user support

Productionservice

ProductionServer

Sense, respond, self manageminimize planning

Hubspot – Great at Mid-scale

• Transformed a monolithic app to 200 services over one year

• 3-person programming teams. Each of 20 teams is responsible for about 10 services

• Dev teams responsible for design, programming, testing, release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.

• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services

• Feedback from customer support also grouped by team

• Transformed a monolithic app to 200 services over one year

• 3-person programming teams. Each of 20 teams is responsible for about 10 services

• Dev teams responsible for design, programming, testing, release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.

• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services

• Feedback from customer support also grouped by team

Scaling

PrioritizedBacklog

CurrentWork

Each team releases

when ready

Service team Productionservice

Service team Productionservice

Add capacity fast by building aroundSingle-function programmer/tech leads

Integrationtest env

Integrationtest env

Teams are not permanently multifunctional

Core ITannual budget

Reliability & security mission

Web API LayerMobile SaaS / Cloud

Marketing

Fast ITmonthly budget

Mission to respond to opportunities

Service team Productionservice

Service team Productionservice

Productionservice

Integrationtest env

Integrationtest env

Integrationtest env

Fast IT (continuous)

Core IT (stable service)

Service team Productionservice

Service team Productionservice

Productionservice

Integrationtest env

Integrationtest env

Integrationtest env

Different culture, different company, or no company

United by Testbeds

Culture does not needto be consistent

Scaling Methods

Brooks and Mythical Man Month– Hypothesis: Scaling

problems comes from n^2 communication explosion

– Solution: Cells and hierarchy to contain communication

Internet reality– Scaling problems come

from dependencies– Solution: more sharing,

more communicationMythical Man Month is wrong40 years of awesome insights, but

SAFe (Copyright Dean Leffingwell)

Ways to Scale

Scrum + SAFe• Add more hierarchy• Complex multifunction

teams• Hold big meetings and

teleconferences• Block everyone into

one cadence• Coordinate big releases

Top Tech CompaniesAutomate management,

as well as testing and deployment.

Dev-lead teamsCommunicate peer to

peerUnblock! teams to move

as fast a possibleRelease more frequently

Competing with MAXOSThe secret weapon that Silicon Valley is using to

disrupt and destroy competitors

• Retailer X deploys changes to their monolithic online ordering app once every six weeks. Ops holds for three weeks to make sure the complete system is stable.

• Amazon has thousands of services and more than 1000 service teams. They release something about once every 11.6 seconds. In the time that Retailer X takes to try one new release, Amazon has made 100,000 changes.

• Amazon hosting competitor: “It’s an emergency”.

SAFe and MAXOS

Debatable Points• More frequent releases can increase quality• Culture does not need to be consistent• Teams are not permanently multifunctional• Programmers approve releases, not QA• Productivity comes from bigger/more machines• Machines will write code• Sense, respond, self-manage, minimize planning• Mythical Man Month is wrong• Machines and continuous integration can replace layers of

management• Scrum and SAFe cannot compete with MAXOS