Beyond CD - JAXLondon · @michieltcs. REDUCE TOIL. the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring

BEYOND CD LEARN, ADAPT & IMPROVEMICHIEL ROOK@MICHIELTCS

BACKGROUND

@michieltcs

LARGE SEMI-GOVIN NL

@michieltcs

MYSQL ELASTIC

INTERNET

@michieltcs

TOPIC TEAMS

@michieltcs

ON-CALL

@michieltcs

FIRE TEAM

@michieltcs

PACKAGE/DEP UPDATES

@michieltcs

BUG TRIAGE

@michieltcs

OFFICE HOURS INCIDENT RESPONSE

@michieltcs

RELEASE ROLLOUT

@michieltcs

A TYPICAL RELEASE...

@michieltcs

RELEASE CHECKLIST

@michieltcs@michieltcs

Create release

Test release

Deploy release

every2 weeks

Thursday(sprint end)

Monday (sprint start)

Tuesday

@michieltcs

1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD

@michieltcs

2-3 DAYSMANUAL WORK

@michieltcs

... HOT FIXES?

@michieltcs

GOALS

@michieltcs

REDUCE COST

@michieltcs

FAST FEEDBACK

@michieltcs

REDUCE TOIL

@michieltcs

REDUCE TOILthe kind of work tied to running a production service that

tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a

service grows

@michieltcs

PUT PO/BIZ BACK IN DRIVER'S SEAT

@michieltcs

RESPONSIVE ORGANIZATION

@michieltcs

WHY CD THEN?

@michieltcs

REDUCE RISK

@michieltcs

REDUCE WASTE

@michieltcs

REDUCE COST

@michieltcs

IF IT HURTSDO IT MORE OFTEN

@michieltcs

BIG STEPS

@michieltcs

FAIL BIG

@michieltcs

SMALL STEPS

@michieltcs

FAIL SMALL

@michieltcs

$ = REALIZED VALUE

CREDITS TO @FGOULDING


HIGH VS. LOW PERFORMING TEAMS

Source: Accelerate - The Science Of DevOps

APPROACH

@michieltcs

3 PHASES

@michieltcs

DEFER SPLITTING THE MONOLITH

@michieltcs

INCREASE TEMPO & AUTOMATION

@michieltcs

SINGLE PIPELINE

@michieltcs

IMPROVEMENTS & TESTING

PHASE 1: INCREASING

RELEASE CADENCE

@michieltcs

WEEKLY RELEASE

@michieltcs

AUTOMATION


Build / Package

AutomatedTests

AcceptanceEnvironment

ProductionEnvironment


Build / Package

AutomatedTests



@michieltcs

pipeline

{

stages

{

stag

e('Run t

ests') {

sh

"gradle

check"

}

stage('B

uild doc

ker imag

e') {

sh

"docker

build -

t jobser

vice:${e

nv.BUILD

_NUMBER}

."

sh

"docker

push jo

bservice

:${env.B

UILD_NUM

BER}"

}

stage('D

eploy st

aging')

{

sh

"ansibl

e-playbo

ok -e BU

ILD=${en

v.BUILD_

NUMBER}

-i stagi

ng deplo

y.yml"

}

stage('D

eploy pr

oduction

') {

sh

"ansibl

e-playbo

ok -e BU

ILD=${en

v.BUILD_

NUMBER}

-i prod

deploy.y

ml"

}

}

}

PIPELINE AS CODE

image: registry.local/runner:latest

stages: - test - images - deploy

phpunit: stage: test script: - bin/phpunit

images: stage: images

script: - make images publish

only: - master deploy: stage: deploy

script: - make deploy

only: - master

@michieltcs

WAY OF WORKING

@michieltcs

PAIRPROGRAMMING

BOY SCOUT RULE

@michieltcs

NO MORE RELEASE BRANCHES

PHASE 2: SINGLE PIPELINE


Build / Package

AutomatedTests



@michieltcs

CONTINUOUS DEPLOYMENT

@michieltcs

ROLLING DEPLOYS


LOAD BALANCER

ROLLING UPDATE

SERVICE V1.0 SERVICE V1.0SERVICE V1.0

@michieltcs


LOAD BALANCER

ROLLING UPDATE


SERVICE V1.1

@michieltcs


LOAD BALANCER

ROLLING UPDATE


SERVICE V1.0

@michieltcs


LOAD BALANCER

ROLLING UPDATE


SERVICE V1.1

@michieltcs


LOAD BALANCER

ROLLING UPDATE


SERVICE V1.0

@michieltcs


LOAD BALANCER

ROLLING UPDATE


SERVICE V1.1

@michieltcs


LOAD BALANCER

ROLLING UPDATE


SERVICE V1.0

@michieltcs


LOAD BALANCER

ROLLING UPDATE


@michieltcs




@michieltcs

PIPELINE FAILURES

@michieltcs

FLAKY TESTS

@michieltcs

TIMEOUTS

@michieltcs

EXTERNAL DEPS IN TESTS

@michieltcs

NETWORK STABILITY ISSUES

@michieltcs

BRITTLE INFRASTRUCTURE

@michieltcs

IF YOU CAN'T TRUST IT - REMOVE IT

@michieltcs

EXTREME FEEDBACK

STOP THE LINE


PHASE 3: IMPROVEMENTS &

TESTING

@michieltcs

PIPELINE SPEED

@michieltcs

BUY A BIGGER BOX

@michieltcs

CLEVER PIPELINE IMPROVEMENTS:

PARALLELIZE

@michieltcs

TESTING IMPROVEMENTS


UNITTESTS

E2E /VISUAL TESTS

INTEGRATIONTESTS

LOTS OF MANUAL TESTING

E2E TESTS

@michieltcs

AUTOMATION


CONTRACT TESTING


UI TESTING


VISUAL TESTING


UNIT TESTS

UI /VISUAL TESTS

Exploratorytesting & user

feedback

Monitoring& alerting

INTEGRATION / CONTRACTTESTS

COST SPEED

@michieltcs

RELEASEAPPROVAL?

FEATURE TOGGLES

@michieltcs

DECOUPLEDEPLOYMENTS

FROMRELEASES



@michieltcs

CAUTION:FEATURE TOGGLE

DEBT

RESULTS & TAKEAWAYS

@michieltcs

LEARNINGS

@michieltcs

PIPELINE SPEED & STABILITY

@michieltcs

WAY OF WORKING & MINDSET

@michieltcs

VISUAL TESTING TOOLS

@michieltcs

(AUTOMATED) SECURITY TESTING

@michieltcs

SOME STATS

@michieltcs

~4000 TESTS PER DEPLOYMENT

@michieltcs

~10 MINUTES PERPIPELINE RUN

@michieltcs

~1.5 HOURS FROM DEV TO PROD

@michieltcs

IN 6 MONTHS:MORE THAN 5000

DEPLOYMENTS


THANK YOU FOR LISTENING! @michieltcs / [email protected]

www.michielrook.nl

http://www.michielrook.nl

Beyond CD - JAXLondon · @michieltcs. REDUCE TOIL. the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring

Documents