The road to continuous deployment (PHPCon Poland 2016)

THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY

MICHIEL ROOK - @MICHIELTCS

▸ Java, PHP & Scala developer

▸ Consultant, trainer, speaker

▸ make.io

▸ Dutch Web Alliance

▸ Maintainer of Phing

▸ @michieltcs

THIS TALK

▸ Background

▸ The approach

▸ Process / standards

▸ Build pipelines

▸ Results & lessons learned

THE SYSTEM - SAN DIEGO

▸ ... or the Big Ball Of Mud

▸ Large legacy monolith

▸ Generates significant income

▸ Slow & complex

▸ Technical debt

SAN DIEGO FRONTEND

MYSQL DB

SAN DIEGO BACKEND

LOAD BALANCERS / VARNISH

ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK

SAN DIEGO FRONTEND

SAN DIEGO FRONTEND

SAN DIEGO FRONTEND

SAN DIEGO BACKEND

SAN DIEGO BACKEND

SAN DIEGO BACKEND

MEMCACHE FTP EXT. SERVICES

SOLR

THE SYSTEM - SAN DIEGO

▸ Infrequent, manual releases

▸ Fragile tests

▸ Low velocity

▸ Frequent outages / bugs / issues

▸ Frustrated team

▸ Low confidence modifying existing code

GOALS

▸ Reduce issues

▸ Reduce cycle time

▸ Increase productivity

▸ Increase motivation

REFACTOR? REBUILD?

APPROACH

▸ Strangler pattern

▸ Services per domain object (job, jobseeker, ...)

▸ Proxy to switch between old/new

▸ Migrate individual pages

ORIGINAL MONOLITH

PROXY

SERVICEORIGINAL MONOLITH

ORIGINAL MONOLITH

SERVICE SERVICE

SERVICE

PROXY

DB

DBDB

DB

DB DB

APPROACH

▸ Services behind load balancers

▸ Access legacy db’s

▸ Continuous deployment

▸ Docker containers

▸ Frontends are services

SAN DIEGO

ELASTIC SEARCHLEGACY

DB

JOB SERVICE

RMQ

ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK

MONGO DB

ITBANEN

JOBSEEKER SERVICE

NVBINTERMEDIAIR

PROCESS

STARTING OFF

▸ Scrum, 1 week sprints

▸ TDD / BDD

▸ Definition of Done

▸ Team mindset / experience

▸ Focus on value

▸ Replace old features with new (legacy becomes obsolete)

CONTINUOUS EVERYTHING

DEV BUILD / TEST

CONTINUOUS INTEGRATION

DEV BUILD / TEST ACCEPTANCE PRODUCTION

CONTINUOUS DELIVERY

DEV BUILD / TEST ACCEPTANCE PRODUCTION

CONTINUOUS DEPLOYMENT

WHY CONTINUOUS DEPLOYMENT

▸ Small steps

▸ Less overhead

▸ Early feedback

▸ Reduce cycle time

▸ Reduce risk

ONLY COMMIT TO MASTER

NO BRANCHES

NO BRANCHES

REALLY.

PAIR PROGRAMMING

BOY SCOUT RULE

QUALITY GATES

100% CODE COVERAGE *

FEATURE TOGGLES, A/B TESTS

DEVOPS

DASHBOARDS

BUILDPIPELINE

AUTOMATE REPEATABLE THINGS

EVERY COMMIT GOES TO PRODUCTION

DEFENSE IN DEPTH

UNIT TESTS

INTEGRATIONTESTS

ACCEPTANCETESTS

UI TESTS

DEFENSE IN DEPTH

UNIT TESTS

INTEGRATIONTESTS

ACCEPTANCE

UI TESTS

public function testJobCannotBeFound() { $jobRepository = $this->prophesize(JobRepository::class); $jobRepository->getById(EXPECTED_JOB_ID) ->shouldBeCalled() ->willReturn(false); $jobService = new JobService($jobRepository->reveal()); $this->assertFalse($jobService->getById(EXPECTED_JOB_ID)); }

DEFENSE IN DEPTH

UNIT TESTS

INTEGRATIONTESTS

ACCEPTANCETESTS

UI TESTS

public function testFindJob() { $expectedJob = $this->loadFixture('active_job.yml'); $actualJob = $this->repository->getById($expectedJob->getId()); self::assertInstanceOf(Job::class, $actualJob); self::assertEquals($expectedJob->getId(), $actualJob->getId()); }

DEFENSE IN DEPTH

UNIT TESTS

INTEGRATIONTESTS

ACCEPTANCETESTS

UI TESTSScenario: Link to related job Given a job exists And there are related jobs available When that job is viewed Then a list of related jobs is shown And each related job links to the detail page of the related job

DEFENSE IN DEPTH

UNIT TESTS

INTEGRATIONTESTS

ACCEPTANCETESTS

UI TESTS

DEFENSE IN DEPTH

MANUAL TESTING?

UNIT TESTS

INTEGRATIONTESTS

ACCEPTANCETESTS

UI

CONTINUOUS TESTING

UNIT TESTS

INTEGRATION TESTS

ACCEPTANCE TESTS

UI TESTS

SMOKETESTS

Cost Speed

Exploratorytesting Monitoring

PIPELINE AS CODE

node { stage 'Run tests' sh "phpunit" sh "behat" stage 'Build docker image' sh "phing build" sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" stage 'Deploy acceptance' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i acc deploy.yml" stage 'Deploy production' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i prod deploy.yml" }

DOCKERFILE

FROM php:7.0-apache

ADD vhost.conf /etc/apache2/sites-available/000-default.conf

ADD . /var/www/html

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

docker pull

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

docker run

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

wait_for: port=8080 delay=5 timeout=15

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

uri: url: http://localhost:8080/_health status_code: 200 timeout: 30

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg

service: name=haproxy state=reloaded

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg

service: name=haproxy state=reloaded

DEPLOYING

PULL IMAGE

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD NEW CONTAINER TO LB

REMOVE OLD CONTAINER FROM LB

STOP OLD CONTAINER

docker stop

docker rm

BUILD PIPELINE

FEEDBACK!

RESULTS

RESULTS

▸ Total build time per service < 10 minutes

▸ Significantly improved page load times

▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc)

▸ Increased confidence and velocity

▸ Experimented with new tech/stacks (angular, jvm, event sourcing)

▸ More fun

LESSONS LEARNED

▸ Team acceptance

▸ Change is hard

▸ Overhead of weekly sprint; requires discipline

▸ Docker orchestration

▸ Issues with traffic between Amazon <-> on-premise datacenter

▸ Javascript testing

LESSONS LEARNED

▸ Experience with new tech

▸ Stability of build pipelines

▸ Management/leadership buy-in

▸ Business alignment

▸ Not enough focus on replacing legacy application

QUESTION TIME

THANK YOU!

@michieltcs / [email protected]