Make continuous integration scale again - Abderrazak Bouadma aux Quarter Backs SFEIR

1

Make Continuous IntegrationScale Again

Abderrazak BOUADMA

2

what is building softwares ?

3

code repository

build factory

reports

artifacts

Developers Data center

push

trigger store

deploy

4

reportspush

trigger store

deploy

5

what is continuous integration ?

6

reportspush

trigger store

deploy

7

so, what’s the problem ?

DOESN’T SCALE

9

reportspush

trigger store

deploy

10

reportspush

trigger store

deploy

11

12

so, how to solve it ?

13

push

trigger

deploy

master

slaves

pros

1. specialize each salve2. adding slaves permits to scale3. each team could manage its own slave(s)4. slaves are isolated

14

cons

1. Provisioning slaves still manual operation (could be automated)

2. it’s not an instant operation (VMs take time before being available)

3. A developer team hasn’t to bother about the build factory (not too much)

15

16

Using Docker Swarm and Ephemeral Containers ?

17

docker swarm cluster

registry

push code

spawn a container

deploy

canarytest

prod

publish

scale

pros

1. scales very well2. tailored for on-premises or the cloud3. slaves images could be specialized at will

18

cons

1. Need a dedicated team to manage the swarm cluster2. There’s not yet a managed Docker container on GCP3. On the cloud, the Docker cluster could cost a lot of

money.

19

20

Managed Container Orchestration ?

21

Kubernetes concepts

22

Pod Selector ReplicaSet

Volume Secret DaemonSet

Service Ingress StatefulSet

Deployment Liveness Probe ConfigMap

Cluster Readiness Probe

Label Job

23

+ +

24

How ?

Read the doc

https://cloud.google.com/solutions/configuring-jenkins-container-engine

https://cloud.google.com/solutions/continuous-delivery-jenkins-container-engine

https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes

https://cloud.google.com/solutions/jenkins-on-container-engine-tutorial

25

Jenkins Kubernetes Plugin

The aim of the Kubernetes plugin is to be able to use a Kubernetes cluster to dynamically provision a Jenkins agent (using Kubernetes scheduling mechanisms to optimize the loads), run a single build, then tear-down that slave.

26

27

Why ?

pros

1. Google takes care of the scalability (almost)2. Kubernetes has already needed concepts to create a

Jenkins cluster3. Native Cluster isolation (namespaces)4. virtually unlimited (Container Auto Scaler

https://cloud.google.com/container-engine/docs/resize-cluster )

28

Links

https://www.martinfowler.com/articles/continuousIntegration.html

https://www.slideshare.net/Docker/build-publish-deploy-and-test-docker-images-and-containers-with-jenkins-workflow

https://engineering.riotgames.com/news/jenkins-ephemeral-docker-tutorial

https://goto.docker.com/continuous-integration-pipeline-ty.html

http://glaforge.appspot.com/article/scale-jenkins-with-kubernetes-on-google-container-engine

29

30

Questions ?

Demo

Coming soon :1. Jenkins/Kubernetes/GCP 2. Jenkins/Nomad/GCP3. (vault, google iam, google Logging)

31

Next session October 3th 2017

1. Retrofit 2. Semantic Web3. Kotlin4. Java Mission Control

32

Next

1. quarterbacks.io (cfp, slides, reports, videos …)2. @quarterbacks_io3. Quarter Backs Tour

33