Into the cloud

Into the Cloud

Making our Continuous Delivery fly.

Tomas Riha

Architect @ VGT/WirelessCar

Passionate about creativity, change and improvement

Horrible at following instructions and performing repetitive tasks

MAJOR Project Liability

mail: [email protected]

twitter: @TomasRihaSE

blog: continuous-delivery-and-more.blogspot.com

Four Years ago

New Telematics Delivery Platform

Sent us on road to

Micro Services

Continuous Delivery

Infrastructure at that time

Low level of virtualization

“Two servers and a Oracle DB” per delivery

Split Operations and Development

Infrastructure and Application Architecture separated

Long process to get a server

Micro Service Architecture

Reality

Continuous Delivery

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Assemble Release Deploy Test

Service Pipes

100s of tests run in parallel testing

Solution Pipes

10s Use Case Requirements Verified

Here is your test server

Build Release Deploy Test

DB

Test

Server

Static virtual instances

Master Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Capacity Planning & Peaks

Master Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

?

Pampering the Snowflakes

Each static server needs maintenance.

Server Maintenance one of the top task the team does

Each server provisioned by different version of our chef scripts, where baseline

is a moving target

Continuous Delivery as a Service

Ok so this Continuous Delivery thing is nice....

....what does it take to deliver it to the entire organization as a Service?

400+ developers

Continuous Delivery as a Service

Blue skies, Sunshine

and ohh yes a CLOUD PROVIDER!!

Amazon Web Services

Load Tests

We have used AWS a bit for Load Tests

Go!

Continuous Delivery in the AWS Cloud

Demo Environments in the AWS Cloud

Build in Cloud host on premises and at partner

AWS

Volvo

Hosting

Partner

Hosting

Partner

Compatibility in

Environments!

What we got

Automated provisioning on our Continuous Delivery infrastructure using chef

Automated provisioning of our Jenkins jobs into build pipes

Rudimentary Image Baking Capabilities from our AWS Load Tests

Basic understanding of how our services behave in a cloud environment

What we need to do

Build a Virtual Private Cloud with networks

Connect VPC to Volvo Network

Migrate about

~800 Jenkins jobs

60 Jenkins master/slaves

1 Nexus server, 2 test databases, 1 Sonar, 1 Elk

2 custom applications, 1 MongoDB

Our approach

Change as little as possible during migration

Automate everything

Establish responsibility between Operations and Development

Responsibilities

Volvo

Network Team

Delivery Engine Team

Defining Application Topology

Load Balancer

Server Instance Server Instance

Data Storage

In Amazon Web Services

● Elastic Load Balancer

○ Security Groups

● Auto Scaling Group

○ Launch Configuration

○ Amazon Machine Image

○ Security Groups

● Data Storages

○ Simple Storage Service

○ Relational Data Service

■ Oracle

■ Postgres

○ Custom Databases

■ MongoDB

○ Security Groups

Defining Application Environment

Load Balancer

Server Instance Server Instance

Data Storage

In Amazon Web Services

● Virtual Private Cloud

○ Subnets

● Auto Scaling Group

○ Scaling Rules

■ Up/Down

■ Max/min

○ Instance Size

■ Mem/Cpu

● Data Storages

○ Size

■ Mem/Cpu/Storage

Infrastructure as Code

Dev TestLoad

TestUAT

INT

Test

Provisioning Interface

Vagrant Impl Cloud Provider Impl PDF Document Impl

Topology

Spec Env Spec

Legacy

PROD

Environment Definition

Networking

Loadbalancing

Servers

Databases

Location

Integration

Scaling

Infrastructure as Code

TestLoad

Test

Provisioning Interface

AWS Cloudformation Impl

Topology

Spec Env Spec

Demo

Delivery

Engine

Tool

Baking Machine Images

Git Repo Base

Packer

Scripts

Bakery

Instance

Git Repo Delta

Packer

Scripts

AMI Alpha

RHEL

AMI

Bakery

Instance

AMI Delta

Baking Machine Images

RHEL

AMI

VGT Base

Java Base Jenkins Base Nexus Graphite ELK

Jenkins Master Jenkins SlaveApplications

Applications

Applications

Chef Issues

Never share same scripts for baking image that go from known

state to a delta with scripts that manage snowflakes!

Phase 1

Politics, Cross Organizational handovers & PoC

Way too much time

Phase 2

Build & Automate Setup

Build & Tear Up the VPC a few 100 times

1.5 months ~ 2 engineers

Phase 3

Migration

Start moving over the repositories.

Add .delivery-engine.yml files to repositories

Disable build pipes in old environment

1 month ~ 3 engineers

The Migration

Daily Cost tracking during migration!!

Did we do something really stupid yesterday?

Bosses love you when you report cost and think about optimizing cost

Cost Tracking

Prepare

Automate

Verify

Rip The Bandaid!!!

Lessons Learnt

Since the migration started we have not launched

one single instance through the AWS Console.

Console only used to terminate instances, never to

create instances.

Done well

Delivery Engine after Migration

More or less the same setup

Different distribution of jobs

Auto Scaling Workers

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker WorkerWorker Worker Worker

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

WorkerWorker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Immediate Benefits

Maintenance down at least 80% at least due to worker recycling

No more capacity planning

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker WorkerWorker Worker Worker

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

WorkerWorker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Not High Availability

Setup is still stateful

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker WorkerWorker Worker Worker

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

WorkerWorker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

250 Git repositories building, testing, releasing

Components, Applications, Solutions

For closer to 150 developers

Usage

Growth during migration wasn't expected but was about 50 repos

Over 100 Git repos added after migration

New teams constantly onboarding to Delivery Engine

Expecting 200 plus developers by end of year

Growth

Grow number of users and content not the size of our Team Size!

We do it through a

Event Driven, Stateless, Application developed using ALL the

Continuous Delivery principles.

Using Continuous Delivery to Deliver Continuous Delivery.

We need to ensure scalability and availability

Stateless Build Environment

Pipes as Code

Unit test pipe task

Integration test pipes

JobPipe as

CodeClient

Stateless Build Environment

Build Data and reports away from Build Environment

JobPipe as

CodeClient

Stateless Build Environment

No Dependency between GIT Repository and Build Job

Only one job exists

JobPipe as

CodeClient

Queue

Gerrit Listener

Job

Stateless Build Environment

Aws Tools

Pipe as

CodeClient

Test Environment

Queue

Gerrit Listener

Split Test Environment from Build Environment

HA Build environment

Test Environment

Rabbit MQ Cluster

Coordinator Jenkins

Factory

ForemanJenkins

WorkersJenkins SlavesWorkers

Jenkins SlavesWorkersJenkins Slaves

Gerrit Factory

ForemanJenkins

WorkersJenkins SlavesWorkers

Jenkins SlavesWorkersJenkins Slaves

Test Environment

Test Environment

Test Environment

ELK Graphite

S3Master

Mind

Custom DeliveryEngine GUI Slack

From idea to rolled out in about 2 months, 1 man month of work

Without AWS not possible

HA Rewrite

+

Lead time for Infrastructural change drastically reduced

Ability to build HA and scalability on a level previously not possible

No more capacity planning

Demo Environment using same tooling and fully up and running

QA and Integration testing environments coming up as well

DevOps

Amazon Web Services works GREAT!

-

Different tooling for creating aws infrastructure to creating in house infrastructure

Retrospective

Questions?

here: @TheBar

mail: [email protected]

twitter: @TomasRihaSE

blog: continuous-delivery-and-more.blogspot.com

Thats it!