Enabling Microservices @Orbitz - DevOpsDays Chicago 2015

#devopsdaysChicago 2015

Enabling Microservices

@Orbitz

Steve HoffmanSenior Principal Engineer

@bacoboy



Outline

• Brief Architecture Overview/History

• From Monolithic to Services to Microservices/Docker

• Automated Pipelines

• Questions



2000


2000

Yes, we are STILL hiring!


2000

Yes, we are STILL hiring!@OrbitzTalent


2000

Web Layer

Business Layer


2000


2000


2000

Business Layer


2000

Business Layer


Business Layer

2000


Business Layer

2003


Business Layer

2003


Business Layer

2003


2003


2004


2004


2004


2012


2012


2012• Multiple Brands

• Websites

• Webservices

• Multiple Backends

• 500+ apps / thousands of instances

• Deployments Daily (sometimes more)


2015• Multiple Brands

• Websites

• Webservices

• Multiple Backends

• 500+ apps / thousands of instances

• Deployments Daily (sometimes more)


Process Overkill


Different Provisioning Tools

DEV OPS


Different Provisioning Tools

DEV OPS

Application Platform!=


A New Experiment• Microservices?

• Decompose single “service” into the actual 40+ sub-services

• Any change to sub-service was a deployment of many

• Could it be a simple Spring Boot App in Docker (12 factor?)

• Backward compatible with existing service infrastructure.

• Code to Production w/o help from other Humans - Minutes not Days



Module

Module

Module Module


OrbitzContent

Orchestration Service

Content (Solr)

Editorial Module

Search Module

Hotel Module


OrbitzContent


Content (Solr)

Editorial Module

Search Module

Hotel Module

Hotel Team Search Team Content Team


OrbitzContent


Content (Solr)

Editorial Module

Search Module

Hotel Module


OrbitzContent


Content (Solr)

Editorial Module

Search Module

Hotel Module

Hotel Team

Search Team

Content Team


Pre-Continuous Delivery

1.2-‐BETA-‐20150401-‐113002

Test Discard

Test Discard1.2-‐BETA-‐20150402-‐093002

Build

Build

And so on…


Pre-Continuous Delivery

1.2-‐BETA-‐20150401-‐113002

Test Discard

Test Discard1.2-‐BETA-‐20150402-‐093002

Build

Build

Test Deploy1.2

Build


App

App

App


App

App

App


AppApp

App

xN


AppApp

App

xN


Continuous Delivery with Jenkins, Docker, Ansible

and Marathon


Committer

Pull Request


Committer

Reviewer

Pull Request


Committer

Reviewer

Pull Request


Committer

Reviewer

Pull Request


Jenkins Pipeline


if(…) { x ++; }

Merge


build unit test publish

deploy dev

acceptance tests

deploy qa

deploy staging open RFC deploy

prod close RFC

Pipeline (Simplified)


if(…) { x ++; }

1.2

MergeBuild


if(…) { x ++; }

1.2

Merge

1.2.

editorial-module.jar

./gradlew build

{BUILD_NUMBER}

Build


1.2

Merge


if(…) { x ++; }

171.2.

Build


1.2

Merge


FROM orbitz/java-8 ADD build/editorial-module.jar /opt/orbitz CWD /opt/orbitz CMD java -jar editorial-module.jar

if(…) { x ++; }

171.2.

Build


1.2

Merge

editorial-module.jar orbitz/editorial-module:1.2.17

if(…) { x ++; }

171.2.

Build


orbitz/editorial-module:1.2.17

Build


Master

MasterSlave

Commit/Push or Pull Request/Merge


Master

MasterSlave

Poll or Push Trigger



Master

Master

Slave

Slave




Master

Master

Slave

Slave




Master

Master

Slave

Slave



Clone & Build Push Artifacts


Master

MasterSlave


Jenkins on Docker on MesosSlides: http://bit.ly/oww-mesoscon2015-slides

Video: Coming to YouTube

http://bit.ly/oww-mesoscon2015-slides


git pull

playbook.yml (abridged)

- hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3

Deploy


- tasks: marathon: …

playbook.yml1.2.16

1.2.16

1.2.16

Deploy



1.2.16

1.2.16

1.2.16

Deploy



1.2.16

1.2.16

1.2.16

Deploy



PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }

1.2.16

1.2.16

1.2.16

Deploy


Deploy



1.2.16

1.2.16

1.2.16

app = GET /v2/apps/editorial-module

if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if

while GET /v2/deployments contains deploy_id // still deploying end

// deploy complete




1.2.16

1.2.16

1.2.16

Deploy



1.2.16

1.2.16

1.2.16

Deploy



1.2.16

1.2.16

1.2.16

1.2.17

1.2.17

1.2.17

Deploy



/health

1.2.16

1.2.16

1.2.16

1.2.17

1.2.17

1.2.17

Deploy



/health

200 OK

200 OK

200 OK

1.2.16

1.2.16

1.2.16

1.2.17

1.2.17

1.2.17

Deploy



1.2.17

1.2.17

1.2.17

Deploy


And off to the next environment…

1.2.17

1.2.17

1.2.17

Deploy


1.2.17

1.2.17

1.2.17

What if?


1.2.17

1.2.17


1.2.17

1.2.17


1.2.17 1.2.17

1.2.17


/health

200 OK

1.2.17 1.2.17

1.2.17


1.2.17 1.2.17

1.2.17


Smoke/Acceptance Testing

1.2.17

1.2.17

1.2.17


Paper Trail

create

tickets

commit(s)


Paper Trail

fail!

close/fail


Paper Trail

ok

close


Build Unit Test

Deploy Dev

Deploy Prod

Deploy Staging

Acceptance TestCode

Review & Push

Production

Pre-Production

Open RFC

Close RFC


“Why didn’t you use {{.x}} ?”


Keeping an eye on…

Hashicorp Vault Rancher Convoy

Kubernetes Amazon Elastic Container

Service

Docker Swarm + Machine


Summary• Create a platform for docker deployments using shared and app-specific “localhost” helpers

— this was ours, yours SHOULD look different — adapt to change, don’t fight it.

• People don’t scale, use automation

• Docker - repeatable applications (environment portable apps)

• Chef - repeatable infrastructure (environment aware services)

• Jenkins - repeatable releases (pipelines)

• Delineate configuration concerns:

• Known at Compile time — Bake into Docker image

• Known at Boot time — Bake into Playbook/Launcher - parameter to Docker run

• Changes Anytime — Externalize (consul K/V, etcd, zookeeper)


Questions?

@bacoboy

Enabling Microservices @Orbitz - DevOpsDays Chicago 2015

Technology