Continuous delivery workflow with Docker

By Igor Moochnick

IgorShare [email protected]

Continuous Delivery workflow with Docker

a) What is Docker?b) None at allc) Developmentd) CI/CD infrastructuree) Testingf) Production

How are you using Docker?

Where do I come from?

Use case: Real-Time high-volume data processing system running on any cloud and on-prem physical machines

Data

pre

-pro

cess

ing:

cl

eanu

p/sa

nitiz

ation

InjectPush/pull

InjectPush/pull

InjectPush/pull

Parti

tioni

ng

Even

t Sou

rcin

g

Sorti

ng

l

l

l

l

l

l

l

l

l

l

l

l

l

l

Processing Graph

•NetflixOSS / SpringCloud•Spark•Cassandra•Kafka•ELK Stack•Mesos•Jenkins•Docker

Technology stack (some of the components…)

System Components in Production (multi-host)

l

l

l

l

l

l

Multi-host

Sandbox environments

ll l

l l

Longevity/Failure/Integration/etc…

•Everything:▫Applications▫Services. Ex: Nginx, Consul, Mesos, etc…▫Operational components. Ex: logstash, PuppetD, DataDog, NewRelic, etc…▫CI/CD infrastructure. Ex: Jenkins, Artifactory, Docker Registry, etc…▫And more …

What is going into containers?

Except …

•Everything else:▫Data▫Secrets (Ex: tokens, keys, certificates, etc…)▫Configuration/Settings

Unless versioned with the container▫Any other volatile piece of data that can change from environment to

environment

What is not going into containers?

Branching model for CD Deploy to Prod

Deploy to Prod

Main / trunk

brunch1

brunch2

Release process as artifact

https://github.com/constantcontact/jenkins_pipeline_builder

Jenkins Pipeline Builder

Jenkins Workflow Plugin (newcomer)

Anatomy of a Continuous Delivery Release process

Commit Acceptance ReleaseProduction

Environment

Code

Artifact

Artifacts

ConfigurationOrchestration

OrchestrationTrigger

Sandbox Environment

Commit + Acceptance Stage

version = 0.0.1

./gradlew build

gradle.properties

compile test

Dockerfile.template

Everything is a release candidate !!!

./gradlew e2e

mysvc-0.0.1.jar

./gradlew docker

mysvcdojo/mysvc:0.0.1

ADD {name}-{version}.jar

Orchestration+

health-checkTear-down

Commit stage Acceptance stage

Sandbox environment

•Make the contents of containers unified▫As repeatable as possible across different types of applications▫Same ports▫Same file locations (logs, traces, configs …)▫Unified way of applying metadata

Anatomy of a Dockerfile

OS+Security+Environment = service environment service environment + service = operational service

Dockerfile.template

Docker layering and inheritance hierarchy

102bb6bb1f36

ce70f38a21dd

d36afbf40e7e 4b74a8a8f2fb

7af2e2ece3a9

c2f61748d146

529b51a5f129

d6bd0ffc4755 FROM img:1.0

EXPOSE 8080

EXPOSE 8080

ADD app-0.3.jar app.jar

ENRYPOINT java –jar app.jar

ENRYPOINT java –jar app.jar

ADD app-0.1.jar app.jar ADD app-0.2.jar app.jar

•Environment variables•EXPOSE ports• Labels:

https://docs.docker.com/engine/userguide/labels-custom-metadata/

Metadata

Describing sandbox environments

https://github.com/IgorShare/docker_rack DockerRack

Why not Docker-compose•Dependency-graph-aware•Validation checks gates•Scriptable / template-driven

Dependency

Check gate

Script

•Most likely it’s not a single container• It’s consist of:▫Application container▫Sidecar▫ Log shipping▫Monitoring▫Etc…▫Sometimes:

DB Memcache Etc…

What is a containerized application?

Pod

Log management

Container 1

Container 2

Container 3

Container 4 Log Volume

Filebeat Logstash

ElasticSearch

Kibana

1) File-based logging

Container 1 Gelf

Logstash

ElasticSearch

Kibana

2) File-less logging

Container 2 Gelf

Container 3 Gelf

Container 4 Gelf

Worker Host Worker Host

Log Volume

Log ManagementCluster

Log ManagementCluster

•Mesos scheduling▫Marathon▫Chronos▫Custom scheduler

Failure and Recovery

by Adrian Cockcroft@adrianco

Micro-services Dojo

http://accordance.github.io/microservice-dojo/

Use it!Give feedback!Contributions arewelcome!

Thank you!@igor_moochnick

[email protected]

http://r44e.wordpress.com/