Docker: automation for the rest of us

1 / 61

Docker:automation forthe rest of us

2 / 61

OutlineWho I am

What I do

How Docker is helping

3 / 61

Who am I?

(And why am I here?)

4 / 61

Jérôme Petazzoni@jpetazzo

Tamer of Unicorns and Tinkerer Extraordinaire*

Grumpy French DevOps

Dislikes: repetitive tasks

Likes: shell scripts

"Go Away Or I Will Replace You Wiz Le Very Small Shell Script!"

* At least one of those is actually on my business card

5 / 61

What do I do?

6 / 61

I am tech support

7 / 61

I am tech support

8 / 61

I am tech support ... for a PaaS!Every day a new kind of fun!

Monday: Node.js

Tuesday: Python

Wednesday: Ruby

Thursday: Java

Friday: PHP

9 / 61

Lessons learnedLots of support requests are: "How do I do X with your product?"

Good documentation → fewer tech support requests

"Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime."

"Give someone good support and you help them today; write some docs and you help everybody forever."

Unless your business model relies on a product that youcan't install unless you have a PhD or a support contract

10 / 61

I am a technical writerWrote 60-80% of dotCloud documentation

Wrote 30-40% of Docker training materials

I ♥ to explain things

11 / 61

I am a technical writerWrote 60-80% of dotCloud documentation

Wrote 30-40% of Docker training materials

I ♥ to explain things

Documentation is part of your product

It's NOT the least important part

We're talking about technical products here. Don't brag because youwere able to place a phone call without reading the user manual.

12 / 61

I am a developer

13 / 61

I am a developer

14 / 61

@iamdevloper

15 / 61

I am a sysadmin

16 / 61

I am a sysadmin opsServer whisperer

Fixes other people's ʇᴉɥs

On-call

17 / 61

Public Service AdvisoryNever, ever be proud of being on-call

Never, ever be on-call for free

18 / 61

Public Service AdvisoryNever, ever be proud of being on-call

Never, ever be on-call for free

You're harming yourself

19 / 61

Public Service AdvisoryNever, ever be proud of being on-call

Never, ever be on-call for free

You're harming yourself

You're harming all of us

20 / 61

Public Service AdvisoryNever, ever be proud of being on-call

Never, ever be on-call for free

You're harming yourself

You're harming all of us

Make sure people are aware of the issues

21 / 61

I am an ImpostorImpostor syndrome, anyone?

Am I really qualified to do this?

What the F am I doing here?

Affects women more than men

This can lead to burn-out and sleep deprivation

This is more important than you think

22 / 61

What's Docker?

23 / 61

an open platform

to build, ship, and run

any app, anywhere

24 / 61

What can Dockerdo for me?

25 / 61

Some challenges with tech support"Works For Me"

"Can't reproduce"

"I would appreciate if you could test between 3 and 4am"

"So to trigger the bug you have to install X and Y thenconfigure A, B, and C, then download the extra file, put it inthis directory (which doesn't exists?!?) and then if yourestart three times in approximatively 5 minutes butsometimes it takes longer you will see that the images areshifted by a few pixels but if it doesn't work try to upgradeY to version Z and try all over again..."

26 / 61

Docker for tech supportGet a well-defined, reproducible environment

Define this environment in a Dockerfile

Build this Dockerfile into a container image

Run this container image anywhere

Same behavior, guaranteed (or your Open Source money back)

27 / 61

Dockerfile

FROM ubuntu:14.04RUN apt-get updateRUN apt-get install -y libx liby git wgetRUN git clone git://github.com/a/b/cRUN mkdir -p /extra/dirRUN wget -O /extra/dir http://extra/fileCMD start-service & sleep 60; \ stop-service; start-service; sleep 60; \ stop-service; start-service; sleep 60; \ start-service

$ docker build -t pixelbug ....$ docker run pixelbug...

Automate those repetitive, unreliable tasks yourself

28 / 61

Some challenges for tech writersWe use sphinx, showoff, LaTeX ...

"To see what the final documentation will look like, you need Ruby, 45 libraries, a custom fork of this project, and a philosopher's stone."

Insert scary anecdote about API documentations

Result:

tech writers can't see what they're doing

other team members won't/can't contribute to the docs

29 / 61

Docker for tech writersFROM ubuntu:14.04MAINTAINER Education Team at Docker <[email protected]>

RUN apt-get updateRUN apt-get install -y curl wget git ruby ruby-dev libxml2-dev libxslt-dev \ build-essential zlib1g-dev

RUN git clone https://github.com/puppetlabs/showoff.gitWORKDIR /showoffRUN gem build showoff.gemspecRUN gem install --no-rdoc --no-ri ./showoff-*.gem

# Let's install prince. The first dpkg will fail because of missing dependencies,# so we'll install the dependencies with "install -f" then try again.WORKDIR /usr/srcRUN wget http://www.princexml.com/download/prince_9.0-5_ubuntu14.04_amd64.debRUN dpkg -i prince_9.0-5_ubuntu14.04_amd64.deb || trueRUN apt-get install -fyRUN dpkg -i prince_9.0-5_ubuntu14.04_amd64.deb

COPY /slides/ /slides/WORKDIR /slides

CMD [ "showoff", "serve" ]

EXPOSE 9090

30 / 61

Docker for tech writersWith this Dockerfile, anyone can run our "doc pipelines"

Results are consistent, no more: "you forgot to install 字形 so the output is different!"

Less wasted time for onboarding, upgrades, reinstalls...

Automate those repetitive, difficult tasks yourself

31 / 61

Some challenges for developersSet up this Ruby + Postgres + Mongo + Cassandra stack

Make sure that all team members have the same env

Have consistent library versions between dev and prod

Basically, the same challenges as before, but worse

32 / 61

Docker for developersWrite Dockerfiles for each component

Put components together with Fig/Compose

Profit!

33 / 61

Describing a complex stackweb: build: src/front links: - redis - postgres:db - api - zookeeper:zk

redis: image: redis

postgres: image: postgres

api: build: src/backend links: - redis - postgres:db - cassandra - zookeeper:zk

zookeeper: image: jplock/zookeeper

cassandra: image: spotify/cassandra

34 / 61

OnboardingHire developer*

Give them a computer

Install Docker

git clone ...

docker-compose up ...

Your stack is up and running

*Actually the most difficult part.

35 / 61

Cold, hard dataHow long does it take for a developer to join a new project?

Before Docker: 2 days

After Docker: 2 hours

(Source: Worldline)

36 / 61

Docker vs Configuration ManagementQuick poll:

37 / 61

Docker vs Configuration ManagementQuick poll:

who is a dev?

38 / 61

Docker vs Configuration ManagementQuick poll:

who is a dev?

who uses Chef/Puppet/Salt/Ansible/...?

39 / 61

Docker vs Configuration ManagementQuick poll:

who is a dev?

who uses Chef/Puppet/Salt/Ansible/...?

who is happy with it and found it easy to learn?

40 / 61

Docker vs Configuration ManagementQuick poll:

who is a dev?

who uses Chef/Puppet/Salt/Ansible/...?

who is happy with it and found it easy to learn?

CM is good, but learning curve is steep

Docker lets you automate deployment yourself

41 / 61

ProductionPeople keep asking all the time:

Is Docker ready for production?

Can I run Docker in production?

Who runs Docker in production?

42 / 61

ProductionPeople keep asking all the time:

Is Docker ready for production?

Can I run Docker in production?

Who runs Docker in production?

Why don't you ask the same questions for:

Eclipse

SublimeText

Vagrant

43 / 61

Repeat after me:

It is OK to use Docker in dev, even without going to prod.

44 / 61

— But can I...

— No! Later.   Thanks.

45 / 61

Some challenges for opsHow do I reduce the pain of deployment?

How do I scale up and down?

How do I move apps from colo to cloud and vice versa?

How do I use my resources efficiently? (i.e. without wasting CPU, RAM, disk)

46 / 61

Deployment painDevelopers send a tarball* Friday evening

Deployment deadline is Monday morning

Installation instructions are not up to date

Dependencies are incompletely specified

External services are hard-coded all over the place

They use Debian, we use CentOS

* Or a git hash, in more sophisticated places.

47 / 61

Deployment with DockerDevelopers ship a container image (or a repo with a Dockerfile)

Configuration is done through environment variables*

External dependencies are expressed with ambassadors (e.g. the redis server address is redis, and Docker takescare of setting up a proper DNS entry in the container)

The dev-to-prod pipeline was already tested by the devs, when they did the dev-to-test deployment

* See also: twelve-factor app principles.

48 / 61

Scaling upwww7 needs to be exactly like www[1-6]

Possible solutions:

Configuration Management (steep learning curve, not a silver bullet)

Golden Images (AMI...) (slow, especially for small changes)

Server Cloning (can affect existing server; requires manual touch-ups)

Docker

49 / 61

Scaling up with DockerCreate new Docker host (with e.g. docker-machine)

Deploy application (SSH + docker run, or with remote API)

Done!

(You still need to update load balancers etc., but you have todo that even without Docker anyway.)

50 / 61

From colo to cloud to coloMust make sure that servers are absolutely identical

Possible solutions:

Configuration Management

Tedious, manual work

It would be nice if we had a self-contained meta-packageholding our application and all its dependencies, all theway down to the OS!

51 / 61

From colo to cloud to coloMust make sure that servers are absolutely identical

Possible solutions:

Configuration Management

Tedious, manual work

It would be nice if we had a self-contained meta-packageholding our application and all its dependencies, all theway down to the OS!

Guess what, that's exactly what Docker does. Cool!

(You can see Docker images as "super-debs" or "super-rpms".)

52 / 61

Optimize resource usage (problem 1)You have:

5 hypervisors (physical machines)

Each server has:

16 GB RAM, 8 cores, 1 TB disk

Each week, your team asks:

one VM with X RAM, Y CPU, Z disk

Difficulty: easy

53 / 61

Optimize resource usage (problem 2)You have:

1000+ hypervisors (and counting!)

Each server has different resources:

8-500 GB of RAM, 4-64 cores, 1-100 TB disk

Multiple times a day, a different team asks for:

up to 50 VMs with different characteristics

Difficulty: ???

54 / 61

MesosGeneric solution to resource usage problem

Open Source project (Apache Foundation)

In production use at Twitter, AirBNB, eBay, ...

Runs your code but doesn't deploy or distribute it

How can we distribute our code on our nodes?

55 / 61

Mesos + DockerPrepare your workload in a container image

Test it locally or on a smaller cluster

Submit it to Mesos

Mesos picks the right nodes to run your workload

Docker downloads the container images and runs them

56 / 61

Mesos + Docker + SwarmMesos is complicated-ish to deploy

Mesos is complicated-ish to manage

Mesos is complicated-ish to use

Docker Swarm can expose a cluster* with the Docker API

Swarm lets you use a cluster without learning a new tool (Deployment and management are still a challenge) (Damn! Where is my free lunch!)

* Currently a vanilla Docker cluster, soon a Mesos cluster

57 / 61

SummaryI am a tech support engineer: Docker helps me to reproduce issues

I am a technical writer: Docker helps me to render my docs

I am a developer: Docker helps me to abstract environments, expressdependencies, be operational faster

I am a sysadmin: Docker helps me to deploy, scale, orchestrate

58 / 61

How much does it cost?

59 / 61

How much does it cost?The Docker Engine is Open Source

The Docker Registry is Open Source

I'm not here to sell anything

(But if you insist, we have commercial products, support, and all that stuff!)

60 / 61

Questions?

@jpetazzo

61 / 61