Alibaba Cloud Conference 2016 - Docker Enterprise

What’s New in Docker John Willis

Director of Ecosystem Development

About Me

• One of the founding members of the “Devops” movement. • Author of the “Devops Handbook”. • Author of the “Introduction to Devops” on Linux

Foundation edX. • Podcaster at devopscafe.org • Devops Enterprise Summit - Cofounder • Found of Socketplane (Acquired by Docker) • Formally Director of Devops at Dell • Formally Director at Chef • 10 Startups over 25 years

2

Github: botchagalupe/my-presentationsTwitter: botchagalupeWechat: botchagalupe

Devops Practices and Patterns• Continuous Delivery

• Everything in version control • Small batch principle • Trunk based deployments • Manage flow (WIP) • Automate everything

• Culture • Everyone is responsible • Done means released • Stop the line when it breaks • Remove silos

3itrevolution.com/devops-handbookhttp://www.bookdepository.com/

The world needs tools of mass innovation

A programmable Internet would be the ultimate tool of mass innovation

Let’s eliminate friction in the development cycle

A commercial product,

built on a development platform,

built on infrastructure,

built on standards.

Docker is building a stack to program the Internet

8

Docker Project Sponsor •Primary authors, contributor maintainer •6B+ Downloads, 3000+ Contributors, 500,000+ Applications •100’s of ecosystem partners •Millions of developers use Docker. Millions of servers run Docker Commercial Docker Solutions •Integrated solutions to build, ship, run Docker at scale

• Orchestration, registry, security, workflow, control plane • CaaS (containers as a service)

•Official providers of commercial technical support •10K’s cloud customers, 300+ F500 customers

About Docker, Inc.

Gerber, Anna. “The State of Containers and the Docker Ecosystem: 2015” O’Reilly, September 2015

Docker users already running in production

60%

China is part of Worldwide Docker Community

Meetups in Beijing, Changsha, Chengdu, Chongqing, Dalian, Fuzhou, Guangzhou, Hangzhou, Hong Kong, Nanjing, Qingdao, Shanghai, Shenzen, Suzhou, Tianjin, Wuhan, Xi’an, Xiamen, and Zhuhai

Cloud and Docker Status Update in China

China cloud relative to the West – DB

! China’s cloud spending to reach 24.5% of the West by 2018 from 5.7% in 2015

! More than 80% respondents are already using or plan to use containers, but only 10% used in production environment.

Survey result from 350+ feedbacks by Alibaba Cloud, June 2016

Docker Adoption in China

Other

Government

Sample of Docker CustomersFinancial Services Healthcare

Consumer Education Tech Services

11

Dockerizing applications

12

Legacy App: One container per app. Microservices: one service per container. App comprised of many containers linked together

Enterprises Are Containerizing both Legacy and Microservices Applications

13

80%46% plan to build new microservices in 2016

#4 container workload is traditional databases

1 in 3 have already containerized legacy apps

Will leverage Docker to enable hybrid cloud initiatives

Docker platform standardizes environment AND enables workload portability

3 Paths to Containers Adoption

1Containerize Monoliths Build-Test for CI; Migrate to the Cloud; Get Better CapEx/OpEx than VM

2

3

Containerize Monolith; Transform to Microservices Look for Shared Services to Transform

Enable New Microservices and Apps Greenfield CaaS

Migrate any workload anywhere

Infrastructure agnostic platform • Standardize: Docker abstracts away

the infrastructure and virtualization away from the standardized app containers

• Portability: Containers move without friction from one environment to another – no recoding needed

• Lift and Shift: Containerize legacy and microservices to gain portability

15

Bins/Libs

App

OS

Bins/Libs

App

OS

Bins/Libs

App

OS

Before: VM formats are proprietary

Bins/Libs

App

Bins/Libs

App

Bins/Libs

App

OS

After: Docker abstracts above VMs for portability

Docker Engine

OS

Docker Engine

Infrastructure optimization with Docker

Swisscom • 20:1 VM consolidation ratio • Running the same 400 MongoDB

instances in 400 containers in 20 VMs

• Reduce CapEx and OpEx costs

Leading Energy Company • Containerize legacy apps for

portability • Entire cloud to datacenter site

migration in 5 months • Dramatically accelerated release

process

16

17

Enable Modern App Initiatives with CaaS

Cloud Migration Hybrid Cloud Multi Cloud

Modernizing Apps DevOps CI/CD

DevOps

Cloud DevOpsApps

18

Enterprises Can Decide How To Adopt

Enable CI and DevOps Better Resource Usage Enable Cloud Initiatives

Transform iteratively Ship faster, with better

reliability

Enable developers to self service Enable business transformations

Containerize Build New AppsContainerize & Transform

The Docker ecosystem

Dev Tools

Official Repositories

Operating Systems

Big Data

Service Discovery

Build / Continuous Integration

Configuration ManagementConsulting &Training

Management

Storage

Clustering & Scheduling

Networking

Infrastructure & Service Providers

Storage

Security

Monitoring & Logging

19

Docker and Alibaba Announce Commercial Agreement

• Localized Docker image store and distribution for Docker Hub on Alibaba Cloud

• Alibaba will resell Commercially Supported (CS) Docker Engine and Docker Datacenter, enabling enterprises to manage their production workloads across the entire application lifecycle.

• Alibaba will provide enterprise support options for CS Docker Engine and Docker Datacenter, backed by Docker, Inc.

20

Docker Platform

COMPARING CONTAINERS AND VIRTUAL MACHINES

Isolation using Linux kernel featuresnamespaces ● pid ● mnt ● net ● uts ● ipc ● user

cgroups ● memory ● cpu ● blkio ● devices

Image layers

Docker Engine extensibility and plugins• Built in orchestration expands the opportunity for

the plugin to manage swarm wide vs a single Engine

• Updated architecture standardizes plugin process for ecosystem partners

– Benefits users and vendors – Standardized process of granting plugin permissions – Containerized plugins on roadmap

Docker Engine

Networking

Swarm Mode

Volumes

Plugins

Developer experience

1. Get out of the way

The best tools…

2. Adapt to you

3. Make the powerful simple

Docker for Mac Docker for Windows

Orchestration

Introducing the best way to orchestrate Docker: Docker.

Docker 1.12: now with orchestration built-in.

Swarm mode

Service API

Node identity

Built-in routing mesh

Docker 1.12: now with orchestration built-in.

Combine your engines in swarms of any scale Self-organizing, self-healing No external data store requiredNo single points of failure Infrastructure-agnostic topology

Swarm mode

Desired state reconciliation ScalingRolling updates Advanced scheduling Application-specified health checks Rescheduling on node failure

Docker Service API

How service deployment works

Declare

ScheduleReconcile

$ docker service create declares the service name, network, port, image:tag and scale

Managers break down service into tasks, schedules them and workers execute tasks

Engines check to see what is running and compared to what was declared to “true up” the environment

API

Allocator

Orchestrator

Scheduler

Dispatcher

R A F T

Manager Node

Worker

ExecutorWorker Node

Accepts command from client and creates service object

Reconciliation loop for service objects and creates tasks

Allocates IP addresses to tasks

Assigns tasks to nodes

Checks in on workers

docker service create

Connects to dispatcher to check on assigned tasks

How service deployment works

Executes the tasks assigned to worker node

Manager ManagerManager

Worker WorkerWorkerWorker Worker

Example service on a swarm

The declarative command describes a new service: • Named Frontend • Made of 5 containers based

on the latest my_frontend_image

• Connected on an overlay network called “my overlay”

• Assigned to port 80

$ docker service create --replicas 5 --name frontend --network myoverlay \

--publish 80/tcp mikegoelzer/my_frontend_image:latest

Manager ManagerManager

Worker WorkerWorkerWorker Worker

Example service on a swarm

This state is what swarm mode and the service deployment API will maintain.

Check to ensure 5 containers are always running for the frontend service$ docker service create --replicas 5 --name frontend

--network myoverlay \

--publish 80/tcp mikegoelzer/my_frontend_image:latest

Manager ManagerManager

Worker WorkerWorkerWorker Worker

Example service on a swarm

This command creates a new service to join an existing overlay network to communicate with frontend

$ docker service create --name redis --network myoverlay redis:latest

Built-in Routing Mesh

Swarm-wide overlay networking Container-native load-balancing DNS-based service discovery No separate cluster to setup Works with your existing load-balancers Rock-solid kernel-only data path with IPVS

Routing mesh for application services • Container-aware dynamic

load balancing • Assign ports to service that

do not change • Built in load balancing into

the Engine • Automatic service discovery

Worker

Load Balancing

Worker

Load Balancing

Worker

Load Balancing

Port 1000Port 1000 Port 1000

Manager ManagerManager

Docker user assigns a global port for a service

End user accessing www.website.com

Docker 1.12 is Simple

> docker swarm init > docker service create

How to get Docker 1.12

Mac OS X

Developer Workstations Cloud Providers

Windows PC

AWS

Azure

Commercially Supported

Docker Datacenter

CS Docker Engine

Docker Cloud

Open Source Docker Engine installers for the following Linux distros

Optimized for and integrated directly into the underlying platform with custom plugins and drivers

Ops experience

Deep integration with native load-balancers, templates, SSH keys, ACLs, scaling groups, firewall rules…

beta.docker.com

Distributed Application Bundle

experimental

www.docker.com/dab

A portable format for multi-container applications

5

Goldilocks and the 3 XaaS

Just rightToo highToo lowIaaS PaaS CaaS

5

Goldilocks and the 3 XaaS

Platform As A Service

Infrastructure As A Service

Software As A Service

Too high

Too low

Just right

Container As A Service

Docker Datacenter core values

48

Management at scale Integrated Content Trust Secure Access (RBAC) Integrates with existing

systems

Full support of Docker API Seamless dev to prod

workflow Infrastructure, network and

storage portability

Easy to setup and use Native Docker solution Extend existing Docker developer experience

+ +Agility Portability Control

Agility, portability and control for developers and IT

Developers IT Operations

• Freedom to create and deploy apps fast

• Define and package application needs

• Quickly and flexibly respond to changing needs

• Standardize, secure, and manage

Frictionless portability across teams, environments, infrastructure

49

Docker Datacenter platform

Management Universal Control Plane

Security Content Trust, RBAC, LDAP/AD

Orchestration Swarm

Container Runtime Engine

Registry Service Trusted Registry

BUILD Development Environments

SHIP Registry: Secure Content &

Collaboration

RUN Control Plane: Deploy,

Orchestrate, Manage, Scale

Networking Volumes MonitoringLoggingConfig MgtCI/CD

IT OperationsDevelopers IT Operations

Docker CaaS Workflow

Docker Containers as a Service platform

52

BUILD Developer Workflows

SHIP Registry Services

RUN Management

Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane

Docker Cloud

Docker Container Engine

Ecosystem Plugins and Integrations

UCP Permission Model

Docker UCP 1.1 - DTR 2.0• HA • Unified Auth • Compose deployment • UI to add nodes

Security scanning in Docker Cloud

Adoption

2013

- 05

2013

- 06

2013

- 07

2013

- 08

2013

- 09

2013

- 10

2013

- 11

2013

- 12

2014

- 01

1,000,0000

2014

- 02

2014

- 03

2014

- 04

2014

- 05

2014

- 06

2014

- 07

2014

- 08

2014

- 09

2014

- 10

2014

- 11

2014

- 12

2015

- 01

2015

- 02

2015

- 03

2015

- 04

2015

- 05

2015

- 06

2015

- 07

2015

- 08

2015

- 09

2015

- 10

2015

- 11

2015

- 12

2016

- 0

1

1,000,000,000~

10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000

6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000

2013

- 05

2013

- 06

2013

- 07

2013

- 08

2013

- 09

2013

- 10

2013

- 11

2013

- 12

2014

- 01

1,000,0000

2014

- 02

2014

- 03

2014

- 04

2014

- 05

2014

- 06

2014

- 07

2014

- 08

2014

- 09

2014

- 10

2014

- 11

2014

- 12

2015

- 01

2015

- 02

2015

- 03

2015

- 04

2015

- 05

2015

- 06

2015

- 07

2015

- 08

2015

- 09

2015

- 10

2015

- 11

2015

- 12

2016

- 0

1

~

2016

- 09

1,000,000,000~

10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000

6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000

NotaryrunC •

containerd •

HyperKit , VPNKit, DataKit •

SwarmKit •

libcontainer •

libnetwork • • Docker 1.8 : Docker Content Trust

• Docker for Mac Docker for Windows

• Docker 1.12 with built-in orchestration

• Docker 0.9 : Pluggable execution

• Docker 1.7 : Multi-Host Networking

• Docker 1.11: OCI support

Docker on Windows

Extra

InfraKit

A toolkit for building declarative, self-healing infrastructure.

Declarative

• JSON configuration for desired infrastructure state: • Specification of instances — vm image, instance type, etc. • Group properties — size, logical identifiers, etc.

• Design patterns encourage • encapsulation • composition

• Config is input to all operations — system figures out what to do66

Self-healing

• Composed of a set of active components / processes that • monitor infrastructure state • detect state divergence • take actions

• Continuous monitoring and reconciliation — always on

• No downtime — rolling update67

Toolkit• Primitives for managing collections of resources

• create, scale, destroy • rolling update

• Abstractions & Developer SPI • Group - manages collection of resources • Instance - describes the physical resource • Flavor - extra semantics for handling instances

• A collection of executable, active components — plugins • Initially, Go daemons in the toolkit • Soon, easy management via Docker Plugins (runc)

Architecture

Instance Plugin

• Spec: specification / model of an instance (e.g. vagrant, EC2): • Logical ID, Init, Tags, and attachment • Platform-specific properties

• Methods: • /Instance.Validate • /Instance.Provision • /Instance.Destroy • /Instance.DescribeInstances

• Examples: instance plugins for EC2, Azure VM, Vagrant, … 70

Flavor Plugin

• Gives more context about the group members: • Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’) • Application-specific notions of ‘health’

Is the node not only present but also joined a swarm? • Methods:

• /Flavor.Validate • /Flavor.Prepare • /Flavor.Healthy

• Examples: flavor for Zookeeper members, Docker swarm nodes71

Group Plugin

• Main entry point for user interaction: • Create, describe update, update, destroy • Config JSON is always the input

• Composed of Instance and Flavor — mix and match to manage cattle (fungible) or pets (special)

• Methods: • /Group.Watch • /Group.Unwatch • /Group.Inspect

72

• /Group.DescribeUpdate • /Group.Update • /Group.StopUpdate

• /Group.Destroy

ConfigurationExample config file (zk.conf): Group configuration = Instance + Flavor

{ "Properties": {

/* raw configuration */

} }

{ "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } } }

Operations

• Make sure the plugins are running: • infrakit/group &; infrakit/zookeeper &; infrakit/vagrant &;

• “Watch” the group starts management: • infrakit/cli group watch zk.conf

• Update the config, e.g. change size or add IP address • Describe changes before committing —

infrakit/cli group describe zk.conf • Begin update —

infrakit/cli group update zk.conf74

Today

75

• InfraKit is just getting started… only primitives for working with groups like clusters of hosts

• But we have big plans • Improve group management strategies • More resource types — networking, load

balancers, storage…

• A cohesive framework for active management of infrastructure — physical, virtual, or containers

Get Involved

• Help define and implement new and interesting plugins • Instance plugins for different infrastructure providers • Flavor plugins for systems like etcd or mysql clusters • Group controller plugins — metrics-driven auto scaling

and more

• Help define interfaces and implement new infrastructure resource types — load balancers, networks and storage volume provisioners

76

More Info

• Github: https://github.com/docker/infrakit

• A quick tutorial: https://github.com/docker/infrakit/blob/master/docs/tutorial.md

77