Open Source Docker John Willis Director of Ecosystem Development
Open Source Docker John Willis
Director of Ecosystem Development
About Me
• One of the founding members of the “Devops” movement.
• Author of the “Devops Handbook”. • Author of the “Introduction to Devops” on
Linux Foundation edX. • Podcaster at devopscafe.org • Devops Enterprise Summit - Cofounder • Found of Socketplane (Acquired by Docker) • Formally Director of Devops at Dell • Formally Director at Chef • 10 Startups over 25 years
2
Github: botchagalupe/my-presentationsTwitter: botchagalupeWechat: botchagalupe
Devops Practices and Patterns• Continuous Delivery
• Everything in version control • Small batch principle • Trunk based deployments • Manage flow (WIP) • Automate everything
• Culture • Everyone is responsible • Done means released • Stop the line when it breaks • Remove silos
3itrevolution.com/devops-handbookhttp://www.bookdepository.com/
The world needs tools of mass innovation
A programmable Internet would be the ultimate tool of mass innovation
Let’s eliminate friction in the development cycle
A commercial product,
built on a development platform,
built on infrastructure,
built on standards.
Docker is building a stack to program the Internet
8
Docker Project Sponsor •Primary authors, contributor maintainer •6B+ Downloads, 3000+ Contributors, 500,000+ Applications •100’s of ecosystem partners •Millions of developers use Docker. Millions of servers run Docker Commercial Docker Solutions •Integrated solutions to build, ship, run Docker at scale
• Orchestration, registry, security, workflow, control plane • CaaS (containers as a service)
•Official providers of commercial technical support •10K’s cloud customers, 300+ F500 customers
About Docker, Inc.
Gerber, Anna. “The State of Containers and the Docker Ecosystem: 2015” O’Reilly, September 2015
Docker users already running in production
60%
China is part of Worldwide Docker Community
Meetups in Beijing, Changsha, Chengdu, Chongqing, Dalian, Fuzhou, Guangzhou, Hangzhou, Hong Kong, Nanjing, Qingdao, Shanghai, Shenzen, Suzhou, Tianjin, Wuhan, Xi’an, Xiamen, and Zhuhai
The Docker ecosystem
Dev Tools
Official Repositories
Operating Systems
Big Data
Service Discovery
Build / Continuous Integration
Configuration ManagementConsulting &Training
Management
Storage
Clustering & Scheduling
Networking
Infrastructure & Service Providers
Storage
Security
Monitoring & Logging
10
Docker Platform
COMPARING CONTAINERS AND VIRTUAL MACHINES
Isolation using Linux kernel featuresnamespaces ● pid ● mnt ● net ● uts ● ipc ● user
cgroups ● memory ● cpu ● blkio ● devices
Image layers
Docker Engine extensibility and plugins• Built in orchestration expands the opportunity for
the plugin to manage swarm wide vs a single Engine
• Updated architecture standardizes plugin process for ecosystem partners
– Benefits users and vendors – Standardized process of granting plugin permissions – Containerized plugins on roadmap
Docker Engine
Networking
Swarm Mode
Volumes
Plugins
Developer experience
1. Get out of the way
The best tools…
2. Adapt to you
3. Make the powerful simple
Docker for Mac Docker for Windows
Orchestration
Introducing the best way to orchestrate Docker: Docker.
Docker 1.12: now with orchestration built-in.
Swarm mode
Service API
Node identity
Built-in routing mesh
Docker 1.12: now with orchestration built-in.
Using the beta? You already have 1.12 installed.
> docker swarm init > docker service create
Docker CaaS
BUILD Development Environments
SHIP Registry: Secure Content &
Collaboration
RUN Control Plane: Deploy,
Orchestrate, Manage, Scale
Networking Volumes MonitoringLoggingConfig MgtCI/CD
IT OperationsDevelopers IT Operations
Docker CaaS Workflow
Docker Containers as a Service platform
25
BUILD Developer Workflows
SHIP Registry Services
RUN Management
Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane
Docker Cloud
Docker Container Engine
Ecosystem Plugins and Integrations
Plumbing
2013
- 05
2013
- 06
2013
- 07
2013
- 08
2013
- 09
2013
- 10
2013
- 11
2013
- 12
2014
- 01
1,000,0000
2014
- 02
2014
- 03
2014
- 04
2014
- 05
2014
- 06
2014
- 07
2014
- 08
2014
- 09
2014
- 10
2014
- 11
2014
- 12
2015
- 01
2015
- 02
2015
- 03
2015
- 04
2015
- 05
2015
- 06
2015
- 07
2015
- 08
2015
- 09
2015
- 10
2015
- 11
2015
- 12
2016
- 0
1
1,000,000,000~
10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000
6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000
2013
- 05
2013
- 06
2013
- 07
2013
- 08
2013
- 09
2013
- 10
2013
- 11
2013
- 12
2014
- 01
1,000,0000
2014
- 02
2014
- 03
2014
- 04
2014
- 05
2014
- 06
2014
- 07
2014
- 08
2014
- 09
2014
- 10
2014
- 11
2014
- 12
2015
- 01
2015
- 02
2015
- 03
2015
- 04
2015
- 05
2015
- 06
2015
- 07
2015
- 08
2015
- 09
2015
- 10
2015
- 11
2015
- 12
2016
- 0
1
~
2016
- 09
1,000,000,000~
10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000
6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000
NotaryrunC •
containerd •
HyperKit , VPNKit, DataKit •
SwarmKit •
libcontainer •
libnetwork • • Docker 1.8 : Docker Content Trust
• Docker for Mac Docker for Windows
• Docker 1.12 with built-in orchestration
• Docker 0.9 : Pluggable execution
• Docker 1.7 : Multi-Host Networking
• Docker 1.11: OCI support
runC
Open Container Initiative (OCI)
An open governance structure for creating
open industry standards: a common container runtime and
image format.
•A Linux Foundation Collaborative Project•Free from control by any particular vendor’s specific cloud stack or ecosystem
• Includes a specification, reference runtime* and now, a specified image format*seeded with runc + libcontainer by Docker
OCI Specs & Status> Announced June 20th,
2015> Charter signed on
December 8th, 2015> 49 current member
companies> Both specifications
nearing 1.0 release targets
https://opencontainers.orghttps://github.com/opencontainers
> Runtime specification: Release 1.0.0-rc2 / September 2016
https://github.com/opencontainers/runtime-spec/releases/tag/v1.0.0-rc2
1. Very close to an official 1.0 release of the runtime spec 2. Includes required core for Linux, Windows, and Solaris> Image format specification: Release 0.5.0 / September
2016
https://github.com/opencontainers/image-spec/releases/tag/v0.5.0
1. Seeded with Docker registry v2.2 specification 2. v1.0.0-rc1 release being voted/approved on mailing list
runc: An open innovation platform for containers
Implement low-level container featuresOperating system level features should be defined in the OCI runtime specificationNew capabilities (PID cgroup controls, checkpoint/restore, seccomp) implemented in
runC
INTEREST
OCI compliance/pluggable execution engineImplement a OS/environment for containers via an OCI spec compliant binaryExamples: runz (Solaris zones), runv (hypervisor-based), Intel Clear Containers
Iterative container configuration test/debugSimple variant of “Docker-like” containers with less friction for quick modificationsLow bar for dependencies: single binary + physical rootfs bundle + JSON config
INTEREST
INTEREST
containerdA daemon to control runC
built for performance and density
http://containerd.tools/
containerd
Docker 1.11
Docker for Mac
Docker for Mac architecture (simplified)
Hypervisor Framework
vmnet Framework
Docker Container Engine
Hypervisor
Linux
VPN
Data Service
Interface
Client Libraries
Admin GUI
CLI
Security Sandbox
Docker for Mac internals
Hypervisor Framework
vmnet Framework
Docker Container Engine
Hyperkit
Linux
VPNKit
DataKit
Client Libraries
Admin GUI
CLI
Security Sandbox
Improving Docker with unikernel tech
InfraKit
Problem:
Managing Docker on different infrastructure is difficult and not portable.
Consistent User Experience
44
How do we handle updates to a cluster??
Docker for AWS
EBS ELB
Container Engine
Storage plugin
Infrastructure Management
Network plugin Orchestration
IAM
CloudFormation
EC2VPC
Admin interface
Linux
User Applications / Services
Docker for AWS
EBS ELB
Container Engine
Storage plugin
InfraKit
Network plugin Orchestration
IAM
CloudFormation
EC2VPC
Admin interface
Linux
User Applications / Services
InfraKit
A toolkit for building declarative, self-healing infrastructure.
Declarative
• JSON configuration for desired infrastructure state: • Specification of instances — vm image, instance type, etc. • Group properties — size, logical identifiers, etc.
• Design patterns encourage • encapsulation • composition
• Config is input to all operations — system figures out what to do48
Self-healing
• Composed of a set of active components / processes that • monitor infrastructure state • detect state divergence • take actions
• Continuous monitoring and reconciliation — always on
• No downtime — rolling update49
Toolkit• Primitives for managing collections of resources
• create, scale, destroy • rolling update
• Abstractions & Developer SPI • Group - manages collection of resources • Instance - describes the physical resource • Flavor - extra semantics for handling instances
• A collection of executable, active components — plugins • Initially, Go daemons in the toolkit • Soon, easy management via Docker Plugins (runc)
Architecture
Instance Plugin
• Spec: specification / model of an instance (e.g. vagrant, EC2): • Logical ID, Init, Tags, and attachment • Platform-specific properties
• Methods: • /Instance.Validate • /Instance.Provision • /Instance.Destroy • /Instance.DescribeInstances
• Examples: instance plugins for EC2, Azure VM, Vagrant, … 52
Flavor Plugin
• Gives more context about the group members: • Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’) • Application-specific notions of ‘health’
Is the node not only present but also joined a swarm? • Methods:
• /Flavor.Validate • /Flavor.Prepare • /Flavor.Healthy
• Examples: flavor for Zookeeper members, Docker swarm nodes53
Group Plugin
• Main entry point for user interaction: • Create, describe update, update, destroy • Config JSON is always the input
• Composed of Instance and Flavor — mix and match to manage cattle (fungible) or pets (special)
• Methods: • /Group.Watch • /Group.Unwatch • /Group.Inspect
54
• /Group.DescribeUpdate • /Group.Update • /Group.StopUpdate
• /Group.Destroy
ConfigurationExample config file (zk.conf): Group configuration = Instance + Flavor
{ "Properties": {
/* raw configuration */
} }
{ "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } } }
Today
56
• InfraKit is just getting started… only primitives for working with groups like clusters of hosts
• But we have big plans • Improve group management strategies • More resource types — networking, load
balancers, storage…
• A cohesive framework for active management of infrastructure — physical, virtual, or containers
Get Involved
• Help define and implement new and interesting plugins • Instance plugins for different infrastructure providers • Flavor plugins for systems like etcd or mysql clusters • Group controller plugins — metrics-driven auto scaling
and more
• Help define interfaces and implement new infrastructure resource types — load balancers, networks and storage volume provisioners
57
More Info
• Github: https://github.com/docker/infrakit
• A quick tutorial: https://github.com/docker/infrakit/blob/master/docs/tutorial.md
58
Docker Open Source Projects Summary
Extra Slides
Docker on Windows