Who Am I? The DevOps Challenge Beyond VMs The How of Docker Docker 101 Docker Examples Docker Limits Hadoop Demo Conclusions Next-Gen Cloud Computing and DevOps with Docker Containers Hamilton Turner [email protected]March 12, 2014 March 2014 1 / 43
MIT Licensed - Reuse freely, but attribute "Hamilton Turner"
An introduction to the Docker container engine. Focuses on how to use Docker and implications of Docker for Cloud-based services. Shows multiple examples of rapidly starting complex environments using Docker. Very minor discussion on how Docker works technically.
Presentation source is available at https://github.com/hamiltont/intro-to-docker
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Next-Gen Cloud Computing and DevOps withDocker Containers
• No more differencesbetween dev andproduction environments!
• This is great! ....Right?
March 2014 4 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Comparison of Docker Containers And VMs
Credit: quay.io - A Secure Hosting Solution For Private Docker Repositories
Docker Container Virtual Machine
Avg Host Resources Consumed Low HighClean Startup Time seconds minutes/hoursEnvironment (FS) Sharing Small (Union filesystem) Large (Entire Snapshot)Environment Reproducibility High Moderate (AWS image)Software Modifications Needed? Perhaps (one process) UnlikelyAttack Surface Untested SmallSystem Monitoring Use Linux Tools Custom systems
March 2014 5 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
The How of Docker
Docker shares the kernel with the host, uses Linuxnamespaces+cgroups+union filesystems to isolate
• process trees (PIDs)
• mounts (mnt)
• network (net)
• inter-processcommunication (ipc)
• user accounts (user)
• hostnames (utc)
• memory
• CPU
• Disk access (blkio)
• Device access (devices)
Summary: Docker combines and standardizes a number ofexisting Linux components (kernel 3.8+)
March 2014 6 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
The How of Docker, Union Filesystem Version
• Each layer of the FS is mountedon top of prior layers
• The first layer is the base image
• Current base images includedebian, ubuntu, busybox, fedora,cent os, etc
• Each read-only layer is called animage (A layer is just acollection of files and folders!)
• The top layer is the onlymodifiable layer - it’s termed thecontainer
March 2014 7 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Docker 101: Run Interactive Container
$ sudo d o c k e r run − i −t ubuntu / b i n / bash
• sudo : Docker has to be run as root!• run : we are running a container• -i -t : we want a terminal (stdin and stdout), and we
want to be connected to those so we can interact withthe continer
• ubuntu : The base image for this container• /bin/bash : Let’s run bash
$ sudo d o c k e r run − i −t ubuntu / b i n / bashroot@03711559d57d : / # cat /etc/*release*
DISTRIB ID=UbuntuDISTRIB RELEASE=12.04DISTRIB CODENAME=p r e c i s eDISTRIB DESCRIPTION=”Ubuntu 1 2 . 0 4 LTS”root@03711559d57d : / # exit
March 2014 8 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Docker 101: Run Non-Interactive Container
Flags -i and -t are good for interacting with a container, butfor scripting or long-running tasks, you’ll want to use detached(-d) mode
$ sudo d o c k e r run −d ubuntu / b i n / bash −c ” echo h i ”94490365 f464bab1f009ec0971e1691213b4562dbaeb04b2e33adcbb1190dafa$
Odd things:
• There was no ‘hi’
• You were given this long string
• You are back at your original shell, even though you ranbash
In detached mode, docker immediately returns a container ID.This ID can be used to fetch container stdout, check containerstatus, stop the container, etc
March 2014 9 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Docker 101: Run Non-Interactive Container, PartTwo
Ok, let’s see what’s happening using our container ID
$ sudo d o c k e r run −d ubuntu / b i n / bash −c ” echo h i ”d2026870efedf09e29dbea146d399e60493e9dd0ebbf6124347d60d801e295fd$ sudo d o c k e r l o g s d2026870efedf09e29dbea146d399e60493e9dd0ebbf6124347d60d801e295fdh i
Container ID’s can be referenced by unique prefix too
$ sudo d o c k e r l o g s d202h i
docker ps shows you what containers are running
$ sudo docke r psCONTAINER ID IMAGE COMMAND CREATED STATUSd2026870ef ubuntu : 1 2 . 0 4 / b in / bash −c wh i l e t 1 minute ago Up 1 min
March 2014 10 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
More on Container IDs
Typically, you will want to store the ID$ MY ECHO=$ ( sudo docke r run −d ubuntu / b in / bash
−c ” echo h i ” )$ sudo docke r l o g s $MY ECHOh i
• Detached Mode (e.g. docker run -d)• Docker run response is the container ID• To capure the output, we use $(...)• This output is stored into variable MY ECHO,
and later retrieved with $MY ECHO• Interactive Mode (e.g. docker run -i -t)
• Run container, modify, then exit. Container is now stopped• Use docker ps -a to show all containers, incl. stopped
ones• Or use docker ps -l -q to show the last container ID
$ sudo docke r ps −aCONTAINER ID IMAGE COMMAND CREATED STATUSd2026870ef ubuntu : 1 2 . 0 4 / b in / bash −c wh i l e t 1 minute ago Ex i t 0$ sudo docke r ps −q − ld2026870ef
Note: Docker now supports container namesMarch 2014 11 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Storing A Container For Reuse(a.k.a. Building an Image)
• Recall: the container filesystem is thefinal union with a stack of images
• docker commit converts this containerfilesystem into an image
• The image can then be used to runother containers
$ APP=$ ( sudo d o c k e r run −d ubuntu / b i n / bash −c‘ ‘ echo h i > c o n f i g . out ’ ’ )
$ sudo d o c k e r commit $APP h a m i l t o n t /myapp$ sudo d o c k e r run − i −t h a m i l t o n t /myapp / b i n / bashroot@3a1f0c28b822 : / # cat config.out
h i
If you could share this image...then others could build newcontainers based on this image!
March 2014 12 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Sharing An Image For Reuse
• Images can be shared using a registry
• docker push and docker pull
• There is a public registry available, or your company canhost it’s own private registry Check out quay.io
• If docker run does not find the image locally, it willautomatically search known registries
$ sudo d o c k e r push h a m i l t o n t /myapp$ sudo d o c k e r p u l l h a m i l t o n t / myotherapp
• The images subcommand can be used to list local images
$ sudo docke r imagesREPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEham i l t on t /myapp l a t e s t d100b411c51e 2 minutes ago 204 .4 MBham i l t on t /myotherapp l a t e s t 7 cb2d9010d39 11 days ago 410 .6 MBubuntu 12 .04 9 cd978db300e 5 weeks ago 204 .4 MBubuntu l a t e s t 9 cd978db300e 5 weeks ago 204 .4 MB
March 2014 13 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Benefits of Using Union Filesystems For Images
• Hypothetical:• I run a ubuntu container, make changes, and exit• I commit my changes to an image and run docker push• My colleage wants to docker pull my image• What do they need to download?
• Answer:• Just your changes!• They have probably already downloaded the ubuntu base
image
• No inherent need for multi-GB images
• Download only files, not arbitrary filesystem junk
• While YMMV,80% of images are ≤ 50MB, 95% are ≤ 500MB
March 2014 14 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Linking Host/Container Filesystems
A nice seperation of concerns would place application inside thecontainer, logs outside the container.The -v flag can mount a host volume to the container
$ sudo mkdir / a p p l o g s$ sudo d o c k e r run − i −t −v / a p p l o g s : / l o g s ubuntu
/ b i n / bashroot@842fa9699353 : / # cd /logs/
root@842fa9699353 : / l o g s # echo "My application log"
> l o g . outroot@842fa9699353 : / l o g s # exit
$ c a t / a p p l o g s / l o g . outMy a p p l i c a t i o n l o g
-v can also be used to access configuration on the host
$ sudo mkdir / a p p c o n f$ sudo d o c k e r run − i −t −v / a p p c o n f : / e t c / app ubuntu
/ b i n / bashroot@842fa9699353 : / # my_app --conf /etc/app
March 2014 15 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Exposing Container Network Ports
Docker container ports are not published unless requested.The -p flag can be used to publish a port
$ SERVER=$ ( d o c k e r run −d −p 8000 ubuntu / b i n / bash−c ‘ w h i l e t r u e ; do s l e e p 5 ; done ’ )
$ sudo d o c k e r p o r t $SERVER 80000 . 0 . 0 . 0 : 4 9 1 5 8
Breakdown:
• Run a bash process inside the container, loopingindefinitely
• -p 8000 caused Docker to find an unused host port andlink it with the container-internal port 8000
• We used the port subcommand to find this public port
• There is nothing listening on port 8000 in the container,so this is kind of boring
March 2014 16 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Exposing Container Network Ports, Part Two
So let’s run an actual webserver!Method 1: Build my own webserver imageMethod 2: Reuse someone else’s pre-built image
$ WEB SERVER=$ ( sudo d o c k e r run −t −d−p 8000 h a m i l t o n t / python−s i m p l e h t t p s e r v e r )
$ sudo d o c k e r l o g s $WEB SERVERS e r v i n g HTTP on 0 . 0 . 0 . 0 p o r t 8000 . . .$ sudo d o c k e r p o r t $WEB SERVER 80000 . 0 . 0 . 0 : 4 9 1 8 6
Note:
• I chose to reuse hamiltont/python-simplehttpserver
• Navigating to http://localhost:49186 will nowconnect me to the webserver
• The container knew what command to run! More on thisnext...
• We know how to run a container, modify it, and commit itas an image
• A Dockerfile lists the steps needed to build an images
• Similar to a Makefile
• docker build is used to run a Dockerfile
• Can define default command for docker run, ports toexpose, etc
March 2014 18 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Locating Community Images
• There are hundreds of community-contributed and/orofficial images online at http://index.docker.io
• This is the official registry, you can also host your own
• You can also use the docker search subcommand tointeract with index.docker.io
$ sudo docke r s e a r c h wordp re s sNAME DESCRIPTION STARS TRUSTEDc t l c / wordp re s s 4 [OK]j b f i n k / docker−wor Same as j b f i n k /wordpress , j u s t a t r u s t e d b 2 [OK]skx skx / wordp re s s Wordpress & SSH i n a c o n t a i n e r . 2eugeneware / docke r 1 [OK]tutum/wordp re s s Wordpress Docker image − l i s t e n s i n po r t 80 1 [OK]j b f i n k / wordp re s s Wordpress 3 . 8 1
IP add r e s s : 1 7 2 . 1 7 . 0 . 4Password : N24DjBM86gPubuEEF i r e f o x : s sh −X webuser@172 . 1 7 . 0 . 4 f i r e f o xGoogle Chrome : s sh −X webuser@172 . 1 7 . 0 . 4 goog le−chrome −−no−sandbox
March 2014 25 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Example: SSH Server
# SSH=$(docker run -d -p 22 dhrp/sshd)
# docker port $SSH 22
0 . 0 . 0 . 0 : 4 9 1 6 0
Ok, SSH is running. Now to connect!
$ s sh −p 49161 root@10 . 0 . 0 . 2root@10 . 0 . 0 . 2 ’ s password :Welcome to Ubuntu 12 .04 LTS (GNU/ L inux 3.11.0−18− g e n e r i c x86 64 )
∗ Documentat ion : h t t p s : // he l p . ubuntu . com/
The programs i n c l u d e d wi th the Ubuntu system a r e f r e e s o f twa r e ;the exac t d i s t r i b u t i o n terms f o r each program ar e d e s c r i b e d i n thei n d i v i d u a l f i l e s i n / u s r / sha r e /doc /∗/ c o p y r i g h t .
Ubuntu comes wi th ABSOLUTELY NO WARRANTY, to the e x t e n t p e rm i t t ed bya p p l i c a b l e law .
root@7d45b427eca1 : ˜ #
March 2014 26 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Example: Continuous Integration
# JEN=$($docker run -d -p 8080 --cpu -shares =20 lzhang/jenkins)
• A project to combine and standardize existing features ofLinux
Unofficially, Docker is...
• The forefront in Linux Containers
• A huge step beyond current VM’s w.r.t. machineutilization and DevOps workflow
• A pragmatic improvement that is here to stay
March 2014 42 / 43
Who Am I?
The DevOpsChallenge
Beyond VMs
The How ofDocker
Docker 101
DockerExamples
Docker Limits
Hadoop Demo
Conclusions
Thank You For Your Time
Questions?
Please feel free to reuse/modify presentation if you wish, justremember to leave my name in there somewhere. It’s online athttps://github.com/hamiltont/intro-to-docker