ICWE 2016, Lugano Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW Using Docker Containers to Improve Reproducibility in Software and Web Engineering Jürgen Cito, Vincenzo Ferme, Harald C. Gall
Using Docker Containers to Improve Reproducibility in Software and Web Engineering
Jan 11, 2017
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ICWE 2016, Lugano
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
Using Docker Containers to Improve Reproducibility in Software and Web Engineering
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
2
Jürgen Cito PhD @ UZH
Vincenzo Ferme PhD @ USI
The speakers
3
This tutorial…
Conceptual, more abstract
notion of Reproducibility
Concrete instructions
to aid Reproducibility in
Research & Industry
4
Agenda
Part I:What is Reproducibility? A theoretical perspective
4
Agenda
Part I:What is Reproducibility? A theoretical perspective
Part II:Docker to Aid Reproducibility Basics and Case Study 1 (Research Project Case Study)
4
Agenda
Part I:What is Reproducibility? A theoretical perspective
Part II:Docker to Aid Reproducibility Basics and Case Study 1 (Research Project Case Study)
Part III:Docker to Aid Reproducibility Advanced Topics and Case Study 2 (Industrial Case Study)
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
Part I - What is Reproducibility?
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
ICWE 2016, Lugano
6
Reproducibility is the ability of an entire experiment or study to be duplicated, either by the same researcher or by someone else working independently.
Reproducing an experiment is called replicating it.
What is Reproducibility?
No research paper can ever be consideredto be the final word, and the replication and corroboration of research results is key to the scientific process.
7
What is Reproducibility?
Repeatability of a certain process in order to establish a fact or the conditions under which we are able to observe the same fact*
A process to share methods, and describe the environment, in order to recreate results.
* Mockus et al. “Experiences from replicating a case study to investigate reproducibility of software development.”
8
Scientific Process
Experiment Design
Data Collection
Data Analysis
Results Interpretation
Hypotheses
Reproducibility
9
What is Reproducibility?
Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge
9
What is Reproducibility?
Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge
Controlling environment > Execution environment> Dependencies
9
What is Reproducibility?
Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge
Controlling environment > Execution environment> Dependencies
Providing data > Ability to interpret data > Computational analyses
9
What is Reproducibility?
Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge
Controlling environment > Execution environment> Dependencies
Providing data > Ability to interpret data > Computational analyses
Low barriers to replicate > Comprehensible results> Ease of achieving replication
10
What is Reproducibility in SE/WE Research?
Algorithms / Computational Analyses
Developed Tools or Prototypes
Quantitative Evaluations
+ internal knowledge of the necessary process to derive/establish results
11
In Software Engineering Research:- FSE’15/16, MSR’15/16
In Programming Languages Research:- PLDI, POPL, OOPSLA
Artifact Evaluation & Replication Packages
12
Current State of Sharing Artefacts
Researcher’s website
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
Part II - Research Project Case Study
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
ICWE 2016, Lugano
14
Case Study 1: ChangeDistiller
https://bitbucket.org/sealuzh/tools-changedistiller
Research Project
15
Why is reproducibility hard?
Why does it fail?
16
Reasons for Failed Reproducibility (1/2)
*https://twitter.com/ianholmes/status/288689712636493824
17Source: Collberg et al., “Measuring Reproducibility in Computer Systems Research”, http://reproducibility.cs.arizona.edu/tr.pdf
Reasons for Failed Reproducibility (2/2)
18
Case Study 1: ChangeDistiller
https://bitbucket.org/sealuzh/tools-changedistiller
Research Project
- Developed 2006-2009 How many Java versions have we passed?
- Dependencies defined in a Maven pom file Are they all still available in the repository?
- How does analysis in ChangeDistiller work?What is the entry point?
19
Challenges in Reproducibility
19
> No standard way of describing experiments, environments, (derived) data, and workflows
Challenges in Reproducibility
19
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis
Challenges in Reproducibility
19
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis
> The experimental nature of the research code and ecosystems makes it often hard to build
Challenges in Reproducibility
19
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis
> The experimental nature of the research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
Challenges in Reproducibility
19
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis
> The experimental nature of the research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
> Infrastructure for storage and distribution
Challenges in Reproducibility
20
… to the rescue
21
Docker Containers to the rescue (1/2)
What is Docker?
Docker allows you to package an application with all of its dependencies into a standardized unit for software development*
Containers consist of everything that enables software to run:> Code> Runtime> System Tools > System Libraries
* https://www.docker.com/what-docker
22
Docker Containers to the rescue (2/2)
What can Docker be for SE/WE research?
Docker allows you to package a - Prototype
- Proof-of-concept Implementation - Computational analysis or experiment
with all of its dependencies into a standardized unit for reproducible research
* https://www.docker.com/what-docker
Technical Overview / Virtual Machines vs Containers
“Lightweight” VM > Container is an isolated process (“chroot on steroids”)> Own process space > Own network interface > Feels like a VM > Shares kernel with the host > Isolation through cgroups/namespaces
23
https://www.docker.com/what-is-docker
https://blog.docker.com/2016/03/containers-are-not-vms/
24
Docker Engine
Centralized runtime environment for containers
Enables portability
Sole dependency for Docker
No Emulation layer (almost no performance impact)
https://www.docker.com/products/docker-engine
25
Benefits of Docker Containers (compared to VMs)
Fast instantiation (~1-3 seconds)
Almost native performance
Transparent build process
Smaller Images
Easy to build, share, and publish
* https://www.docker.com/what-docker
25
Benefits of Docker Containers (compared to VMs)
Fast instantiation (~1-3 seconds)
Almost native performance
Transparent build process
Smaller Images
Easy to build, share, and publish
* https://www.docker.com/what-docker
also compared to
other container te
chnologies
26
Single Container Docker Workflow
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container
run
27
Dockerfile Declarative definition of an environment for producing an imageDocker Image Immutable artifact built from a Dockerfile, has one to many layers
Docker ContainerExecution environment - Instantiation/running version of an image (can be parameterized)
Docker MachineCreates Docker hosts on your local Mac or Windows box
Docker RegistryPublic or private repository stores images and allows for their distribution(Docker Hub - https://hub.docker.com/ or CoreOS Quay - https://quay.io/)
Terminology
build
28
Single Container Docker Workflow
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
Image
Docker Image Docker Container
run
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Base Image
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Base ImageBase Image can be an OS (Ubuntu)
or a different, existing image
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base ImageBase Image can be an OS (Ubuntu)
or a different, existing image
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base ImageBase Image can be an OS (Ubuntu)
or a different, existing image
Runs commands as if you were typing them in the command line
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base Image
Install
Base Image can be an OS (Ubuntu) or a different, existing image
Runs commands as if you were typing them in the command line
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base Image
Install
Base Image can be an OS (Ubuntu) or a different, existing image
Runs commands as if you were typing them in the command line
Copies local files from build context into container
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base Image
Install
Volume
Base Image can be an OS (Ubuntu) or a different, existing image
Runs commands as if you were typing them in the command line
Copies local files from build context into container
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base Image
Install
Open PortVolume
Base Image can be an OS (Ubuntu) or a different, existing image
Runs commands as if you were typing them in the command line
Copies local files from build context into container
29
DockerfileDefines the infrastructure and dependencies of a container through instructions
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
COPY redis.conf /var/www/redis.conf
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dependencies
Base Image
Install
Open Port
Start Server
Volume
Base Image can be an OS (Ubuntu) or a different, existing image
Runs commands as if you were typing them in the command line
Copies local files from build context into container
30
Data Volumes
A specially-designated directory within one or more containers that bypasses the Union File System*
Volumes allow you to manage data within containers > Mount a host directory (dependency to the host filesystem)> Mount a data volume container (dependency to another container)> Mount a shared-storage volume (NFS, iSCSI, etc.)
* https://docs.docker.com/engine/userguide/containers/dockervolumes/
31
Single Container Docker Workflow
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container
run
32
Dockerfile —> ImageDefines the infrastructure and dependencies of a container through instructions
docker build -t <imagename> .
32
Dockerfile —> ImageDefines the infrastructure and dependencies of a container through instructions
docker build -t <imagename> .
Build Context containing all local dependencies and
Dockerfile
33
Docker Images
# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MBhakyll latest d575da1e730c 2 weeks ago 1.487 GBredis alpine 50405530a7e5 4 weeks ago 15.95 MB
# docker rmi hakyllUntagged: hakyll:latestDeleted: sha256:3240943c9ea3f72db51…Deleted: sha256:a3aeefae0d4b8f61…Deleted: sha256:16a7ebd378002f1261…
33
Docker Images
# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MBhakyll latest d575da1e730c 2 weeks ago 1.487 GBredis alpine 50405530a7e5 4 weeks ago 15.95 MB
# docker rmi hakyllUntagged: hakyll:latestDeleted: sha256:3240943c9ea3f72db51…Deleted: sha256:a3aeefae0d4b8f61…Deleted: sha256:16a7ebd378002f1261…
Lists all previously built images
33
Docker Images
# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MBhakyll latest d575da1e730c 2 weeks ago 1.487 GBredis alpine 50405530a7e5 4 weeks ago 15.95 MB
# docker rmi hakyllUntagged: hakyll:latestDeleted: sha256:3240943c9ea3f72db51…Deleted: sha256:a3aeefae0d4b8f61…Deleted: sha256:16a7ebd378002f1261…
Lists all previously built images
Removes an image ‘hakyll:latest’
and all its layers from the disk
34
Single Container Docker Workflow
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image
run
Docker Container
35
Image —> Container
docker run -d --name <containername> -p 80:5000 <imagename>
https://docs.docker.com/engine/reference/run/Many more possibilities to run containers, see full reference here:
(A typical example)
35
Image —> Container
docker run -d --name <containername> -p 80:5000 <imagename>
Run container in the background (d for daemon)
https://docs.docker.com/engine/reference/run/Many more possibilities to run containers, see full reference here:
(A typical example)
35
Image —> Container
docker run -d --name <containername> -p 80:5000 <imagename>
Run container in the background (d for daemon)
https://docs.docker.com/engine/reference/run/
Give the container a unique name
Many more possibilities to run containers, see full reference here:
(A typical example)
35
Image —> Container
docker run -d --name <containername> -p 80:5000 <imagename>
Run container in the background (d for daemon)
https://docs.docker.com/engine/reference/run/
Give the container a unique name
Port mappingFirst the exposed port (80)
Second the port within the container (5000)
Many more possibilities to run containers, see full reference here:
(A typical example)
36
Container Managementdocker ps
docker ps -a
docker stop <container>
docker rm <container>
docker rm -v <container>
36
Container Managementdocker psLists all running containers
docker ps -a
docker stop <container>
docker rm <container>
docker rm -v <container>
36
Container Managementdocker psLists all running containers
docker ps -aLists all containers (including the ones that are stopped)
docker stop <container>
docker rm <container>
docker rm -v <container>
36
Container Managementdocker psLists all running containers
docker ps -aLists all containers (including the ones that are stopped)
docker stop <container>Stops a running container
docker rm <container>
docker rm -v <container>
36
Container Managementdocker psLists all running containers
docker ps -aLists all containers (including the ones that are stopped)
docker stop <container>Stops a running container
docker rm <container>Removes a stopped container
docker rm -v <container>
36
Container Managementdocker psLists all running containers
docker ps -aLists all containers (including the ones that are stopped)
docker stop <container>Stops a running container
docker rm <container>Removes a stopped container
docker rm -v <container>Removes a stopped container and its volumes
37
Container Debugging
docker run -ti --entrypoint=bash <imagename>
docker exec -ti <container> bash
docker inspect <container>
docker logs <container>
37
Container Debugging
docker run -ti --entrypoint=bash <imagename>
docker exec -ti <container> bash
docker inspect <container>Low-level information on a container or image
docker logs <container>
37
Container Debugging
docker run -ti --entrypoint=bash <imagename>
docker exec -ti <container> bash
docker inspect <container>Low-level information on a container or image
docker logs <container>Retrieves logs printed to stdout/err within the container
37
Container Debugging
docker run -ti --entrypoint=bash <imagename>
docker exec -ti <container> bashStarts an interactive shell into a running container
docker inspect <container>Low-level information on a container or image
docker logs <container>Retrieves logs printed to stdout/err within the container
37
Container Debugging
docker run -ti --entrypoint=bash <imagename>Starts a container with a different entrypoint
docker exec -ti <container> bashStarts an interactive shell into a running container
docker inspect <container>Low-level information on a container or image
docker logs <container>Retrieves logs printed to stdout/err within the container
38
Docker Hub: Public Registry
39
Pulling Docker Images Getting started with existing images
docker pull nginx:latest
40
Pulling Docker Images Getting started with existing images
docker pull nginx:latestReference to a Docker Image in the Docker Hub
41
Pulling Docker Images Getting started with existing images
docker pull nginx:latestImages can have many “tags”
42
Pulling Docker Images Getting started with existing images
docker pull nginx:latestPulls an image from a Docker registry
43
Pushing Docker Images to a Registry Tag Image
docker tag c6fdd6639541 <username>/<imagename>:<tagname>
Image Id (retrieve through )docker images
44
Pushing Docker Images to a Registry Push Image
44
Pushing Docker Images to a Registry Push Image
docker login --username=<username> --email=<email>
44
Pushing Docker Images to a Registry Push Image
docker login --username=<username> --email=<email>
docker push <username>/<imagename>:<tagname>
45
Case Study 1: ChangeDistiller
https://bitbucket.org/sealuzh/tools-changedistiller
Research Project
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container
run
46
Prerequisites for attending the tutorial
Prerequisites
• git on your terminal
• OSX or Windows: Docker Toolbox, https://www.docker.com/products/docker-toolbox. Then follow the getting started guide on the same page.
• Linux: 1) https://docs.docker.com/linux/, 2) https://docs.docker.com/machine/install-machine/, 3) https://docs.docker.com/compose/install/
Git clone
• git clone https://github.com/ICWE2016DockerTutorial/ChangeDistiller.git
• git clone https://github.com/ICWE2016DockerTutorial/NodeJSApp.git
• git clone https://github.com/ICWE2016DockerTutorial/MongoDB.git
PDF on: http://icwe2016.inf.usi.ch/program/tutorials/docker
47
Case Study 1: ChangeDistiller
https://goo.gl/3h7DGA
We paste the commands we run, live at the following url:
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Base Image
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Copy sources
Base Image
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Copy sources
Base Image
Dependencies
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Copy sources
Base Image
Install
Dependencies
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Copy sources
Base Image
Install
Dependencies
Volume for generated files
48
Case Study 1: ChangeDistiller
(Simple) Dockerfile for ChangeDistiller
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Copy sources
Base Image
Install
Start/enter at analysis
Dependencies
Volume for generated files
49
Case Study 1: ChangeDistiller
docker build -t icwe2016dockertutorial/changedistiller:master .Executes in the folder with the Dockerfile. Creates an image
Illustrative execution of Docker commands
docker run --name cd icwe2016dockertutorial/changedistiller:masterCreates the container and runs the analysis
Context
cd ChangeDistiller
50
Case Study 1: ChangeDistiller Version Pinning
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven=3.0.5-3 RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
50
Case Study 1: ChangeDistiller Version Pinning
FROM java:openjdk-7u101-jdk
ENV app_dir /usr/src/changedistiller
COPY . ${app_dir} WORKDIR ${app_dir}
RUN apt-get update && apt-get install -y maven=3.0.5-3 RUN ${app_dir}/build.sh RUN mvn install
VOLUME ${app_dir}/target
ENTRYPOINT ["mvn", "test"]
Version Pinning for Dependencies
51
Docker Hub: Automated/Trusted Build Improves transparency
https://docs.docker.com/docker-hub/builds/
Docker Hub
51
Docker Hub: Automated/Trusted Build Improves transparency
https://docs.docker.com/docker-hub/builds/
Docker Hubpush
51
Docker Hub: Automated/Trusted Build Improves transparency
https://docs.docker.com/docker-hub/builds/
Docker Hubbuildpush
Recap: Challenges in Reproducibility
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
> Trusted infrastructure for storage and distribution
52
Recap: Challenges in Reproducibility
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
> Trusted infrastructure for storage and distribution
Dockerfile
Dockerfile
52
Recap: Challenges in Reproducibility
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
> Trusted infrastructure for storage and distribution
Dockerfile
Docker Image
Dockerfile
52
Recap: Challenges in Reproducibility
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
> Trusted infrastructure for storage and distribution
Dockerfile
Docker Image
Docker Container
Dockerfile
52
Recap: Challenges in Reproducibility
> No standard way of describing experiments, environments, (derived) data, and workflows
> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build
> Unresolved or undocumented dependencies
> Trusted infrastructure for storage and distribution
Dockerfile
Docker Image
Docker Container
Registries (Docker Hub, Quay, …)
Dockerfile
52
ICWE 2016, Lugano
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
\
Question Time!
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
54
Part III - Multi Containers Docker Application Preview
Reproduce Bugs present in a Version of a CRUD Web Application54
Redis
NodeJS App
MongoDB
network
54
Part III - Multi Containers Docker Application Preview
Reproduce Bugs present in a Version of a CRUD Web Application54
Redis
NodeJS App
link
MongoDB
link
network
54
Part III - Multi Containers Docker Application Preview
Reproduce Bugs present in a Version of a CRUD Web Application54
Redis
NodeJS App
link
MongoDB
link
network
Docker Compose
ICWE 2016, Lugano
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
End of Part II - Lunch Break!
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
ICWE 2016, Lugano
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
Part III - Industrial Case Study
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
57
What is Reproducibility?
57
What is Reproducibility?
Challenges in Reproducibility
57
What is Reproducibility?
Single Container Docker Workflow# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container
run
Challenges in Reproducibility
58
Multi Containers Docker Workflow# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container 1
run
58
Multi Containers Docker Workflow
Docker HubDocker Container 2
run
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container 1
run
network
58
Multi Containers Docker Workflow
Docker HubDocker Container 2
run
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container 1
run
network
58
Multi Containers Docker Workflow
Docker HubDocker Container 2
run
link
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)
RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379
ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]
Dockerfile
build
Image
Docker Image Docker Container 1
run
59
Case Study 2: NodeJS + MongoDB + Redis
Reproduce Bugs present in a Version of a CRUD Web Application
Redis
NodeJS App
MongoDB
network
59
Case Study 2: NodeJS + MongoDB + Redis
Reproduce Bugs present in a Version of a CRUD Web Application
Redis
NodeJS App
link
MongoDB
link
network
60
Case Study 2: NodeJS + MongoDB + Redis
Reproduce Bugs present in a Version of a CRUD Web Application
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
60
Case Study 2: NodeJS + MongoDB + Redis
Reproduce Bugs present in a Version of a CRUD Web Application
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Custom Schema and Data
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
60
Case Study 2: NodeJS + MongoDB + Redis
Reproduce Bugs present in a Version of a CRUD Web Application
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Custom Schema and Data
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
61
Docker Networks
docker network ls
61
Docker Networks
NETWORK ID NAME DRIVERbbd6e3c22dad bridge bridged0bcac7befe4 host hosta4f078c19762 none null
docker network ls
61
Docker Networks
docker network create icwe2016Creates a bridge network named icwe2016
NETWORK ID NAME DRIVERbbd6e3c22dad bridge bridged0bcac7befe4 host hosta4f078c19762 none null
docker network ls
61
Docker Networks
docker network create icwe2016Creates a bridge network named icwe2016
NETWORK ID NAME DRIVERbbd6e3c22dad bridge bridged0bcac7befe4 host hosta4f078c19762 none null
docker network ls
docker network create -d overlay icwe2016Creates an overlay network named icwe2016
62
Docker Networks
docker network create icwe2016
62
Docker Networks
docker network create icwe2016
icwe2016
62
Docker Networks
docker network create icwe2016
icwe2016
docker network connect icwe2016 NodeJSAppConnects NodeJSApp to the icwe2016 network
icwe2016
62
Docker Networks
docker network create icwe2016
icwe2016
docker network connect icwe2016 NodeJSAppConnects NodeJSApp to the icwe2016 network
icwe2016NodeJS App
63
Docker Networks
icwe2016NodeJS App
https://docs.docker.com/engine/reference/commandline/network_create/Many more possibilities for networks, see full reference here:
63
Docker Networks
icwe2016NodeJS App
docker run -d --net icwe2016 ... MongoDBStarts and connects MongoDB to the icwe2016 network
https://docs.docker.com/engine/reference/commandline/network_create/Many more possibilities for networks, see full reference here:
63
Docker Networks
icwe2016NodeJS App
docker run -d --net icwe2016 ... MongoDBStarts and connects MongoDB to the icwe2016 network
MongoDB
https://docs.docker.com/engine/reference/commandline/network_create/Many more possibilities for networks, see full reference here:
64
Docker Networks
icwe2016
NodeJS App MongoDB
EXPOSE 27017
64
Docker Networks
icwe2016
NodeJS App MongoDB
mongodb:27017 EXPOSE 27017
link
64
Docker Networks
icwe2016
NodeJS App MongoDB
mongodb:27017 EXPOSE 27017
link
Docker takes care of:
• adding the relevant reference to your container’s /etc/hosts
• keeping the link valid across linked containers’ restarts
65
Case Study 2: NodeJS + MongoDB + Redis
Reproduce Bugs present in a Version of a CRUD Web Application
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Custom Schema and Data
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
66
Case Study 2: NodeJS + MongoDB + Redis
https://goo.gl/3h7DGA
We paste the commands we run, live at the following url:
67
Case Study 2: Web Application
https://github.com/scotch-io/node-todoRepo with the Application we have started from:
67
Case Study 2: Web Application
https://github.com/scotch-io/node-todoRepo with the Application we have started from:
67
Case Study 2: Web Application
https://github.com/scotch-io/node-todoRepo with the Application we have started from:
67
Case Study 2: Web Application
https://github.com/scotch-io/node-todoRepo with the Application we have started from:
+ a couple of bugs, one client-side and one server-side
68
Case Study 2: NodeJS + MongoDB + Redis
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Custom Schema and Data
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
69
Case Study 2: NodeJS App Relevant files for Docker
config/database.js
config/redis.js
bin/*
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
Base Image
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
Install Dependencies
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
Install Dependencies
Copy App.
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
Install Dependencies
Copy App.
Utility
Utility
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
Install Dependencies
Copy App.
Private Port
Utility
Utility
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
Install Dependencies
Copy App.
Enter
Private Port
Utility
Utility
70
Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App
FROM node:6.2.0
RUN mkdir -p /usr/src/app WORKDIR /usr/src/app
COPY package.json /usr/src/app/ RUN npm install
COPY . /usr/src/app
COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh
COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh
EXPOSE 3000
ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]
App. Dir.
Base Image
Install Dependencies
Copy App.
Enter Start
Private Port
Utility
Utility
71
Case Study 2: NodeJS App
docker build -t icwe2016dockertutorial/nodejsapp:master .
Illustrative execution of Docker commands
Context
cd NodeJSApp
72
Case Study 2: NodeJS + MongoDB + Redis
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Custom Schema and Data
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
73
Case Study 2: MongoDB Relevant files for Docker
data/*
bin/*
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
Base Image
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
Base Image Defines also the private port
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
Base Image
Copy Data
Defines also the private port
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
Base Image
Utility
Copy Data
Defines also the private port
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
Base Image
Utility
Copy Data
Enter
Defines also the private port
74
Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data
FROM mongo:3.3.6
COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial
COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh
ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]
Base Image
Utility
Copy Data
Enter Start
Defines also the private port
75
Case Study 2: MongoDB
docker build -t icwe2016dockertutorial/mongodb:master .
Illustrative execution of Docker commands
Context
cd MongoDB
76
Case Study 2: NodeJS + MongoDB + Redis
Redis
NodeJS App
link
MongoDB
link
network
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Custom Schema and Data
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile
FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y
Web App
Illustrative execution of Docker commands
77
Case Study 2: Redis
docker pull redis:3.2.0-alpine
78
Case Study 2: Starting the NodeJS App
docker run -d --net icwe2016 --name mongodb icwe2016dockertutorial/mongodb:masterdocker
docker network create icwe2016
docker run -d --net icwe2016 --name redis redis:3.2.0-alpine
docker run -d --net icwe2016 -p 8080:3000 --name nodejsapp icwe2016dockertutorial/nodejsapp:master
Illustrative execution of Docker commands
79
Case Study 2: Connect to the NodeJs App
79
Case Study 2: Connect to the NodeJs App
79
Case Study 2: Connect to the NodeJs App
docker-machine ls
Illustrative execution of Docker commands
79
Case Study 2: Connect to the NodeJs App
docker-machine ls
Illustrative execution of Docker commands
Better solution (in Beta): https://beta.docker.com
80
Case Study 2: Server-side Fix (live)
docker exec -ti nodejsapp bash
Illustrative execution of commands
Context
app/routes.js
Bug: the App. creates the same item twice
docker restart nodejsapp
apt-get update && apt-get install -y vimvim routers.js \ exit
81
Docker in Continuous Integration and Delivery
Live Fix
CI
CD+
+
81
Docker in Continuous Integration and Delivery
Local Build
ImageLive Fix
CI
CD+
+
81
Docker in Continuous Integration and Delivery
Local Build
ImageLive Fix
CI
CD+
+
Docker Hub Build
Push
81
Docker in Continuous Integration and Delivery
Local Build
ImageLive Fix
CI
CD+
+
Docker Hub Build
Push
81
Docker in Continuous Integration and Delivery
Local Build
ImageLive Fix
CI
CD+
+
Docker Hub Build
82
Case Study 2: Client-side Fix (new build)
docker build -t icwe2016dockertutorial/nodejsapp:master .
Illustrative execution of commands (cd NodeJSApp)
Context
public/js/controllers/main.js
Bug: the App. creates empty todos
docker run -d --net icwe2016 -p 8080:3000 --name nodejsapp icwe2016dockertutorial/nodejsapp:master
docker rm -f -v nodejsapp
83
Case Study 2: Undeploy the App
docker rm -f —v mongodb redis nodejsappIllustrative execution of Docker commands
84
Case Study 2: A Better Way of Starting the NodeJS App
Docker Compose
85
Docker ComposeA tool for defining and running multi-container Docker applications
version: '2'
build:
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
volumes:
85
Docker ComposeA tool for defining and running multi-container Docker applications
version: '2'
build:
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
volumes:
Build settings
85
Docker ComposeA tool for defining and running multi-container Docker applications
version: '2'
build:
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
volumes:
Build settings
Services’ deployment settings
85
Docker ComposeA tool for defining and running multi-container Docker applications
version: '2'
build:
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
volumes:
Build settings
Services’ deployment settings
Networks’ settings
85
Docker ComposeA tool for defining and running multi-container Docker applications
version: '2'
build:
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
volumes:
Build settings
Services’ deployment settings
Networks’ settings
Volumes’ settings
86
Docker ComposeA tool for defining and running multi-container Docker applications
docker-compose up -d
docker-compose start MongoDB
docker-compose logs
https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:
86
Docker ComposeA tool for defining and running multi-container Docker applications
docker-compose up -dStarts the services and detaches from the command
docker-compose start MongoDB
docker-compose logs
https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:
86
Docker ComposeA tool for defining and running multi-container Docker applications
docker-compose up -dStarts the services and detaches from the command
docker-compose start MongoDBStarts the MongoDB service
docker-compose logs
https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:
86
Docker ComposeA tool for defining and running multi-container Docker applications
docker-compose up -dStarts the services and detaches from the command
docker-compose start MongoDBStarts the MongoDB service
docker-compose logsShows the aggregated logs from all the services
https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:
87
Case Study 2: Docker Compose File (1/4)
version: '2'
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
Context
cd NodeJSApp
87
Case Study 2: Docker Compose File (1/4)
version: '2'
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
Services’ deployment settings
Context
cd NodeJSApp
87
Case Study 2: Docker Compose File (1/4)
version: '2'
services: mongodb: ...
redis: ...
nodejsapp: ...
networks: icwe2016: driver: bridge
Services’ deployment settings
Networks’ settings
Context
cd NodeJSApp
88
Case Study 2: Docker Compose File (2/4)
services: mongodb: image: icwe2016dockertutorial/mongodb:master container_name: mongodb # make it explicit the ports exposed by the Dockefile expose: - 27017 # make it explicit the volumes exposed by the Dockerfile volumes: - /data/db - /data/configdb networks: - icwe2016
89
Case Study 2: Docker Compose File (3/4)
redis: image: redis:3.2.0-alpine container_name: redis # make it explicit the ports exposed by the Dockefile expose: - 6379 # make it explicit the volumes exposed by the Dockerfile volumes: - /data networks: - icwe2016
90
Case Study 2: Docker Compose File (4/4)
nodejsapp: image: icwe2016dockertutorial/nodejsapp:master container_name: nodejsapp # make it explicit the ports exposed by the Dockefile expose: - 3000 depends_on: - mongodb - redis ports: - "8080:3000" networks: - icwe2016
91
Case Study 2: Starting the NodeJS App with Docker Compose
docker-compose up -dStarts the services and detaches from the command
docker-compose logs -fShows and follows the aggregated logs from all the services
Illustrative execution of Docker commands
Context
cd NodeJSApp
92
Recap: Multi Containers Deployment
92
Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse
92
Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse
> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version
92
Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse
> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version
> Docker speeds up the debugging of applications composed of many services
92
Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse
> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version
> Docker speeds up the debugging of applications composed of many services
> Docker provides network abstraction for portable deployments of multi container services
92
Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse
> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version
> Docker speeds up the debugging of applications composed of many services
> Docker provides network abstraction for portable deployments of multi container services
> Docker Compose enables portable deployment descriptors
ICWE 2016, Lugano
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
End of Part III - Summary Time!
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
94
Steps for ReproducibilityA recipe for enhancing reproducibility with Docker
94
Steps for ReproducibilityA recipe for enhancing reproducibility with Docker
> Provide the source code (documented) on a public repository
94
Steps for ReproducibilityA recipe for enhancing reproducibility with Docker
> Provide the source code (documented) on a public repository
> Release the Dockerfiles using version pinning
94
Steps for ReproducibilityA recipe for enhancing reproducibility with Docker
> Provide the source code (documented) on a public repository
> Release the Dockerfiles using version pinning
> Setup an automated/trusted build on the Docker Hub
94
Steps for ReproducibilityA recipe for enhancing reproducibility with Docker
> Provide the source code (documented) on a public repository
> Release the Dockerfiles using version pinning
> Setup an automated/trusted build on the Docker Hub
> Provide portable deployment descriptors using Docker Compose
95
Steps for ReproducibilityAdditional useful services
> DOI for Source Code and Docker Images:
• Citable Codehttps://guides.github.com/activities/citable-code/
95
Steps for ReproducibilityAdditional useful services
> DOI for Source Code and Docker Images:
• Citable Codehttps://guides.github.com/activities/citable-code/
• Citable Docker Images http://www.software.ac.uk/blog/2016-03-29-reproducible-research-citing-your-execution-environment-using-docker-and-doi
96
Docker’s Limitations
96
Docker’s Limitations
> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]
96
Docker’s Limitations
> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]
> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]
96
Docker’s Limitations
> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]
> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]
> Limits with proprietary software and dependencies
96
Docker’s Limitations
> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]
> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]
> Limits with proprietary software and dependencies
> Non-Disclosure Agreements / Intellectual Property
96
Docker’s Limitations
> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]
> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]
> Limits with proprietary software and dependencies
> Non-Disclosure Agreements / Intellectual Property
> Can we build the same artifact from the specification (Dockerfile) even in 10 years? [Suggestion: Version Pinning]
Sum up and Conclusion
> Containers enable a standard, fast, and easy way of describing experiments and environments
> Containers help your future self, reviewers, and other researchers and colleagues to make use and improve your work
98
Further Reading> Docker for beginners: http://prakhar.me/docker-curriculum/
> Docker Hands-on: https://github.com/alexellis/HandsOnDocker/blob/master/Labs.md
> Experience in using Docker for reproducible research: http://melissagymrek.com/science/2014/08/29/docker-reproducible-research.html
> Multi-host networking with Docker: https://docs.docker.com/engine/userguide/networking/get-started-overlay/
ICWE 2016, Lugano
Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW
Question Time!
Jürgen Cito, Vincenzo Ferme, Harald C. Gall
Material: https://github.com/ICWE2016DockerTutorial https://hub.docker.com/u/icwe2016dockertutorial/
Speakers References + Slides + Material
Slides: http://icwe2016.inf.usi.ch/program/tutorials/docker
Vincenzo Ferme PhD @ [email protected]@VincenzoFerme
Jürgen Cito PhD @ UZH
[email protected]@citostyle
Related Documents
