1 WhiteHedge Baking Docker using Chef Mukta Aphale DevOps Practice Head, WhiteHedge Technologies [email protected]
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
WhiteHedge
Baking Docker using ChefMukta AphaleDevOps Practice Head, WhiteHedge Technologies
2
WhiteHedgeWhiteHedge
2
TABLE OF CONTENTS - AGENDA
WelcomeAbout WhiteHedge
and Me
1
DockerA quick introduction
2
Chef + DockerGetting best of both worlds
3
Push Jobs
5
Chef Cookbook
6
Chef Containers
7
Our Story
8
CD PipelineUse of knife + ssh
4
3
WhiteHedgeWhiteHedge
3
INTRODUCTION- HELLO
Hello! Welcome to WhiteHedge
WhiteHedge is an Agile Software Product Development company. But what reallydescribes us is what we think about Our Work , Our Clients.
We have deep experience in three of the most sought-after technology domains in theindustry today: Big Data Analytics, Cloud Computing, and DevOps. Having deepknowledge in these domains make us stand out as the best and the coolest provider ofservices. Our Dev-Ops will bring you a single team that works together with continuousdevelopment, continuous testing, continuous deployments, logging, monitoring andsecurity.
4
WhiteHedgeWhiteHedge
4
WHITEHEDGE - AN OVERVIEW
Global Presence
Agile + Flexible
Thorough + Quick Learner
Competitive + Comprehensive
Honest + Transparent
Young + Mature
Innovative + Creative
More about us …What defines us ?
California
New Jersey Rotterdam
Pune
Envision Products |
Convert into businesses
100+ employees | 50+ live
products world-wide
The best of the Talent and
Infrastructure
Started 2003 | Focused Agile
Product Development
Self funded | Well funded |
Profitable
5
WhiteHedge
5
ABOUT ME
Ruby, Java, C Developer turned into DevOps Architect
Contributed to Chef development
Chef azure extension
Knife plugins: knife-azure, knife-ec2, knife-openstack
Knife WinRM, knife windows listener
Technology, innovation and the thirst to keep learning
are what define me
Love to travel, read, write
Above all, I am a mother to two boys!
@muktaa
https://in.linkedin.com/in/muktaa
6
WhiteHedge
DockerA Quick Introduction
7
WhiteHedgeWhat is Docker?
Linux Container
3 Components:Docker Engine
Docker HubDocker Images
Benefits:Speed
PortabilityDensity
Open Source
sufficient containers from any
8
WhiteHedgeDocker is not a VM
Virtual Machine Docker
9
WhiteHedge
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install
libfuse-dev
ADD dev.conf/etc/myapp-
config/
Dockerfiles
Codify your configuration
Set of bash commands
Example:
• HelloScala
Dockerfile
dev.conf
• Docker build HelloScala
10
WhiteHedgeUse Cases of Docker
Shared Hosting PaaS
Microservices
Lightweight Testing
11
WhiteHedge
Chef and DockerGetting the best of both worlds!
12
WhiteHedgeTHE CHALLENGE
AutomateMake Whole
EnchiladaDeliver!
13
WhiteHedgeConfig Management Vs Golden Images
Control the environment Vs System Image / Runtime image
Tradeoff between flexibility and manageability
CM is the vein of DevOps
• Shell scripts -> Chef
Immutable Infrastructure
14
WhiteHedge
Docker
Chef
Awesomeness
15
WhiteHedgeChef and Docker
• Replaces Human Tasks,• Idempotence, • Thick client - thin servers,• Order Matters, • Huge Community Support
• An improved Robot, • Fast & Easy,• Fresh fish in the market,• Ready to be baked!
16
WhiteHedge
Simple CD PipelineBecause simple things can bring the most happiness!
17
WhiteHedgeSimple CI/CD Pipeline
Deploy using knife-ssh or Push Jobs
docker pull
docker stop
docker run
Docker Registry
Unique tag
Docker Image
Save image
Build Process
Build tools have docker support
Build tools generate a docker image
Code
git push
Triggers Build
CI Server
18
WhiteHedgeThe Simple Steps
git push to https://github.com/muktaa/HelloScala
Triggers a build on your CI server• sbt docker
• docker push muktaa/hello-scala
• knife ssh 'role:test' 'deploy.sh' -x ssh-user -i ssh-key -c knife.rb
Build tools offer docker integration
Eg: Maven has docker-maven-plugin
• https://github.com/spotify/docker-maven-plugin• mvn clean package docker:build -DpushImage
19
WhiteHedge
~/github/HelloScala > sbt docker
[info] Loading project definition from /Users/muktaaphale/github/HelloScala/project
[info] Set current project to hello-scala (in build file:/Users/muktaaphale/github/HelloScala/)
[info] Creating docker image with name: 'muktaa/hello-scala':
[info] Sending build context to Docker daemon
[info] Step 0 : FROM dockerfile/java
[info] ---> 1126c85d8a06
[info] Step 1 : ADD /app/hello-scala_2.11-1.4-one-jar.jar /app/hello-scala_2.11-1.4-one-jar.jar
[info] ---> Using cache
[info] ---> 61871958f108
[info] Step 2 : ENTRYPOINT java -jar /app/hello-scala_2.11-1.4-one-jar.jar
[info] ---> Using cache
[info] ---> a8005b32ddc4
[info] Successfully built a8005b32ddc4
[info] Successfully built Docker image: muktaa/hello-scala
[success] Total time: 1 s, completed Mar 3, 2015 2:10:04 PM
~/github/HelloScala > docker images | grep hello-scala
muktaa/hello-scala latest a8005b32ddc4 12 hours ago 715 MB
~/github/HelloScala > docker run muktaa/hello-scala
Hello, world! #1
Hello, world! #2
Hello, world! #3
20
WhiteHedge
Docker Hub
Link:
https://registry.hub.docker.com/u/muktaa/hello-scala
Docker Registry
Automated Build in Docker:
https://registry.hub.docker.com/u/muktaa/helloscala-
automated-build/
21
WhiteHedge
Push JobsDo you need to push harder?
22
WhiteHedgePush Jobs
Knife-ssh
Journey from pull to push
be run against
nodes independently of a chef-
Job: set of commands to be run on node
• Docker pull
• Docker stop
• Docker run
23
WhiteHedge
Push Jobs
Use message bus (zeromq)
Claims to attack the scalability issue
Deployment status is relayed back
New born baby
Complex at the moment, ready with just the basic foundation
Knife SSH
Parallel ssh
SSH Protocol is slow and CPU hungry at scale
Feedback on deployment status is not as easy
Been in the market for long
Easy to use
How are Push Jobs different from knife-ssh?
24
WhiteHedgeChef Push Jobs Server
Enterprise Chef 11 or Chef server 12
Standalone or HA
Run the commands on Chef Server:
• chef-server-ctl install opscode-push-jobs-server
• opscode-push-jobs-server-ctl reconfigure
• chef-server-ctl reconfigure
25
WhiteHedgeSetup Workstation
Install knife push plugin
• Gem install knife-jobs
Knife cookbook site download push-jobs
Extract and save to your cookbook path
Edit the attributes file (push-jobs/attributes/default.rb)
• default['push_jobs']['package_url'] = 'https://opscode-private-
chef.s3.amazonaws.com/ubuntu/12.04/x86_64/opscode-push-jobs-client_1.1.5-1_amd64.deb'
• default['push_jobs']['package_checksum
Upload the push-jobs cookbook to your ChefServer
26
WhiteHedgeCreate Groups & Setup Node
Create 2 groups
• Pushy_job_writers
• Pushy_job_readers
Add user to the groups
Sudo chef-client -
From Workstation:
• Knife node status
• Knife node status <node-name>
27
WhiteHedgeRun
-client r recipe[run-docker -name>
my_node
Where docker.sh:
• Docker pull muktaa/hello-scala
• docker ps | grep muktaa/hello-scala| awk -
• Docker run muktaa/hello-scala
28
WhiteHedgeRetrospect
29
WhiteHedge
When Reality Strikes…If only applications were Hello World programs!
30
WhiteHedgeDocker Image
Application Configuration Docker Image
31
WhiteHedgeWhat is Configuration?
Packages Custom SetupsCredentials
Softwares Database
FilesEnvironment Specific Configuration
Ports
32
WhiteHedgeENVIRONMENTS
DEV
Docker Container
Docker Container
Docker Container
PRE PROD
Docker Container
Docker Container
Docker Container
PROD
Docker Container
Docker Container
Docker Container
33
WhiteHedgeSecure Credential Management
Unsolved problem with Docker today
Credentials inside docker containers
• Hard codes
• Set environment variables
34
WhiteHedgeWorkaround?
Create Base Image Manually, with configuration embedded
Build Tool uses the custom Base Image
Deploy using knife-ssh
35
WhiteHedge
Docker Chef CookbookTo manage docker images and deployment
36
WhiteHedgeDocker Cookbook
Available in Supermarket: https://supermarket.chef.io/cookbooks/docker
Install docker
Build docker image
Pull image and run container
Push docker image to registry
LWRPs
• Docker_container
• Docker_image
• Docker_registry
https://github.com/bflad/chef-docker/blob/master/README.md
37
WhiteHedgeCredential Management
secret = Chef::EncryptedDataBagItem.load_secret
@docker_cred = Chef::EncryptedDataBagItem.load
(
node['docker']['creds']['databag'],
node['docker']['user'],
secret
)
docker_registry ‘https://registry.hub.docker.com/u/muktaa/hello-scala/’ do
email docker_cred['email']
username docker_cred['username']
password docker_cred['password']
end
38
WhiteHedgeDocker_image
# Build a docker image using docker_image
resource
docker_image node['docker']['image'] do
tag node['docker']['image']['tag']
source '/var/docker'
action :build
end
# Push the image to docker registery
docker_image node['docker']['image'] do
action :push
end
# Delete the image from the machine
docker_image node['docker']['image'] do
action :remove
end
39
WhiteHedgeDocker_container
# Run Container
docker_container ‘muktaa/hello-scala’
detach true
port ‘8081:8081’, ‘8085:8085’
env ‘ENVIRONMENT=pre-prod’
volume ‘/mnt/docker/docker-storage’
action :run
end
40
WhiteHedgeGENERATE DOCKERFILE
# Generate a docker file using template.
template "#{node['docker']['directory']}/Dockerfile" do
source 'dockerfile.erb'
variables image: node['docker']['base']['image']['name'],
maintainer: @docker_cred['maintainer'],
email: docker_cred['email'],
build_cmd: node['docker']['build']['commands'],
entry_point: node['docker']['build']['entry_point']
action :create
end
41
WhiteHedgeWORKFLOW
Build Application Save the Artifact to a Repository Manager
Build Docker Image Docker cookbook would build and save the docker image
Deploy Docker cookbook runs the container on the nodes
42
WhiteHedge
Chef ContainersContains Awesome.
43
WhiteHedgeWhat is a Chef Container?
Package
Provides Configuration Management for containers
44
WhiteHedgeChef Container Components
chef-client
runitchef-init
45
WhiteHedgeWhy Chef Containers?
Bootstrap chef-client without SSH connection
Manage multiple services inside your container
Manage running state of your container
Consistency across Architectures
Mixed Architecture Applications
46
WhiteHedgeBest Suited For
Transitioning traditional architecture to containers
Handling last mile configuration when container boots
Getting the best of two worlds without complexity
47
WhiteHedgeKnife container docker init
Gem install knife-container
knife container docker init NAMESPACE/IMAGE_NAME [options]
• -f base docker image (default is ubuntu 12.04) - chef container should be already installed on it
• -r runlist
• -z chef client local mode
• -b use berkshelf
48
WhiteHedgeEXample
$ sudo knife container docker init muktaa/hello-scala-cc
Compiling Cookbooks...
Recipe: knife_container::docker_init
* directory[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc] action create
* template[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/Dockerfile] action create
- update content in file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/Dockerfile from none to 943017
- * template[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/.dockerignore] action create
- create new file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/.dockerignore
- update content in file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/.dockerignore from none to e3b0c4
* directory[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef] action create
- create new directory /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef
* template[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/client.rb] action create
- create new file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/client.rb
- update content in file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/client.rb from none to 7de61f
* file[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/first-boot.json] action create
- create new file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/first-boot.json
- update content in file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/first-boot.jsonfrom none to 5269ef
* template[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/.node_name] action create
- create new file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/.node_name
- update content in file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/.node_name from none to 4764d2
* template[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/Berksfile] action create (skipped due to only_if)
* directory[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/secure] action create
- create new directory /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/secure
* file[/home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/secure/validation.pem] action create
- create new file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/secure/validation.pem
- update content in file /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc/chef/secure/validation.pem from none to ec1f3e
- change mode from '' to '0600'
Downloading base image: chef/ubuntu-12.04:latest. This process may take awhile...
Tagging base image chef/ubuntu-12.04 as muktaa/hello-scala-cc
Context Created: /home/ubuntu/chef-repo/dockerfiles/muktaa/hello-scala-cc
49
WhiteHedgeKnife container docker build
run command docker images
knife container docker build
• resolve docker dependencies
• build docker image
• cleanup chef artifacts
50
WhiteHedgeEXAMPLE
$ sudo knife container docker build muktaa/hello-scala-cc
Sending build context to Docker daemon 9.728 kB
Sending build context to Docker daemon
Step 0 : FROM muktaa/hello-scala-cc
---> 50d3c5c9e133
Step 1 : ADD chef/ /etc/chef/
---> 4933cc9e13e0
Removing intermediate container da0a08413a91
Step 2 : RUN chef-init --bootstrap
---> Running in add27db609cc
[2015-03-31T21:44:44+00:00] INFO: Starting Supervisor...
[2015-03-31T21:44:44+00:00] INFO: Supervisor pid: 9
[2015-03-31T21:44:49+00:00] INFO: Starting chef-client run...
[2015-03-31T21:44:50+00:00] INFO: Forking chef instance to converge...
[2015-03-31T21:44:50+00:00] INFO: *** Chef 11.16.2 ***
[2015-03-31T21:44:50+00:00] INFO: Chef-client pid: 16
[2015-03-31T21:44:53+00:00] INFO: Client key/etc/chef/secure/client.pem is not present - registering
[2015-03-31T21:44:53+00:00] INFO: HTTP Request Returned 404 Object Not Found: error
[2015-03-31T21:44:54+00:00] INFO: Setting the run_list to [] from CLI options
[2015-03-31T21:44:54+00:00] INFO: Run List is []
[2015-03-31T21:44:54+00:00] INFO: Run List expands to []
[2015-03-31T21:44:54+00:00] INFO: Starting Chef Run for muktaa-hello-scala-cc-build
[2015-03-31T21:44:54+00:00] INFO: Running start handlers
[2015-03-31T21:44:54+00:00] INFO: Start handlers complete.
[2015-03-31T21:44:55+00:00] INFO: Loading cookbooks []
[2015-03-31T21:44:55+00:00] WARN: Node muktaa-hello-scala-cc-build has an empty run list.
[2015-03-31T21:44:55+00:00] INFO: Chef Run complete in 1.121705004 seconds
[2015-03-31T21:44:55+00:00] INFO: Running report handlers
[2015-03-31T21:44:55+00:00] INFO: Report handlers complete
[2015-03-31T21:44:55+00:00] INFO: Sending resource update report (run-id: 6f637baf-18cc-4620-b3e2-9afc90e8cd6b)
---> 2c2ec6fab1ef
Removing intermediate container add27db609cc
Step 3 : RUN rm -rf /etc/chef/secure/*
---> Running in 30a3611b083f
---> cab28d6eed90
Removing intermediate container 30a3611b083f
Step 4 : ENTRYPOINT ["chef-init"]
---> Running in 0a9f4e96bbf7
---> a8577b66b103
Removing intermediate container 0a9f4e96bbf7
Step 5 : CMD ["--onboot"]
---> Running in f9a444817229
---> 21b3800bc9b3
Removing intermediate container f9a444817229
Successfully built 21b3800bc9b3
51
WhiteHedgeDocker images
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
muktaa/hello-scala-cc latest 21b3800bc9b3 2 hours ago 311.9 MB
<none> <none> b343c8301cc8 2 hours ago 311.9 MB
chef/ubuntu-12.04 latest 50d3c5c9e133 6 months ago 311.9 MB
$ sudo docker push muktaa/hello-scala-cc
$ sudo docker –d run muktaa/hello-scala-cc
52
WhiteHedge
Our StoryProduct under Development.
Super Cool DevOps Culture.
53
WhiteHedge
54
WhiteHedgeLessons Learnt
Running apps in containers is easy
Debugging apps in containers is difficult
You can very well run multiple services inside a docker container
Ah the woes of Docker networking!
Sequential Progression
Related Documents
![Assorted Baking CupsAssorted Baking Cups - Pastry Pro Cup [NP-2016-09-09].pdf · Assorted Baking CupsAssorted Baking Cups PET Laminated Baking Cup Paper Baking Cup Food grade and](https://static.cupdf.com/doc/110x72/5a9dbea27f8b9abd0a8c98bb/assorted-baking-cupsassorted-baking-cups-pastry-cup-np-2016-09-09pdfassorted.jpg)
