Hello. Nice to meet you! Deploying Dockerized apps with Salt
Hello.Nice to meet you!Deploying Dockerized apps with Salt
1
Who am I?
Senior Developer @ SOON_
I'm @krak3n on GitHub, Twitter etc
Hi, I’m Chris
2
What is SOON_?
We make brands successful by combining insightful strategy, seamless technology, persuasive design and relevant content
We can help you be more digital** we don’t like the word “digital”, but it’s the word people use. Ask us and we’ll tell you why.
3
What is Do-It?
Do-It is the UK’s biggest volunteering network
Listing volunteering opportunities from thousands of charities and social action groups throughout the UK
Many of the opportunities on Do-it come from physical Volunteer Centres around the UK
4
The approach
Complete re-platform of the Java application. Start from scratch
Use modern technologies and frameworks
Separate Backend and Frontend builds for an API driven application
5
Docker 101
Wrapper around LXC* - method of running multiple isolated Linux systems
Run single processes
Ship your application with your OS
docker run --rm -it ubuntu:14.04 /bin/bashroot@7623d871ac08:/#
Easy to build and run applications anywhere (that supports docker)
* Linux Containers - Released 2006
Version control for LXC
$ docker run ubuntu:14.04 apt-get update$ docker ps -lCONTAINER ID IMAGE 9a0fa929a3c7 ubuntu:14.04 $ docker commit 9a0fa929a3c7 ubuntu:14.04
$ docker run apt-get install -y curl$ docker ps -lCONTAINER ID IMAGE97957263ea5c ubuntu:14.04$ docker commit 97957263ea5c ubuntu:14.04
$ docker run ubuntu:14.04 curl http://google.com
Build Images with Dockerfile$ cat ./DockerfileFROM ubuntu:14.04RUN apt-get update && apt-get install -y curl
$ docker run --rm ubuntu_with_curl curl http://google.com
$ docker build -t ubuntu_with_curl .
Push images to a central repository
$ docker login -e [email protected] -u you -p 123$ docker push ubuntu_with_curl
$ docker login -e [email protected] -u you -p 123 http://your.index.com$ docker push your.index.com/ubuntu_with_curl
Pull Images from a central repository.
$ docker pull ubuntu_with_curl$ docker run --rm ubuntu_with_curl curl http://google.com
Pass environment variables to containers$ docker run --rm -it -e FOO=foo ubuntu:14.04 bashroot@2281fb4f13a4:/# echo $FOOfoo
Run containers in detached mode
$ docker run -d foo ubuntu:14.04 foo
You can name containers
$ docker run -d --name foo ubuntu:14.04 some_process
Then you can stop, start and restart your contains like processes
$ docker stop foo$ docker start foo$ docker restart foo
View the logs of your containers
$ docker logs --follow foo
Or attach* to running containers$ docker attach foo$ docker start -a foo* does not allow you to execute commands inside the container
Execute commands in running containers - useful for debugging$ docker run -d --name foo ubuntu:14.04 foo$ docker exec -it foo /bin/bashroot@4cc94bc02b5f:/# ps aux
Remove containers
$ docker rm foo$ docker rm -f foo
Also remove images$ docker rmi ubuntu:14.04$ docker rmi -f ubuntu:14.04
Containers can be linked with each other$ docker run -d -e RABBITMQ_USER=chris -e RABBITMQ_PASS=chris --name rabbitmq tutum/rabbitmq
$ docker run --rm -it --link rabbitmq:rabbitmq ubuntu:14.04 /bin/bash
$ root@507079e8b54a:/# echo $RABBITMQ_PORTtcp://172.17.0.19:5672
Sharing data with volumes
$ docker run --rm -it \ -v /host/path:/container/path/ foo ubuntu:14.04 \ /bin/bash
$ docker run --rm -it \ -v /host/path/file.x:/container/path/file.x foo \ ubuntu:14.04 /bin/bash
e.g your PostgreSQL data directory
$ docker run -v /data/psql:/var/lib/postgresql \ --name db orchardup/docker-postgresql
$ docker run --privileged --rm -it \ foo ubuntu:14.04 \ /bin/bash
6
A simple application(we will get to Salt SOON_)
Let's look at a simple Flask application we'll deploy later with Salt and Docker
/demo├── Dockerfile├── setup.py└── foo ├── __init__.py └── app.py
#!/usr/bin/env python# encoding: utf-8
from flask import Flaskapp = Flask(__name__)
@app.route('/')def hello_world(): return 'Hello World!'
if __name__ == '__main__': app.run(host='0.0.0.0')
Tree app.py
And the DockerfileFROM ubuntu:14.04RUN apt-get update && apt-get install -y python python-dev \ && apt-get clean \ && apt-get autoclean \ && apt-get autoremove -y \ && rm -rf /var/lib/{apt,dpkg,cache,log}/ADD . /fooWORKDIR /fooRUN python setup.py installEXPOSE 5000CMD ["python", "/foo/foo/app.py"]
Build it, Push it, Run It!
$ docker build -t soon/foo .$ docker push soon/foo$ docker run --rm -p 5000:5000 foo
7
Salt ♥ Docker
Before Salt can use docker we need to install docker and docker-Py
docker-ppa: pkgrepo.managed: - name: deb https://get.docker.io/ubuntu docker main - keyserver: hkp://keyserver.ubuntu.com:80 - keyid: 36A1D7869245C8950F966E92D8576A8BA88D21E9 - require: - pkg: software-properties-common - pkg: apt-transport-https
lxc-docker: pkg: - installed service.running: - name: docker - sig: /usr/bin/docker - require: - pkg: lxc-docker
docker-py: pip.installed: - reload_modules: True - require: - pkg: python-pip
We have already pushed soon/foo to Docker Hub*, using Salt, we can pull down that image
soon/foo: docker.pulled: - tag: latest - force: True - require: - pip: docker-py - service: lxc-docker
* this is a public repo - anyone can download it: docker pull soon/foo
docker-py must be installed
force: True - Always pull the latest image
Now we need to create the containerfoo-container: docker.installed: - name: foo - image: soon/foo - require: - docker: soon/foo
Then we can run the new containerfoo: docker.running: - container: foo - port_bindings: "5000/tcp": HostIp: "" HostPort: "5000" - require: - docker: foo-container
Simples! But thats not enough...
We also need to be able to stop and remove containers when the image has changed.
We can watch for changes to the image
foo-absent: cmd.wait: - name: docker rm -f foo - watch: - docker: soon/foo
We can’t use docker.absent
Like the private IP address of the container
$ salt-call docker.get_containers$ salt-call docker.get_containers inspect=True$ salt-call docker.inspect_container foo
Getting information about our running containers
"NetworkSettings": { "IPAddress": "10.1.0.13", "Gateway": "10.1.42.1", "Ports": { "5000/tcp": null }}
Now we can set up an Nginx* config to proxy directly to the container* assumes you have Nginx installed via Salt
foo-nginx-config: file.managed: - name: /etc/nginx/conf.d/foo.conf - source: salt://foo.nginx.conf - template: jinja - watch_in: - service: nginx - require: - docker: foo
In our foo nginx config, we can get the IP of our foo container and proxy directly to it without the need for port binding
{% set foo = salt['docker.inspect_container']('foo')['out'] %}{% set ip = foo['NetworkSettings']['IPAddress'] %}
server { listen 80; server_name foo.com; location / { proxy_pass http://{{ ip }}:5000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $remote_addr; }}
We also need to clean up after ourselves
$ docker rmi $(docker images -q -f dangling=true)
This only needs to be run if the image changed and after the container was removed
cleanup: cmd.wait: - name: docker rmi $(docker images -q -f dangling=true) - watch: - cmd: foo-absent
We can also run multiple containers of the same image{% for x in range(1, 5) %} # 4 containers
foo-{{ x }}-absent: cmd.wait: - name: docker rm -f foo-{{ x }} - watch: - docker: soon/foo - watch_in: - cmd: cleanup
foo-{{ x }}-container: docker.installed: - name: foo-{{ x }} - image: soon/foo - require: - docker: soon/foo - watch: - cmd: foo-{{ x }}-absent
foo-{{ x }}: docker.running: - container: foo-{{ x }} - require: - docker: foo-{{ x }}-container - require_in: - file: foo-nginx-config
{% endfor %}
Tell nginx how many containers we run
foo-nginx-config: file.managed: - name: /etc/nginx/conf.d/foo.conf - source: salt://foo3.nginx.conf - template: jinja - context: no_containers: 4 - watch_in: - service: nginx
Add an upstream backend to nginx to proxy to our 4 containersupstream backend {{% for x in range(1, no_containers + 1) -%}{%- set foo = salt['docker.inspect_container']('foo-' + x|string)['out'] -%} server {{ foo['NetworkSettings']['IPAddress'] }}:5000;{% endfor -%}}
server { listen 80; server_name foo.com; location / { proxy_pass http://backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $remote_addr; }}
Be careful with docker logs
They are not rotated and everything from STDOUT and STDERR ends up in them
They are only cleaned out when you remove a container
https://github.com/docker/docker/issues/7333
Set your app to log to a centralised log store, like LogStash
Private docker repositories for your secret source
docker-registries: https://index.docker.io/v1/: email: [email protected] password: 124 username: foo
Just add this to your pillars
8
Bonus Round
Continuous delivery with CircleCImachine: services: - docker environment: REPO: soon/foo TAG: $(sed 's/master/latest/;s/\//\-/' <<<$CIRCLE_BRANCH)
dependencies: pre: - sed "s/<EMAIL>/$DOCKER_EMAIL/;s/<AUTH>/$DOCKER_AUTH/" < .dockercfg.template > ~/.dockercfg override: - docker build -t $REPO:$TAG .
test: override: - docker run -it --name test --net=host $REPO:$TAG python setup.py test
deployment: release: branch: master commands: - docker push $REPO:$T AG
Salt REST API to trigger state runsrest_cherrypy: port: 8000 host: 127.0.0.1 disable_ssl: True webhook_disable_auth: True
$ service salt-api start
Add reactor events to react to webhook callsreactor: - 'salt/netapi/hook/circleci/success': - /srv/salt/reactor/deploy.sls
Start the Salt API Service
Create a reactor sls to handle the event
def run(): ret = {} ret['deploy'] = { 'cmd.state.highstate': [ {'tgt': '*'}, ] } return ret
Send the event by hitting the webhook urlcurl -sS -k http://domain.com/hook/circleci/success
How about Slack notifications?import jsonimport urllibimport urllib2
def slack(message, color=None): attachment = { "text": message, } if color: attachment['color'] = color payload = { "channel": "#your-channel", "attachments": [attachment] } payload = urllib.pathname2url(json.dumps(payload)) payload = 'payload=' + payload request = urllib2.Request('https://slack.hook', payload) urllib2.urlopen(request)
... slack('Deploying...')... return ret
And update the CircleCI config
deployment: release: branch: master commands: - docker push $REPO:$TAG - curl -sS -k https://domain.com/hook/circleci/success
9
Questions?
Thank you! :)