@bridgetkromhout
Docker in Production Reality, Not Hype
Bridget Kromhout
@bridgetkromhout
Bridget Kromhout
Operations Engineer @DramaFeverMinneapolis, Minnesota
@devopsdays@devopsdaysMSP
@arresteddevops
bridgetkromhout.com
@bridgetkromhout
K-dramas since 2009. Docker in prod since October 2013.
@bridgetkromhout
Why Docker?
consistent development repeatable deployment
How?
not: a tutorial
but: repeatable
@bridgetkromhout
private registry: the official party line
https://github.com/docker/docker-registry#quick-start
S3 is a storage engine optionbut… a central registry server didn’t scale well for us
@bridgetkromhout
private registry: dramafever
@bridgetkromhout
# this goes in /etc/default/docker to control
docker's upstart config
DOCKER_OPTS="--graph=/mnt/docker --insecure-
registry=local-repo-alias.com:5000"
● local-repo-alias.com in DNS with A record to 127.0.0.1● OS X /etc/hosts: use the boot2docker host-only network IP
registry upstartdocker pull public_registry_image
docker run -p 5000:5000 --name registry \
-v /etc/docker-reg:/registry-conf \
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml \
public_registry_image
@bridgetkromhout
config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true
@bridgetkromhout
what even is flate?!Pulling repository local-repo-alias.com:5000/www4dda2b433370: Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 flate: corrupt input before offset 54393671d497ad3926c8: Error downloading dependent layers2014/12/07 02:34:54 Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671
@bridgetkromhout
registry rewrite coming!
DOCKER_OPTS="--graph=/mnt/docker --insecure-registry=local-repo-alias.com:5000 -e STORAGE_REDIRECT=true"
...until we get to the promised go lan(d|g), there’s a workaround for the flate errors we’re seeing:
@bridgetkromhout
Achievement unlocked:distributed privateDocker registry
@bridgetkromhout
@bridgetkromhout
Next up:build pipeline
starringeveryone’s
favorite butler
weekly base builds
FROM local-repo-alias.com:5000/www-base
● include infrequently-changing dependencies○ ubuntu packages○ pip requirements○ wheels
● other builds can start from these images (so they’re faster):
@bridgetkromhout
sudo docker build -t="a12fbdc" .
sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 --
name test.a12fbdc a12fbdc py.test -s
sudo docker tag a12fbdc local-repo-alias.com:
5000/www:'dev'
sudo docker push local-repo-alias.com:5000/www:'dev'
@bridgetkromhout
www-master build
2014/10/30 21:35:31 Error getting container init rootfs b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f3108f86386aab67 from driver devicemapper: open /dev/mapper/docker-9:127-14024705-b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f3108f86386aab67-init: no such file or directorymake: *** [build-django] Error 1Build step 'Execute shell' marked build as failure
@bridgetkromhout
breaking builds
https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
@bridgetkromhout
@bridgetkromhout
Retry the build…...only if a specific regex appears
@bridgetkromhout
useful for unattended base builds
need to change how it reports to Slack
@bridgetkromhout
tag for staging
tag for prodout of ELBrestart upstartback in ELB
Ship it!
What about local development?
@bridgetkromhout
before summer 2014Vagrant for local development
chef-solo provisioner
17 minutes to install everything
@bridgetkromhout
now: boot2docker
devs pull down images built on jenkinsmysql image is built with fixturescan run master or qa image (or even prod)
can build new local images from Dockerfiles
@bridgetkromhout
local registry for devdocker run -d -p 5000:5000 --name
docker-reg -v ${DFHOME}:${DFHOME} -e
DOCKER_REGISTRY_CONFIG=${DFHOME}
/config/docker-registry/config.yml
public_registry_image
@bridgetkromhout
$ boot2docker ssh date -u
Mon Nov 24 16:09:02 UTC 2014
$ date -u
Tue Nov 25 01:43:49 UTC 2014
@bridgetkromhout
time is what turns kittens into cats
S3 requires clock sync$ docker pull local-repo-alias.com:5000/mysqlPulling repository local-repo-alias.com:5000/mysql2014/11/24 19:44:31 HTTP code: 500
$ boot2docker ssh sudo date --set \"$(env TZ=UTC date '+%F %H:%M:%S')\"
@bridgetkromhout
Devs can use their preferred editing environment:
-v ${DFHOME}/www:/var/www
We still want logs, too, so we expose those for the dev here:
-v ${DFHOME}/www/run:/var/log
volume mounting & our fork
@bridgetkromhout
Until 1.3 we ran a forked boot2dockerWe needed to mount local files into the VM
containerizing front-enduseful for building front-end apps on Jenkinsalso allows consistent testing
RUN apt-get install -y nodejs nodejs-legacy npmRUN npm install -g [email protected] npm install -g [email protected] npm install -g [email protected] bower.json /var/www/dependencies/bower.jsonRUN cd /var/www/dependencies && bower install --allow-root --config.interactive=false --force
@bridgetkromhout
@bridgetkromhout
django: image: local-repo-alias.com:5000/www:dev ports: - "8000:8000" links: - mysql - redis environment: - PYTHONPATH=/var/local - DJANGO_ENVIRON=LOCAL - DB_PORT_3306_TCP_ADDR=mysql command: /var/local/config/local/start-django-local volumes: - ${DFHOME}/www/run:/var/log - ${DFHOME}/www:/var/local
mysql: image: local-repo-alias.com:5000/mysql:dev expose: - "3306:3306"
for persistent instances# remove stopped containers
@daily docker rm `docker ps -aq`
# remove images tagged "none"
@daily docker rmi `sudo docker images | grep none
| awk -F' +' '{print $3}'`
@bridgetkromhout
failure modes
cron zombiesout of memory errors
race conditions
@bridgetkromhout
what isolation?-v /var/log/containers:/var/log
@bridgetkromhout
Host instances moving into
through a container darkly: monitoring
@bridgetkromhout
containers building (lighter) containers
easier with statically linked binaries
go microservicesandroid apk
@bridgetkromhout
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB
local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB
local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB
google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB
public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB
scratch latest 511136ea3c5a 18 months ago 0 B
$
ever-smaller images
@bridgetkromhout
@bridgetkromhout
www.dramafever.com/company/careers.html
Thank you!(and we’re hiring!)