Docker in production at the Aurora team
Or: Now that we found docker what are we going to do with it?Timisoara Docker Meetup, Sept 4th 2017
Adina-Claudia Toma, Victor Dan Daneasa, Martin Danielsson
AgendaWho are we and why are we allowed to talk about this?
Our journey towards Docker in Production
What do you need for it?
How do you know it’s working?
Q & A
Who are we?
Aurora Project (iDesk2)Research Database for Lawyers,
Tax AccountantsLive with docker in Production
since MarchStill in transition towards
Microservices
Haufe Group~1600 employees in multiple countriesHQ in Freiburg, GermanyDevelopment Center in Timisoara
Adina-Claudia TomaSenior Developer
Victor Dan DaneasaSenior
Developer
Martin DanielssonSolution Architect
Who are we?
Moving from tinkering to production
Feels familiar?What’s this? $ docker pull postgres
$ docker run -d -p 5432:5432 postgres
AMAZEBALLS!I’m totally writing everything for docker!
A single VM will do. Right?
Dammit.
How hard can it be to get into production...
The Dev to Prod ChasmAmazeballs
factor
TraditionalOps Level Dev Tinkering Dev Process
SetupMature Production OpsIncl. CI/CD
Production Rollout
Building Blocks
● Hardware/Physical servers● OS● VM● Docker containers● Application
Levels of Abstraction
Building Blocks for running Docker in productionWhat you have to do by yourself:
● Image management● Container Orchestration● Automated CI/CD Pipelines● Log management● Monitoring on all levels● Data Persistence
What you can get for “free” if you use a cloud provider and orchestration framework:
● Security patches & restricted network access● Load balancing & service discovery● Automatic recovery from failure
Image Management● Consistent process to build and tag docker images● Private Docker image repository
○ Artifactory (JFrog)○ Azure Container Registry (ACR)○ Amazon EC2 Container Registry (ECR)○ Self-hosted with Docker○ Docker Hub○ Quay.io
● Security scanning of docker images for vulnerabilities
Container OrchestrationAbstracts the host infrastructure & allows to treat a cluster as a single deployment target
● Declarative configuration● Scheduling & high-availability● Service discovery & load-balancing● Health monitoring
Container Orchestration Tools
Providers
Automated CI/CD pipelines● Infrastructure as code● Reproducible environments● Automated tests● No downtime deployments:
○ Rolling updates○ Blue/green deployments○ Rollback on failure
Log Management
Data PersistenceContainers should be stateless.
State can be stored in:
● Data volumes per host -> non-portable between hosts● Shared filesystems: NFS, Ceph, GlusterFS● Docker volume plugins● Database/Storage as a service: AWS, Azure
our solution vectorWhat you have to do by yourself:
● Image management: private Haufe docker repository/Azure Container Registry● Container Orchestration: Kubernetes with Docker● Automated CI/CD Pipelines: Jenkins pipelines, bash, Ansible, Azure CLI● Log management: fluent-bit, fluentd, Graylog/Elasticsearch/Mongodb● Monitoring on all levels: Prometheus, Alertmanager, Grafana● Data Persistence: Postgres VM, NFS Server, Redis
What you can get for “free” if you use a cloud provider and orchestration framework:
● Azure Container Services Engine with Kubernetes● Security patches & restricted network access● Load balancing & service discovery● Automatic recovery from failure
Getting Application Insight (Or: How to not fly blind)
Prometheus● Whitebox monitoring● Scalable● Simple to setup● Discovery service● Built-in exporters (pull metrics)● Easy to integrate into your applications● PromQL (yet another query language)● Alerting included
Not fully blind And Getting better● Started with what we knew we need (the basics):
CPU, memory, IO
● Run into some problems:
Disk space, nodes failing, monitoring itself, API changes
● Things get better and better:
Alerting, app insights, moving parts
Overview
ApI Overview
Post MortemsResulted from a failure
Every member of the team participates
● What caused it?● What were the affected components?● Actions● Lessons learned
Tl;dr(Our) Conclusions
Trimmed for scale out
What we ended up with
Fully microservice enabled infrastructure
Insights on all levels
Full DevOps responsibility
Perhaps not what YOU need...Might single Docker Host be enough?
AWS Elastic Container Services?Docker DataCenter?k8s-as-a-service?
Traditional VMs?Google Container Engine?
Assess YOUR use case!
For us - absolutely worth the effort to
gain speed and flexibility
Invest only worth it with certain size and load
$Large upfront effort to get infrastructure
right
Your CI/CD pipelines are your safety net - make them rock solid
Practice provisioningdaily or weekly!
Steep learning curve If possible, start with something new, then move old workloads
Blue-eyed approach will fail - it is (a lot of)
work!
Many more moving parts - additional
complexity
Our conclusions and recommendations
Consider persistence early on
Q & A