Play Framework + Docker + CircleCI + AWS + EC2 Container Service

Play Framework + Docker + CircleCI + AWS =

An Automated Microservice Build Pipeline

Josh Padnick Wednesday, November 11, 2015

[email protected] @OhMyGoshJosh

What do I want out of a Java-based microservices

infrastructure?

Java-Based

• Java-based (or modern hipster JVM language)

• No Java EE

• Reload without compile (e.g. refresh the browser)

• Native support for JSON, REST, and Websockets

• Supports “reactive” mindset (async, non-blocking, etc.)

Microservices Infrastructure

• A Universal unit of deployment (i.e. Docker)

• Continuous integration

• Continuous deployment

• Ability to run multiple containerized services on the same VM

• Simple setup

• Long-term scalability

• Minimal “undifferentiated heavy lifting”

Does this mythical beast exist?

Actually, it’s increasingly less mythical.

PadnickJosh

• Full-stack engineer for 12+ years

• Professional AWS & DevOps Guy via Phoenix DevOps

• Experienced Java programmerLover of Scala Favorite Web Framework is Play Framework

• [email protected]@OhMyGoshJosh

DevOps & AWS• I wrote a 12,000+ word article on building

scalable web apps on AWS at https://goo.gl/aD6gNC

• See JoshPadnick.com for prior DevOps & AWS presentations.

• Interested in getting in touch? Contact me via PhoenixDevOps.com.

Today’s talk is about putting together a quick but scalable solution for

this problem.

First we’ll cover the big picture concepts.

Then we’ll show it working.

We’ll end by talking about how it could be even better.

Let’s start with the world’s most generic

build pipeline

VCS

Developer commits code to Version Control System.

VCS

VCS notifies Build Server we have a new build.

Build Server

VCS

Build Server- build/compile - run the “fast” automated tests - prepare a deployment artifact

Build Server

VCS Build Server

Build server pushes deployment artifact to artifact repository.

Artifact Repository

VCS Build Server

We’d like to do Continuous Deployment. So let’s assume this was a deployable commit. We immediately deploy the artifact.

Artifact Repository

VCS Build Server Artifact Repository

Deploy to infrastructure.

Now let’s pick our technologies.

Docker Hub

Developer commits code to GitHub.

Options• GitHub

De facto source control system.

• BitBucketHosted but more enterprisey. Theoretical tighter integration with other Atlasssian tools.

• AWS CodeCommitNo fancy UI but fully hosted git repo in AWS.

GitHub uses web hooks to automatically kick off a build in CircleCI.

Options• CircleCI

Hosted build tool. Awesome UI. Get up and running in an hour or less. But no first-class support for Docker.

• Travis Hosted build tool. Built on Jenkins behind the scenes. Comparable to Circle. More expensive.

• Shippable First-class Docker support, but clunky UI. Fast and customizable. Use your own Docker container for your build environment!

• JenkinsThe self-hosted stalwart. Medium overhead in exchange for maximum customizability.

Docker Hub

Circle will: - build/compile - run automated tests - build a docker image - push image to Docker Hub

Options• Docker Hub

The “official” place to house Docker registries. Free for public repos; paid for private. Poor UI, sometimes goes down. Easiest integration with rest of Docker ecosystem, but easy to switch to another repo.

• Amazon EC2 Container Registry (ECR)AWS’s private container registry service. Looks like a winner. Coming out by end of year. Unless Amazon really screws up, obvious alternative to Docker Hub.

• Google Cloud Registry (GCR)Mature, solid solution. Lowest pull latencies with Google Cloud Engine, but usable anywhere.

• Quay Early docker registry upstart with superior UX. Acquired by CoreOS. Solid solution, but probably not as compelling as AWS ECR.

Docker Hub

Docker Hub

Deploy to AWS.

Options• Let’s just assume all AWS for now.

Options within AWS• AWS EC2 Container Service (ECS)

Amazon’s solution for running multiple services on a single VM in docker. Not perfect, but does an excellent job of being easy to setup and start using right away.

• AWS Elastic Beanstalk AWS’s equivalent of Platform-as-a-Service. Works great when using one Docker container per VM, and meant to be scalable, but eventually you’ll want more control over your infrastructure.

• Roll Your Own Use a custom method to get containers deployed on your VMs.

• Container Framework Use a framework like CoreOS+Fleet, Swarm, Mesos, Kubernetes or Nomad.

• Container Framework PaaSUse a pre-baked solution like Deis or Flynn. Or a tool like Empire that sits on top of ECS.

What about our app code?

Let’s talk about it.

• Re-architected the web framework from scratch.

• Nice dev workflow

• Young enough to be hipster; mature enough to be stable

• Solid IDE support (IntelliJ)

• Non-blocking / async

• Outstanding performance

• Designed for RESTful APIs

Live Demo

Now let’s build up our build pipeline live.

Step #1: Create a base docker image

• We may have many different microservices using Docker.

• A common base image = standardization

• See my base docker image at: https://github.com/PhoenixDevOps/phxjug-ctr-base

• # BUILD THE BASE CONTAINER cd /repos/phxdevops/phxjug-ctr-base docker build -t "phxdevops/phxjug-ctr-base:3.2" .docker push “phxdevops/phxjug-ctr-base:3.2"

• NOTE: You won’t have rights to push to my repo. So replace this with your own Docker Hub repo.

Step #2: Create a base Play

Framework docker image

• We may have many different microservices using Play.

• Also, one of Play’s downsides is that Activator (which is really just a wrapper around SBT) uses Ivy for dependencies, and it is painfully slow on initial downloads.

• If we create a Docker image with all our dependencies pre-downloaded, our docker build times will be MUCH faster.

• Even if some of our dependencies are off, it’s not a big deal. The point is that we’ll get most of them here.

• See my base docker image at:https://github.com/PhoenixDevOps/phxjug-ctr-base-play

• # BUILD THE BASE PLAY CONTAINER cd /repos/phxdevops/phxjug-ctr-base-play docker build -t "phxdevops/phxjug-ctr-base-play:2.4.3" .docker push "phxdevops/phxjug-ctr-base-play:2.4.3"

• NOTE: You won’t have rights to push to my repo. So replace this with your own Docker Hub repo.

Step #3: Take our Play app and build

a Docker image out of it.

• SBT includes a “dist” plugin that will create an executable binary for our entire Play app!

• We’ll run that and make that the process around which the Docker container executes.

• See my image at: https://github.com/PhoenixDevOps/phxjug-play-framework-demo

• Note that this is a standard Play app with a Dockerfile in the root directory. “docker build” takes care of the rest.

• # BUILD A PLAY APP IN A CONTAINER cd /repos/phxdevops/phxjug-play-framework-demodocker build -t "phxdevops/phxjug-play-framework-demo:demo"docker push "phxdevops/phxjug-play-framework-demo:demo"

• NOTE: You won’t have rights to push to my repo. So replace this with your own Docker Hub repo.

Step #4: Define our ECS

Infrastructure in AWS.

Options• Point and click around the AWS Web Console

Good for learning. Bad for long-term maintainability

• AWS CloudFormationAWS’s official “infrastructure as code” tool. Pretty stable and mature, but painfully slow to work with, and JSON format gets too verbose.

• Terraform A brilliant achievement of infrastructure as code tooling! But still suffers from some bugs. You can work around them once you get the hang of it, or with guidance from experienced hands.

• AnsibleOffers similar tool, but doesn’t compare in sophistication to CloudFormation or Terraform.

Our Choice• We’ll use terraform.

• To save time, I’ve already provisioned the infrastructure for today.

• But you can see the entire set of Terraform templates I used to create my ECS cluster at https://github.com/PhoenixDevOps/phxjug-ecs-cluster

Wait, how does ECS work?

Key ECS Concepts• Cluster

• Container Instances

Cluster

Container Instance

Container Instance

Container Instance

Key Concepts• Task Definitions

Task Definitions• JSON object

• Describes how 1 or more containers should be run and possibly links Container A to Container B.

• You can also use Docker Compose yml files as an alternative to the proprietary ECS JSON format.

Components of a Task Definition

• Task Family Name (e.g. “MyApp”)

• 1 or more container definitions:

• docker run command + args

• Resource requirements (CPU, Memory)

Deploying new versions of your app

• All your app’s versions are individual “Task Definitions” within a “Task Definition Family”

• Each time you need to deploy a new version of your app, you’ll need a new Docker image with a new tag. Then just create a new Task Definition that points to the new Docker image.

• ECS handles deployment for you, but there are some pitfalls here.

Key Concepts• Task Definitions

• Task Definition Families

• Tasks

• Services

Task DefinitionTask

Task

Task

Task

Tasks• An “instance” of a Task Definition is a Task.

• Really, this just means a single Docker container (or a “group” of Docker containers if the Task Definition specified more than one Docker image).

Tasks as Services• Should your task always remain running?

• Should it be auto-restarted if it fails?

• Might it need an ELB?

• Then you want to run your Task Definition as a Service!

Tasks and Services• Note that the same Task Definition…

• …can be used to run as a one-time Task

• …or a long-running Service.

• That’s because Task Definitions are really just definitions of Docker containers and how they should run. It doesn’t “know” anything else about the container itself.

Task

Task

Task

Task

Task

ECS Pro’s• Very little to manage.

• Built-in service discovery, cluster state management, and container scheduler.

• Allows for resource-aware container placement.

• Container scheduling is pluggable.

• Fully baked GUI that allows you to learn/do most anything.

• Tolerable learning curve.

• Supported by Amazon.

• Feel free to build your own service discovery!

ECS Con’s• Default Service Discovery:

One ELB per service = $18/service per month —> potentially expensive

• Less flexible on deployments than you’d like.

• Lacks the power of a more general purpose “data center operating system” such as Mesos or Kubernetes.

Live Clickthrough

Step #5: Configure our Play app

for circle.yml

• See https://github.com/PhoenixDevOps/phxjug-play-framework-demo/blob/master/circle.yml

Now let’s see it all live in action!

Q&A