Amazon EC2 Container Service: Deep Dive

Daniele Stroppa

AWS Solutions Architect

@moviolone

Amazon EC2 Container Service:

Deep Dive

Agenda

• Cluster management

• Container scheduling

• Container deployment

• Scaling Amazon ECS

• Logging & Monitoring

• Service discovery

Cluster management

Cluster management with Amazon ECS

• Management of followers via ECS Agent

• Dispatching of sub-tasks to proper location

• Cluster state inspection

http://amzn.to/1jlHvnU

Cluster management under the hood

• Paxos-based transactional journal based data store

• Writes are committed as transaction in the journal with

order-based ID. The current value is the sum of all

transactions made as recorded by the journal.

• Reads are simply a snapshot in time of the journal. For a

write to succeed, the write proposed must be the latest

transaction since the last read.

http://bit.ly/1M9gGiv

Container scheduling

Amazon ECS Schedulers

• Task scheduler

• Run tasks once

• Batch jobs

• Service scheduler

• Load balancing

• Health management

• Scale-up and scale-down

• Update management

• AZ aware

Custom Schedulers

1. Calls the ECS List* and Describe* API operations to

determine the current state of the cluster.

2. Selects one (or more) container instances according to

the logic implemented.

3. Calls StartTask API to start a task on the selected

container instance.

http://amzn.to/1L9jmHT

Integration with Apache Mesos schedulers

• Allows you to use Apache Mesos schedulers like

Marathon and Chronos with ECS

• The ECSSchedulerDriver interprets the command given

when scheduling jobs with Mesos and starts a task with

TaskDefinition family:revision

http://bit.ly/1WXd6L3

Container deployment

CI/CD Partners

Continuous delivery with Jenkins

Build image

Push image

Update service

Continuous delivery with Jenkins

• Webhook to trigger job execution

• Docker Build and Publish plugin to build Docker image

• Amazon ECS API to update service

• Bonus – Cloudbees ECS plugin to run slave

http://amzn.to/1GbheTp

Amazon ECS CLI

• Easily create Amazon ECS clusters & supporting

resources such as EC2 instances

• Run Docker Compose configuration files on Amazon

ECS

• Available today – http://amzn.to/1jBf45a

Deploy Compose app with Amazon ECS CLI

> ecs-cli compose up

> ecs-cli compose ps

> ecs-cli compose service create

> ecs-cli compose service start

http://amzn.to/1jBf45a

Scaling Amazon ECS

Scaling with Amazon ECS CLI

> ecs-cli scale

> ecs-cli compose scale

> ecs-cli compose service scale

http://amzn.to/1jBf45a

Scaling with CloudWatch and AWS Lambda

http://amzn.to/1QbuG9Y

Logging & monitoring

Logging and monitoring on Amazon ECS

• Amazon CloudWatch Logs

• Amazon CloudTrail

• Partners & third party tools

Logging with Amazon CloudWatch Logs

• Centralized logs

• Collect, set up metrics, alarms

• View as graphs

• No need to modify your containers

• One additional container:

• rsyslog

• CloudWatch Logs agent

Logging Amazon ECS API with AWS CloudTrail

{

"eventVersion": "1.03",

"userIdentity": {…},

"eventTime": "2015-10-12T13:57:33Z",

"eventSource": "ecs.amazonaws.com",

"eventName": "CreateCluster",

"awsRegion": "eu-west-1",

"sourceIPAddress": "54.240.197.227",

"userAgent": "console.amazonaws.com",

"requestParameters": {

"clusterName": "ecs-cli"

},

http://amzn.to/1LjJT8N

Logging Amazon ECS API with AWS CloudTrail

"responseElements": {

"cluster": {

"clusterArn": "arn:aws:ecs:eu-west-1:560846014933:cluster/ecs-cli",

"pendingTasksCount": 0,

"registeredContainerInstancesCount": 0,

"status": "ACTIVE",

"runningTasksCount": 0,

"clusterName": "ecs-cli",

"activeServicesCount": 0

}

},

[…]

http://amzn.to/1LjJT8N

Monitoring Amazon ECS with Datadog

http://bit.ly/1R723Lm

Monitoring Amazon ECS with Sysdig Cloud

http://bit.ly/1jrmvvD

Service discovery

Service discovery on Amazon ECS

• Amazon ECS Service Scheduler

• Amazon Route 53 private zone

• Partners & third party tools

Service Discovery with Amazon Route 53

http://bit.ly/1IjrCDj

Service Discovery with Weaveworks

http://bit.ly/1LkRjJ9

Service Discovery with Consul

http://amzn.to/1JZL5gz

Daniele Stroppa

AWS Solutions Architect

@moviolone

Thank you!