Page 1
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
DMG202 - Zero to Sixty:
AWS OpsWorks
Thomas Metschke, Amazon
November 13th, 2013
Page 2
AWS OpsWorks
Helps you to model your entire application from
load balancers to databases starting from
templates for common technologies or building
your own.
You have full control of deployments, scaling,
monitoring, and automation of each component.
Page 3
AWS Application Management Services
Elastic Beanstalk OpsWorks CloudFormation EC2
Convenience Control
Higher-level services Do it yourself
Page 4
The Heart of AWS OpsWorks
Agent on each
EC2 instance
OpsWorks talks with
Page 5
The Heart of AWS OpsWorks
Agent on each
EC2 instance Understands a set of commands
that are triggered by OpsWorks.
The agent then runs Chef solo.
Page 6
The Heart of AWS OpsWorks
Agent on each
EC2 instance Understands a set of commands
that are triggered by OpsWorks.
The agent then runs Chef solo.
Page 7
The Heart of AWS OpsWorks
Agent on each
EC2 instance Understands a set of commands
that are triggered by OpsWorks.
The agent then runs Chef solo.
Page 8
AWS OpsWorks Agent Events
setup configure deploy undeploy shutdown
Page 10
A Stack with Layers
Page 11
A Stack with Layers and Instances
Page 12
Enough Talking
DEMO TIME
Page 13
• Created a new Stack from scratch
• Created an app and database layer
• Started instances in different Availability Zones
• Configured Auto Scaling
• Load balanced by Elastic Load Balancing
• Deployed an application
Recap of Demo
Page 14
Continuous Integration
Page 15
John Gaa VP of Engineering at BeachMint
Page 16
• Founded in 2011
• Next generation social commerce company
• Parent to: JewelMint, ShoeMint, StyleMint, & IntiMint
• Over 6 million registered customers
Page 18
Our Store Verticals are Powered By
• Amazon RDS
• Amazon ElastiCache
• Amazon SQS
• Amazon SNS
• Amazon CloudFront
• Amazon Redshift
RDS
ElastiCache
SQS
SNS
CloudFront
Redshift
Page 19
• Multiple releases a day
• 1 sysadmin
• 100+ instances between production
and test environments
Page 20
Jenkins & AWS OpsWorks to the Rescue
Page 21
Confidence to Release Every Day
• Unit tests
• Integration tests
• Load testing
• Manual testing
Page 22
What it Looks Like in AWS OpsWorks
Environments are represented as
stacks
Our different verticals are
represented as separate layers within
each stack
Page 23
Architecture Repeated in Each Environment
JewelMint IntiMint ShoeMint StyleMint
JavaScript widget
API
Checkout Registration Product Social
RDS Solr ElastiCache Drupal Drools
Layers
Sta
ck
Page 25
AWS OpsWorks in Our CI Processes
• Get instances associated to a layer
• Update code based on gittag version stored in
custom JSON
• Target the instances and run Chef scripts
Page 26
The Glue: Custom JSON
{
"environment":{
"env":"production"
},
"jewelmint":{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net"
}
},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{ "1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }
}
}
"environment":{
"env":"production”
},
Determines what
environment to be built
Page 27
The Glue: Custom JSON
"jewelmint"{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
Code version to be
deployed to a layer
{
"environment":{
"env":"production"
},
"jewelmint":{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net"
}
},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{ "1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }
}
}
Page 28
The Glue: Custom JSON
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net" }},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{
"1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }} Configuration and
attributes of
supporting systems
{
"environment":{
"env":"production"
},
"jewelmint":{
"gittag":"20131008d_release" },
"intimint":{
"gittag":"v3.7.0_release" },
"stylemint":{
"gittag":"v3.7.0_release" },
"shoemint":{
"gittag":"v3.7.0_release" },
"cdn":{
"media":{
"origin":"beachmint-origin-domain.com",
"domain":"beachmint-cloudfront.cloudfront.net"
}
},
"solr":{
"gittag":"master",
"server_name":"beachmint-solr-domain.com",
"nodes":{ "1":"node-ip-1",
"2":"node-ip-2",
"3":"node-ip-3" }
}
}
Page 29
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
use Aws\Common\Aws;
$aws = Aws::factory('./config.php');
$opsWorks = $aws->get('OpsWorks');
//Get the Stack we are updating
$onlineInstanceIds = array();
$allInstances = $opsWorks->describeInstances(array('LayerId' => $layerId))-
>get("Instances");
foreach($allInstances as $instance) {
if ($instance['Status'] == 'online') {
$onlineInstanceIds[] = $instance['InstanceId'];
}
}
Page 30
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
use Aws\Common\Aws;
$aws = Aws::factory('./config.php');
$opsWorks = $aws->get('OpsWorks');
$deployment = $opsWorks->createDeployment(array(
'StackId' => $stackId,
'Command' => array(
'Name' => 'execute_recipes',
'Args' => array(
'recipes' => $recipe
)
),
'InstanceIds' => $onlineInstanceIds
));
Page 31
Load Testing
• Code
• Configuration
• EC2 instance type
• System architecture configuration
• Capacity planning
Page 32
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
//Get the Stack’s custom JSON from OpsWorks
$customJsonArray = json_decode($theBuildStack[0]["CustomJson"]);
//Replace the gittag with the new tag
$customJsonArray->{"jewelmint"}->{"gittag"} = "thenewgittag";
//Convert back to string
$modifiedJson = json_encode($customJsonArray);
//Update Stack settings in OpsWorks
$updateRes = $opsWorks->updateStack(array("StackId" => "the-build-
stackid","CustomJson" => $modifiedJson));
Page 33
Commit
Jenkins post commit hook
Get instances
Update code on instances
Run selenium or unit tests
Stop
pass Ready for
QA Create new
tag Update git tag on custom json
Update code on instances
Run load test
Build
Sta
ck
Load Stack
Y
N
use Aws\Common\Aws;
$aws = Aws::factory('./config.php');
$opsWorks = $aws->get('OpsWorks');
$deployment = $opsWorks->createDeployment(array(
'StackId' => $stackId,
'Command' => array(
'Name' => 'execute_recipes',
'Args' => array(
'recipes' => $recipe
)
),
'InstanceIds' => $onlineInstanceIds
));
Page 34
Path to Production
• QA and Product group determines what goes to
production
• Each release candidate has a tag associated
with it
• QA updates QA and Stage environments using a
tool based on the AWS SDK
Page 35
Thanks!
www.jewelmint.com
www.stylemint.com
www.intimint.com
www.shoemint.com
Page 36
Automate Standard Operating Procedures
Page 37
3 AM – Alarm goes off
Page 38
Wouldn’t it be nice to have help?
Page 39
The idea: gather and ship logs to Amazon S3
as soon as the CPU load is to high
Page 40
We will use the
following setup
Page 41
AWS OpsWorks stores 1-minute
metrics in CloudWatch
Page 42
Every instance creates an
alarm for high CPU load
Page 43
CloudWatch alarm action:
write to SNS topic
Page 44
SNS publishes to
queue in SQS
Page 45
Watcher instance polls
SQS for notifications
Page 46
On alarm notification,
call OpsWorks API to …
Page 47
Execute a script on the
affected server
Page 48
Logs are gathered and
written to Amazon S3
Page 49
And Again
DEMO TIME
Page 50
Demo Recap
• Instances set up alarms on CloudWatch
• Alarms create notification in SNS
• SNS publishes to SQS queue
• Watcher instance polls queue and calls AWS
OpsWorks to execute recipe on instance
• The recipe runs a script to upload logs to S3
Page 51
More Information about AWS OpsWorks
• If not done yet, do the AWS OpsWorks lab!
• Find us in the AWS Booth
• Follow us on Twitter @AWSOpsWorks
• Find us on YouTube
• AWS OpsWorks survey
http://tinyurl.com/OpsWorksSurvey2013
Page 52
Other Talks During re:Invent
DMG304 - AWS OpsWorks Under the Hood
• Jonathan Weiss & Reza Spagnolo
• Thursday, Nov 14, 3:00 PM - 4:00 PM – Murano 3206
DMG305 - How Intuit Leveraged AWS OpsWorks as the Engine of Our PaaS
• Capen Brinkley & Rick Mendes of Intuit, Inc.
• Thursday, Nov 14, 4:15 PM - 5:15 PM – Murano 3206
Page 53
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
DMG202
Page 55
Recipes
• Default Writes Ruby script to the instance and runs it
• create_alarm Writes a Ruby script to create an alarm and ties it to the right SNS
topic
• send_logs Writes a Ruby script to the instance that can pack and ship the logs
to S3
Page 56
Watcher Code
queue = AWS::SQS.new.queues["<%= node[:watcher][:sqs][:url] %>"]
queue.poll(:initial_timeout => false) do |msg|
begin
alarm = JSON.parse(msg.body)['Subject'].start_with?('ALARM')
instance_id = JSON.parse((JSON.parse(msg.body)['Message']))['Trigger']
['Dimensions'].first['value']
if alarm
opsworks = AWS::OpsWorks.new.client
deployment = opsworks.create_deployment(
:stack_id => "<%= node[:opsworks][:stack][:id] %>",
:instance_ids => [instance_id],
:command => {
:name => 'execute_recipes',
:args => {'recipes' => <%= node[:watcher][:execute_recipes] %>}
},) end end end
Page 57
Alarm Creation
id = "<%= node[:opsworks][:instance][:id] %>"
action ="<%= node[:watcher][:sns][:arn] %>"
alarm = AWS::CloudWatch.new.alarms.create("#{id}_cpu_idle", {
:namespace => 'AWS/OpsWorks',
:metric_name => 'cpu_idle',
:dimensions => [{:name => 'InstanceId', :value => id}],
:comparison_operator => 'LessThanOrEqualToThreshold',
:evaluation_periods => 1,
:period => 60,
:statistic => 'Average',
:threshold => 20,
:actions_enabled => true,
:alarm_actions => ["#{action}"],
:alarm_description => 'watching the available CPU on the instance'
})
Page 58
Pack and Send Logs – Generated Ruby Script
timestamp = Time.now.utc.to_i
archive = "/tmp/#{timestamp}.tgz"
source = "/var/log"
s3bucket = "<%= node[:watcher][:s3][:bucket] %>"
object = "<%= node[:opsworks][:stack][:name] %>/<%= node[:opsworks][:instance][:hostname]
%>/#{timestamp}.tgz"
`cd #{File.dirname(source)} && tar czf #{archive} #{File.basename(source)}`
s3 = AWS::S3.new.buckets[s3bucket].objects[object].write(:file => archive)