5 things you don't know about Amazon Web Services

Simone BrunozziSr. Technology Evangelist, AWS

@simon

things you don’t know about Amazon Web Services5

1

01AWS CLI

Delete a bucket and all its content with just one command

2

3

aws s3 rb s3://bucket-name --force

4

5

6

02AWS CLI

Recursive copy of a directory and its subfolders to Amazon S3 (and vice versa)

7

8

aws s3 cp MyFolder s3://bucket-name --recursive [--region us-west-2]

9

03Autoscaling

Hmm...

10

Autoscaling

Amazon S3Amazon DynamoDBAmazon ELB (Elastic Load Balancing)Amazon GlacierAmazon EB (Elastic Beanstalk)Etc.

11

03EC2 Autoscaling

Automatically scale out EC2 within ELB

(old version)

12

Get the AS Command Line Tools:

aws.amazon.com/developertools/2535

1) Launch config: AMI to be used

2) Autoscaling group: where/how to launch

3) Autoscaling policy: what should AS do?

4) Autoscaling trigger: what will activate AS

13

AutoScaling (AS):Four main components

13

14 AutoScaling tools

14

15 CloudWatch API Tools

15

FILE: ec2.bat

@echo off

set EC2_HOME=c:\ec2

set PATH=%PATH%;%EC2_HOME%\bin

set JAVA_HOME=C:\Program Files\Java\jre6

"%JAVA_HOME%\bin\java" -version

set JAVA_HOME=C:\Program Files (x86)\Java\jre6 (Windows 7)

set EC2_PRIVATE_KEY=c:\ec2\PrivateKey.pem

set EC2_CERT=c:\ec2\509Certificate.pem

set AWS_AUTO_SCALING_HOME=c:\ec2\AutoScaling

set AWS_CLOUDWATCH_HOME=c:\ec2\CloudWatch

set PATH=%PATH%;%AWS_AUTO_SCALING_HOME%\bin;%AWS_CLOUDWATCH_HOME%\bin

16

Windows setup

16

FILE: .bash_profile

export EC2_HOME=~/ec2

export PATH=$PATH:$EC2_HOME/bin

export EC2_PRIVATE_KEY=pk-[removed].pem

export EC2_CERT=cert-[removed].pem

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/

export AWS_AUTO_SCALING_HOME=~/ec2/AutoScaling

export AWS_CLOUDWATCH_HOME=~/ec2/CloudWatch

export PATH=$PATH:$AWS_AUTO_SCALING_HOME/bin:$AWS_CLOUDWATCH_HOME/bin

17

Linux setup

17

% as-version

Amazon AutoScaling CLI version 1.0.61.3 (API 2011-01-01)

18

Test if it works

18

STEP 1: Create Launch Configas-create-launch-config LC1 --image-id ami-31814f58 --instance-type m1.small

[--region us-east-1] --group SG

STEP 2: Autoscaling with ELBas-create-auto-scaling-group ASG1 --launch-configuration LC1 --availability-

zones us-east-1b us-east-1c --min-size 2 --max-size 20 --load-balancers aws201

Check if group is created, any running EC2as-describe-auto-scaling-groups --headers

as-describe-auto-scaling-instances --headers

19

AutoScaling + ELB(1, 2)

19

Scale in EC2 instances to ZERO, if neededas-update-auto-scaling-group ASG1 --min-size 0 --max-size 0

20

AutoScaling + ELB“Emergency break”

20

STEP 3: Create Scaling Out Policyas-put-scaling-policy SOut1 --auto-scaling-group ASG1 --adjustment=1 --type

ChangeInCapacity --cooldown 120

arn:aws:autoscaling:us-

east-1:132891460296:scalingPolicy:a4a80ac4-3e3e-46a7-9b4a-72d893803618:autoSca

lingGroupName/ASG1:policyName/SP1

21

AutoScaling + ELB(3)

21

STEP 4: Create Metric Out Alarmmon-put-metric-alarm MOutA1 --comparison-operator GreaterThanThreshold --

evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --

period 60 --statistic Average --threshold 70 --alarm-actions

arn:aws:autoscaling:us-

east-1:132891460296:scalingPolicy:a4a80ac4-3e3e-46a7-9b4a-72d893803618:autoSca

lingGroupName/ASG1:policyName/SOut1 --dimensions "AutoScalingGroupName=ASG1"

22

AutoScaling + ELB(4)

22

STEP 5: Create Scaling In Policyas-put-scaling-policy SInP1 --auto-scaling-group ASG1 --adjustment=-1 --type

ChangeInCapacity --cooldown 120

arn:aws:autoscaling:us-east-1:132891460296:scalingPolicy:5ddb0046-a571-417f-

b025-4268baf9d860:autoScalingGroupName/ASG1:policyName/SInP1

If you are using Windows, wrap the --adjustment parameter in quotation marks:

"--adjustment=-1".

23

AutoScaling + ELB(5)

23

STEP 6: Create Metric In Alarmmon-put-metric-alarm MInA1 --comparison-operator LessThanThreshold --

evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --

period 60 --statistic Average --threshold 30 --alarm-actions

arn:aws:autoscaling:us-east-1:132891460296:scalingPolicy:5ddb0046-a571-417f-

b025-4268baf9d860:autoScalingGroupName/ASG1:policyName/SInP1 --dimensions

"AutoScalingGroupName=ASG1"

24

AutoScaling + ELB(6)

24

#!/bin/bash

aws_instance=$(wget -q -O- http://169.254.169.254/latest/meta-data/instance-

id)

aws_region=$(wget -q -O- http://169.254.169.254/latest/meta-data/hostname)

aws_region=${aws_region#*.}

aws_region=${aws_region%%.*}

aws_zone=`ec2-describe-instances $aws_instance --region $aws_region`

aws_zone=`expr match "$aws_zone" ".*\($aws_region[a-z]\)"`

print "Instance: $aws_instance<br>Region: $aws_region<br>Availability Zone:

$aws_zone<br><br>END" > index.html

25

Script you can usewithin each EC2 instance, at boot

25

03EC2 Autoscaling

Automatically scale out EC2 within ELB

(NEW version)

26

27

04EC2 metadata

How to get details related to the EC2 machine that I’m running (PHP)

28

29

30

05VPC + Cloudformation

Using CF to create a 3-tier architecture within VPC.(first steps)

31

{ "AWSTemplateFormatVersion" : "2010-09-09",! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",! "Parameters" : {! ! "VpcCidr" : {! ! ! "Description" : "CIDR for the VPC",! ! ! "Type" : "String",! ! ! "MinLength": "9",! ! ! "MaxLength": "18",! ! ! "Default" : "10.0.0.0/16",! ! ! "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/x."! ! } },! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC", "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}, "Tags" : [ {"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} } ] } }! }}

32

{ "AWSTemplateFormatVersion" : "2010-09-09",! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",! "Parameters" : {! ! "VpcCidr" : {! ! ! "Description" : "CIDR for the VPC",! ! ! "Type" : "String",! ! ! "MinLength": "9",! ! ! "MaxLength": "18",! ! ! "Default" : "10.0.0.0/16",! ! ! "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/x."! ! } },! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC", "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}, "Tags" : [ {"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} } ] } }! }}

Do you hate me?Too small?

33

{ "AWSTemplateFormatVersion" : "2010-09-09",! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",! "Parameters" : {! ! "VpcCidr" : {! ! ! "Description" : "CIDR for the VPC",! ! ! "Type" : "String",! ! ! "MinLength": "9",! ! ! "MaxLength": "18",! ! ! "Default" : "10.0.0.0/16",! ! ! "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/x."! ! } },! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC", "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}, "Tags" : [ {"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} } ] } }! }}

34

{ "AWSTemplateFormatVersion" : "2010-09-09",! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",! "Parameters" : {! ! "VpcCidr" : {! ! ! "Description" : "CIDR for the VPC",! ! ! "Type" : "String",! ! ! "MinLength": "9",! ! ! "MaxLength": "18",! ! ! "Default" : "10.0.0.0/16",! ! ! "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/x."! ! } },! "Resources" : { "VPC" : { "Type" : "AWS::EC2::VPC", "Properties" : { "CidrBlock" : {"Ref" : "VpcCidr"}, "Tags" : [ {"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} } ] } }! }}

35

cfn-create-stack CustomerLittleNetwork --tag "Key=env; Value=temp" --template-file stack_cfn_learn_001_vpc_step1.template

cfn-describe-stacks

cfn-describe-stacks CustomerLittleNetwork

cfn-list-stack-resources CustomerLittleNetwork

36

37

38

cfn-update-stack CustomerLittleNetwork --template-file stack_cfn_learn_001_vpc_step2.template

cfn-describe-stacks CustomerLittleNetwork

cfn-list-stack-resources CustomerLittleNetwork

39

{ "AWSTemplateFormatVersion" : "2010-09-09",! "Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",

! "Parameters" : {! ! "VpcCidr" : {

40

"Parameters" : {! ! "VpcCidr" : {! ! ! "Description" : "CIDR for the VPC",! ! ! "Type" : "String",! ! ! "MinLength": "9",! ! ! "MaxLength": "18",! ! ! "Default" : "10.0.0.0/16",! ! ! "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/x."! ! },

41

! ! "PublicSubnetCidrA" : {! ! ! "Description" : "Subnet for public resources in the first AZ",! ! ! "Type" : "String",! ! ! "MinLength": "9",! ! ! "MaxLength": "18",! ! ! "Default" : "10.0.1.0/24",! ! ! "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",! ! ! "ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/x."! ! },

42

43

06CloudFormer

CloudFormation templates from an existing deployment

44

45

07Custom errors

Enabling custom error pages on CloudFront

46

47

48

49

08ZFS on Amazon EC2

How to install and run ZFS on Amazon EC2(yes, you can!)

Thanks to:Constantin Gonzalez Dougal Ballantyne

(demo coming soon)

50

What is ZFS?

ZFS is a file system + logical volume manager. Features:- protection against data corruption- support for high storage capacities- integration of filesystem and volume management- snapshots and copy-on-write clones- continuous integrity checking and automatic repair- RAID-Z- native NFSv4 ACLs.

OpenZFS is an Open Source implementation of it.

51

How can I use it?

Launch an EC2 instance with an AMI which supports ZFS.

1) Linux

2) OmniOS (native ZFS support based on a fork of the original OpenSolaris project that open-sourced ZFS)

3) FreeBSD (native ZFS support based on a port of the OpenSolaris ZFS code)

52

1) Linux

Linux AMI:https://aws.amazon.com/amazon-linux-ami/http://zfsonlinux.org/

53

2) OmniOS

54

3) FreeBSD

55

And now what?

It works the same as with any other hardware platform: 1) Attach a few EBS volumes to your EC2 instance;2) use them as you would use regular disks with ZFS

An example follows.

56

$ sudo modprobe zfs [attach 3 EBS volumes to the instance] $ sudo zpool create test raidz sdf sdg sdh $ sudo zpool status pool: test state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 errors: No known data errors

57

$ mount /dev/xvda1 on / type ext4 (rw,noatime) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) test on /test type zfs (rw,xattr) $ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.9G 1.6G 6.3G 20% / tmpfs 15G 0 15G 0% /dev/shm test 196G 0 196G 0% /test

58

What can I do with ZFS?

Many things! Here’s one example:Use an EBS Provisioned IOPS volumes as ZFS cache and log (ZIL) devices to accelerate your zpools.

This works much like you would use SSDs in the real world to accelerate ZFS reads and writes.

For instance, you could provision an EBS volume with 1000 IOPS and use it as a ZIL device for your zpool in order to accelerate database writes.

59

Learn more

http://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/

60

09Extra stuff

Friends, nice people, cool things, etc.

61

62

63

64

65

66

Thank you!

67

Simone BrunozziSr. Technology Evangelist, AWS

@simon

things you don’t know about Amazon Web Services5

68