AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS Infrastructure (SVR401)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Randall Hunt, Developer Evangelist, AWS

Using AWS Lambda to Build Automated

Control Systems for Your AWS

Infrastructure

December 1, 2016

SVR401

Welcome to re:Invent!

Randall Hunt - @jrhunt

Developer Evangelist at AWS

[email protected]

Formerly of SpaceX, NASA

Agenda

• Brief overview of AWS Lambda

• Why automate?

• Why Lambda for automation and control systems?

• Event-driven policy enforcement

• Lambda as an infrastructure control plane

• Best practices

Operations and management Scaling

Provisioning and utilization Responsible for availability

and fault tolerance

Owning servers means dealing with ...

Serverless compute: AWS Lambda

COMPUTE SERVICE

EVENT-DRIVEN

Run code

without

managing

servers

Code runs only

when it needs to

run

Code is all you need(Native: Java/Python/NodeJS)

Event-driven scaling

Never pay for idle servers Availability and fault tolerance built in

Going serverless with AWS Lambda

Things to remember: Lambda function

Memory = “Power level”

• Higher levels offer more memory and more CPU power

Functions don’t have a notion of state

• Use Amazon DynamoDB, Amazon S3, or Amazon ElastiCache

• Wrap your config in a function and call it from your published code

Use the right access control for downstream services

• IAM roles and permissions for AWS services

• VPC for private endpoints

Why automate?

Benefits of automation done correctly

Increased Agility Improved Quality

Improved Innovation

Increased Consistency

Reduced Expenses and

Labor Costs

Reduced Human Errors

Dangers of incorrect automation

Code Maintenance Issues Bad Permissions Control Auditing Nightmares

Everyone Blames You!Accountability

Why is automation key?

Amazon EC2 Amazon ECS Amazon CloudWatch AWS CloudTrail AWS IAM

What sorts of things can we automate?

Just about anything!

Amazon

API GatewayAWS

IoT

Amazon

Alexa

Automatic Infrastructure Cleanup Allow other endpoints and AWS services to

trigger infrastructure change

Create copies/snapshots of data in response

to events

Integrate / think outside the box

Amazon EC2 Amazon ECS Amazon ECR

Amazon RDS Amazon

DynamoDB

Instant Core Infrastructure Changes

Amazon

VPC

Amazon

Route 53AWS

CloudFormation

Why Lambda for automation?

Lambda as a building block for automaton

Because Lambda is event-driven, it offers a very

powerful framework for automated infrastructure

control planes and policy engines.

AWS Lambda

Near real-time reactionEvent

Benefits for automation with AWS Lambda

Single knob configuration Easy to integrate Logging and auditability

Rapid delivery and versioning Get started fast

Event-driven automation as a building block

Being proactive rather than reactive to changes in your

infrastructure is key. However, event-driven automation can

be part of the solution. We can remove human errors when

we need to be reactive through use of automation.

Amazon

S3

Amazon

DynamoDB

Amazon

Kinesis

AWS

CloudFormation

AWS

CloudTrail

Amazon

CloudWatch

Logs

Amazon

SNSAmazon

SES

Amazon

API Gateway

Amazon

Cognito

AWS

IoT

Cron events

Trigger an event based on Data Store changes Interact with other endpoints and AWS services

to trigger infrastructure change

Respond to custom events as they happen

near real-time

Respond to messaging or scheduled

events in your infrastructure

Event sources

Event-driven automation

EC2 Instance

“Event”

Manual Event

Data Streams / Logs

Lambda.Invoke

Event-driven policy enforcement

Policy enforcement

What do you mean by policy enforcement?

You have the freedom to define what

policy enforcement means to you!

Policy enforcement

AWS Config Rules Amazon

CloudWatch

AWS

CloudTrail

Lambda Policy Engine

AWS Config

Resource Inventory

AWS Config

Configuration History

AWS Config

Config Rules

VPC policy enforcement


• Allocate Address

• Associate Address

instance instance instance



Validate Security Group


Identity and access management enforcement


Validate Role Path

AWS IAM

CreateRole

Verify No Role Elevation

Verify No Resource Level Elevation

Allow Role

Delete Roles

Tagging enforcement




RunInstancesValidate Team Tag

Validate Billing Group Tag

Validate Environment Stage Taginstance instance instance

StopInstances

SNS Notification


Demo!AWS Config compliance checks

Infrastructure control plane

Infrastructure control plane

Lambda gives you the freedom to

design the automation you need for

your infrastructure.

Infrastructure automation

Amazon CloudWatch Events

Auto Scaling Event AWS API Call EC2 Instance State Change

Automatic security group placement




RunInstancesCheck VPC

Validate Security Group

Remove SG

Add Correct SG


Do Nothing

Create/Update Amazon Route 53 records from tag




RunInstances

Do Nothing


Check VPC

Update DNS instance

Get CNAME Tag

Add DNS Record

instance instance

Demo!Automated Route 53 DNS updates on instance creation

CloudFormation Bootstrapping

Bootstrapping With Lambda and Await/Notify

• Need to load a schema into RDS? Lambda function.

• Need to load data into RDS? Lambda function.

• Need to load data into DynamoDB? Lambda function.

• Need to perform a complex bootstrapping action?

Lambda function.

• Need to depend on an external action? Lambda function

+ API Gateway.

• For everything else there’s EC2 dependson/await/notify.

Best practices

Best practices

Log Everything

Bite-Sized Code

Report Failures

Version Lambda Functions

Use Source Control

Iterate

Recap

• Event-Driven Response to Policy Management and Infrastructure Events.

• AWS Config Rules backed by Lambda to visualize compliant versus non-compliant

infrastructure.

• Think outside the box. Number of available CloudWatch Events API Call triggers is

large.

• Don’t forget the ability to schedule AWS Config rule validation, as well as Scheduled

CloudWatch Event Triggers to Lambda.

• CloudWatch Events can point to multiple Targets (Not just Lambda functions).

Thanks for your attention!

Follow @jrhunt

Remember to complete

your evaluations!

AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS Infrastructure (SVR401)

Technology