Masterless Puppet Using AWS S3 Buckets and IAM Roles

©opyright 2015 Cloudten Industries


Copyright statement:

This document contains a presentation given to the Puppet User Group by Cloudten Industries in January 2016. It has been made available freely for educational purposes. No part of this document may be reproduced or modified without the express written consent of the author.

Copyright 2015 Cloudten Industries

• Project Overview

• Technology Stack

• Puppet builds and deployments

• Security

• Issues

• Q&A


Project Overview

• Client was a startup with VC backing.

• They wanted to build a mobile social media app:

• Dynamically scalable up to 100,000 hits per second

• Cross site active/active with real time DR

• Multiple test environments to be stood up/down at will

• End to end security with encryption at rest

• Engaged separate mobile and web app developers


Multiple Development Teams

Company A: Mobile development team in Melbourne

Company B: Web App development team in

Perth/India


Multiple Development Teams

• Using Xamarin Studio to create Apple and Android front ends

• Require access to code base to publish WSDLs under web app

• Require access to app logs

• Using a JDK to create a J2EE app running in Tomcat

• Require access to entire web app code base

• Require access to more logs


What Did We Need to Solve ?

• Client didn’t have (or want) any full time IT staff.

• Client did want:

– Hands off builds and app deployments

– Self managed consistent server fleet

– No outage deployments

– No direct access to infrastructure from developers


Hosting in AWS

• AWS provide on-demand scalable resources

• Facility to implement “Infrastructure as Code”

• Secure and durable object storage for code drops

• Fine grained security controls to create server roles and limit developer access.

• Additional services to co-ordinate deployments (Lambda, SNS, SQS)


AWS Autoscaling

AWS can automatically add (and remove) servers to a load balancer pool based on a given metric ( eg. CPU or number of connections)

Scaling trigger hit Scale out to share the load


How does it work ?

• Launches and builds identical virtual machines

• Destroys them with reckless abandon.

• Essential to get all logs into a central store

• Any generated content must be shared (i.e. can’t be stored on a local machine)

• The instance launch can trigger a build process (e.g. Puppet )


Automated Server Builds

Start with base AmazonLinux Image

Invoke bootstrap script

Install Puppet RPM

S3 sync config andPuppet manifests

Puppet Apply


Puppet Build Tasks

• Linux security patching and kernel hardening

• Define custom package repositories

• AWS tagging using facter

– Querying metadata to set instance specific tags

– Set tags for environment variables to be used later

• Package installation, config and version enforcement– NGINX, WAF, Tomcat, monit, sumo agents etc

• Configuration management

– functional users, public keys, cron jobs, log rotations, system health checks


EC2 Tagging with Facter

# Sets hostnameclass common::hostname {

require aws

file { '/etc/hostname':ensure => 'present',owner => 'root',group => 'root',mode => '0644',content => template('common/hostname.erb'),notify => [ Exec['Set Hostname'],

Exec['Set EC2 Name-tag']],

}...exec { 'Set Hostname':command => "/bin/hostname -F /etc/hostname",unless => "/usr/bin/test `hostname` = `/bin/cat /etc/hostname`",refreshonly => 'true',

}

exec { 'Set EC2 Name-tag':command => '/usr/local/bin/setec2nametag',unless => '/usr/bin/test `/usr/local/bin/facter ec2_tag_Name` = `/bin/cat /etc/hostname`',

}}


Kernel Tuning

# sysctl class

class common::sysctl {

file { '/etc/sysctl.conf':

source => 'puppet:///modules/common/sysctl.conf',

owner => 'root',

group => 'root',

mode => '0644',

notify => Exec['Refresh sysctl'],

}

exec { 'Refresh sysctl':

command => '/sbin/sysctl -q -p',

refreshonly => 'true',

}

}


Developer Code Drops

MySQL

App


Event Based Triggers

cron

12

334 4

5 5

cron

control

scrip

t

6

7

Complete deployment and set a marker file as a trigger

Check for marker.

If present put message on SQS queue

CS checks for messageon queue

Get message from queue

S3 sync of changesto staging area

Puppet applies changes and restarts services


How Does that Queue Work ?

• Lambda creates the queue once it detects trigger file (if it doesn’t exist already)

• Lambda queries the auto-scaling group and creates a message on the SQS queue for each member. It then deletes trigger file

• Each message has a 30 minute expiry ( deployments usually take <5 minutes)

• SQS queue has multiple consumers ( app servers)

• App server checks for its own message, retrieves then deletes it once deployment is successful.

• Lambda periodically checks for messages about to expire and the dead letter queue. Triggers an email alert if there is an issue.


Puppet Deployment Tasks

Check for changes in staging content area

Sync content

Check for configuration file changes in staging areas

Apply changes

Restart/reload relevant services

Random sleep


Puppet Deployment Tasks

# Conf dir file { '/etc/nginx/conf':ensure => ‘directory’,source => 'puppet:///modules/nginx/conf',recurse => truenotify => Service[’nginx'],require => Package[’nginx'],

}

# WAF rulesfile { '/etc/nginx/waf/modsec_waf.rules':

source => 'puppet:///modules/nginx/waf/modsec_waf.rules',... notify => Service[’waf'],require => Service[‘nginx’].require => Package[’nginx'],

}


AWS IAM Users/Groups/Roles

• IAM (Identity & Access Management) allows fine grained user, group and role definitions

• S3 Bucket policies add a further level of security to restrict access to resources stored in S3

• Web developer bucket policy allows full access for web devgroup

• Mobile developer bucket policy allows full access for mobile dev group and read only access for web dev group


IAM Users


IAM Users

{…

{"Sid": "AllowS3ListAccessToBucket","Effect": "Allow","Principal": {

"AWS": "arn:aws:iam::123456789101:group/webdevs"},"Action": "s3:ListBucket","Resource": "arn:aws:s3:::fdsr-‐webcontent-‐puppet-‐prod"

},{

"Sid": "AllowS3AccessToModules","Effect": "Allow","Principal": {

"AWS": ["arn:aws:iam::123456789101:group/webdevs","arn:aws:iam::123456789101:role/iam-‐ec2-‐webrole",

]},"Action": [

"s3:AbortMultipartUpload","s3:GetObjectAcl","s3:DeleteObject","s3:GetObject","s3:PutObjectAcl","s3:PutObject"

],"Resource": "arn:aws:s3:::fdsr-‐webcontent-‐puppet-‐prod/modules/webapp/f iles*"

}]

}


IAM Users/Groups/Roles

AppServer


IAM Users/Groups/Roles

IAM roles allow dynamically launched virtual servers to securely access credentials by querying locally accessible metadata at the special use IP address 169.254.169.254

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3read-only

{"Code" : "Success","LastUpdated" : "2015-04-26T16:39:16Z","Type" : "AWS-HMAC","AccessKeyId" : "AKIAIOSFODNN7EXAMPLE","SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY","Token" : "token","Expiration" : "2015-04-27T22:39:16Z"

}


Why Masterless Puppet

• Less instances to manage ( no IT staff to manage it )

• No access for developers to internal infrastructure

• Not Enterprise Puppet

• No need to manage Puppet certificates ( AWS API calls are all encrypted and IAM enforces authentication)


What Issues Did We Have ?

• Enforced versions being removed from repos

• Auto-scaling and healthcheck tuning

• S3 sync not handling zero byte files properly

• Event trigger mechanism needed tweeking

• Developers storing files on local instances

• S3FS. Don’t use it ! EVER !


Who Are Cloudten ?

• Advanced AWS Consulting Partner

• Specialise in the design, delivery and support of cloud based infrastructure projects

• Focus on cloud security and hybrid integration

• We are a Puppet shop !


Masterless Puppet Using AWS S3 Buckets and IAM Roles

Data & Analytics