Masterless Puppet Using AWS S3 Buckets and IAM Roles
Post on 14-Jan-2017
650 Views
Preview:
Transcript
©opyright 2015 Cloudten Industries
Copyright statement:
This document contains a presentation given to the Puppet User Group by Cloudten Industries in January 2016. It has been made available freely for educational purposes. No part of this document may be reproduced or modified without the express written consent of the author.
Copyright 2015 Cloudten Industries
• Project Overview
• Technology Stack
• Puppet builds and deployments
• Security
• Issues
• Q&A
Copyright 2015 Cloudten Industries
Project Overview
• Client was a startup with VC backing.
• They wanted to build a mobile social media app:
• Dynamically scalable up to 100,000 hits per second
• Cross site active/active with real time DR
• Multiple test environments to be stood up/down at will
• End to end security with encryption at rest
• Engaged separate mobile and web app developers
Copyright 2015 Cloudten Industries
Multiple Development Teams
Company A: Mobile development team in Melbourne
Company B: Web App development team in
Perth/India
Copyright 2015 Cloudten Industries
Multiple Development Teams
• Using Xamarin Studio to create Apple and Android front ends
• Require access to code base to publish WSDLs under web app
• Require access to app logs
• Using a JDK to create a J2EE app running in Tomcat
• Require access to entire web app code base
• Require access to more logs
Copyright 2015 Cloudten Industries
What Did We Need to Solve ?
• Client didn’t have (or want) any full time IT staff.
• Client did want:
– Hands off builds and app deployments
– Self managed consistent server fleet
– No outage deployments
– No direct access to infrastructure from developers
Copyright 2015 Cloudten Industries
Hosting in AWS
• AWS provide on-demand scalable resources
• Facility to implement “Infrastructure as Code”
• Secure and durable object storage for code drops
• Fine grained security controls to create server roles and limit developer access.
• Additional services to co-ordinate deployments (Lambda, SNS, SQS)
Copyright 2015 Cloudten Industries
AWS Autoscaling
AWS can automatically add (and remove) servers to a load balancer pool based on a given metric ( eg. CPU or number of connections)
Scaling trigger hit Scale out to share the load
Copyright 2015 Cloudten Industries
How does it work ?
• Launches and builds identical virtual machines
• Destroys them with reckless abandon.
• Essential to get all logs into a central store
• Any generated content must be shared (i.e. can’t be stored on a local machine)
• The instance launch can trigger a build process (e.g. Puppet )
Copyright 2015 Cloudten Industries
Automated Server Builds
Start with base AmazonLinux Image
Invoke bootstrap script
Install Puppet RPM
S3 sync config andPuppet manifests
Puppet Apply
Copyright 2015 Cloudten Industries
Puppet Build Tasks
• Linux security patching and kernel hardening
• Define custom package repositories
• AWS tagging using facter
– Querying metadata to set instance specific tags
– Set tags for environment variables to be used later
• Package installation, config and version enforcement– NGINX, WAF, Tomcat, monit, sumo agents etc
• Configuration management
– functional users, public keys, cron jobs, log rotations, system health checks
Copyright 2015 Cloudten Industries
EC2 Tagging with Facter
# Sets hostnameclass common::hostname {
require aws
file { '/etc/hostname':ensure => 'present',owner => 'root',group => 'root',mode => '0644',content => template('common/hostname.erb'),notify => [ Exec['Set Hostname'],
Exec['Set EC2 Name-tag']],
}...exec { 'Set Hostname':command => "/bin/hostname -F /etc/hostname",unless => "/usr/bin/test `hostname` = `/bin/cat /etc/hostname`",refreshonly => 'true',
}
exec { 'Set EC2 Name-tag':command => '/usr/local/bin/setec2nametag',unless => '/usr/bin/test `/usr/local/bin/facter ec2_tag_Name` = `/bin/cat /etc/hostname`',
}}
Copyright 2015 Cloudten Industries
Kernel Tuning
# sysctl class
class common::sysctl {
file { '/etc/sysctl.conf':
source => 'puppet:///modules/common/sysctl.conf',
owner => 'root',
group => 'root',
mode => '0644',
notify => Exec['Refresh sysctl'],
}
exec { 'Refresh sysctl':
command => '/sbin/sysctl -q -p',
refreshonly => 'true',
}
}
Copyright 2015 Cloudten Industries
Event Based Triggers
cron
12
334 4
5 5
cron
control
scrip
t
6
7
Complete deployment and set a marker file as a trigger
Check for marker.
If present put message on SQS queue
CS checks for messageon queue
Get message from queue
S3 sync of changesto staging area
Puppet applies changes and restarts services
Copyright 2015 Cloudten Industries
How Does that Queue Work ?
• Lambda creates the queue once it detects trigger file (if it doesn’t exist already)
• Lambda queries the auto-scaling group and creates a message on the SQS queue for each member. It then deletes trigger file
• Each message has a 30 minute expiry ( deployments usually take <5 minutes)
• SQS queue has multiple consumers ( app servers)
• App server checks for its own message, retrieves then deletes it once deployment is successful.
• Lambda periodically checks for messages about to expire and the dead letter queue. Triggers an email alert if there is an issue.
Copyright 2015 Cloudten Industries
Puppet Deployment Tasks
Check for changes in staging content area
Sync content
Check for configuration file changes in staging areas
Apply changes
Restart/reload relevant services
Random sleep
Copyright 2015 Cloudten Industries
Puppet Deployment Tasks
# Conf dir file { '/etc/nginx/conf':ensure => ‘directory’,source => 'puppet:///modules/nginx/conf',recurse => truenotify => Service[’nginx'],require => Package[’nginx'],
}
# WAF rulesfile { '/etc/nginx/waf/modsec_waf.rules':
source => 'puppet:///modules/nginx/waf/modsec_waf.rules',... notify => Service[’waf'],require => Service[‘nginx’].require => Package[’nginx'],
}
Copyright 2015 Cloudten Industries
AWS IAM Users/Groups/Roles
• IAM (Identity & Access Management) allows fine grained user, group and role definitions
• S3 Bucket policies add a further level of security to restrict access to resources stored in S3
• Web developer bucket policy allows full access for web devgroup
• Mobile developer bucket policy allows full access for mobile dev group and read only access for web dev group
Copyright 2015 Cloudten Industries
IAM Users
{…
{"Sid": "AllowS3ListAccessToBucket","Effect": "Allow","Principal": {
"AWS": "arn:aws:iam::123456789101:group/webdevs"},"Action": "s3:ListBucket","Resource": "arn:aws:s3:::fdsr-‐webcontent-‐puppet-‐prod"
},{
"Sid": "AllowS3AccessToModules","Effect": "Allow","Principal": {
"AWS": ["arn:aws:iam::123456789101:group/webdevs","arn:aws:iam::123456789101:role/iam-‐ec2-‐webrole",
]},"Action": [
"s3:AbortMultipartUpload","s3:GetObjectAcl","s3:DeleteObject","s3:GetObject","s3:PutObjectAcl","s3:PutObject"
],"Resource": "arn:aws:s3:::fdsr-‐webcontent-‐puppet-‐prod/modules/webapp/f iles*"
}]
}
Copyright 2015 Cloudten Industries
IAM Users/Groups/Roles
IAM roles allow dynamically launched virtual servers to securely access credentials by querying locally accessible metadata at the special use IP address 169.254.169.254
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3read-only
{"Code" : "Success","LastUpdated" : "2015-04-26T16:39:16Z","Type" : "AWS-HMAC","AccessKeyId" : "AKIAIOSFODNN7EXAMPLE","SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY","Token" : "token","Expiration" : "2015-04-27T22:39:16Z"
}
Copyright 2015 Cloudten Industries
Why Masterless Puppet
• Less instances to manage ( no IT staff to manage it )
• No access for developers to internal infrastructure
• Not Enterprise Puppet
• No need to manage Puppet certificates ( AWS API calls are all encrypted and IAM enforces authentication)
Copyright 2015 Cloudten Industries
What Issues Did We Have ?
• Enforced versions being removed from repos
• Auto-scaling and healthcheck tuning
• S3 sync not handling zero byte files properly
• Event trigger mechanism needed tweeking
• Developers storing files on local instances
• S3FS. Don’t use it ! EVER !
Copyright 2015 Cloudten Industries
Who Are Cloudten ?
• Advanced AWS Consulting Partner
• Specialise in the design, delivery and support of cloud based infrastructure projects
• Focus on cloud security and hybrid integration
• We are a Puppet shop !
top related