Achieving Infrastructure Portability with Chef

Achieving Infrastructure Portability with Chef

CloudOpen 2012

Matt [email protected]

IRC/Twitter/GitHub: mattraywww.opscode.com

Data Centers...

To the Cloud!

• Instant infrastructure

• Unlimited capacity

• Autoscaling

• No commitment

• Immediate replacement

Why the Cloud?

• Each cloud defines themselves against Amazon

• Entry into the market is easier

• Not a lot of price competition

• Feature parity is growing

Cloud Differentiation

The Dark Side of the Cloud

• Reliability

• Performance

• Security

• Price

Why not the Cloud?

Data Gravity

Know our escape plan for every infrastructure

provider

Why Chef?

See Node

Application Server

See Nodes

Application Server

Application Database

See Nodes Grow

Application Server

Application Databases

Application Servers

Application Databases

See Nodes Grow

Application Servers

Application Databases

Load Balancer

See Nodes Grow

See Nodes Grow

Application Servers

Application Databases

Load Balancers

See Nodes Grow

Application Servers

Application Database Cache

Load Balancers

Application Databases

Tied together with Config

Application Servers

Application Database Cache

Load Balancers

Application Databases

Infrastructure is a Snowflake

Application Servers

Application Database Cache

Load Balancers

Floating IP?

Application Databases

Evolving Complexity

Load Balancers

Application Servers

NoSQL

Database Slaves

ApplicationCache

Database Cache

Database

Complexity Grows Quickly

DC1

DC3

DC2

http://www.flickr.com/photos/16339684@N00/2681435235/

And it Continues to Evolve

That's great and all, but tell me about

Chef!

Chef is Infrastructure as Code

http://www.flickr.com/photos/louisb/4555295187/

• Programmatically provision and configure

• Treat like any other code base

• Reconstruct business from code repository, data backup, and bare metal resources.

• Chef-Client generates configurations directly on nodes from their run list

• Reduce management complexity through abstraction

• Store the configuration of your programs in version control

http://www.flickr.com/photos/ssoosay/5126146763/

Nodes

Collections of Resources

• Networking

• Files

• Directories

• Symlinks

• Mounts

• Routes

• Users

• Groups

• Tasks

• Packages

• Software

• Services

• Configurations

• Other Stuffhttp://www.flickr.com/photos/stevekeys/3123167585/

Declarative Interface to Resources

• Define policy

• Say what, not how

• Pull not Push

http://www.flickr.com/photos/bixentro/2591838509/

Ruby!

extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend

Recipes and Cookbooks

• Recipes are collections of Resources

• Cookbooks contain recipes, templates, files, custom resources, etc

• Code re-use and modularity

• Hundreds already on Community.opscode.com

http://www.flickr.com/photos/shutterhacks/4474421855/

http://www.flickr.com/photos/kathycsus/2686772625

• IP addresses

• Hostnames

• FQDNs

• Search for nodes with Roles

• Find configuration data

Search

pool_members = search("node","role:webserver”)

template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end

Pass Results to Templates

# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>

Pass Results to Templates

Jboss App

Memcache

Postgres Slaves

Postgres Master

So when this

NagiosGraphite

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Becomes this

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Updates can be automatic

NagiosGraphite

Count the resources

Jboss App

Memcache

Postgres Slaves

• Load balancer config

• Nagios host ping

• Nagios host ssh

• Nagios host HTTP

• Nagios host app health

• Graphite CPU

• Graphite Memory

• Graphite Disk

• Graphite SNMP

• Memcache firewall

• Postgres firewall

• Postgres authZ config

• 12+ resource changes for 1 node addition

Build anything

• Simple internal applications

• Complex external applications

• Workstations

• Hadoop clusters

• IaaS infrastructure

• PaaS infrastructure

• SaaS applications

• Storage systems

• You name it

http://www.flickr.com/photos/hyku/245010680/

And manage it simply

http://www.flickr.com/photos/helico/404640681/

• Automatically reconfigure everything

• Linux, Windows, Unixes, BSDs

• Load balancers

• Metrics collection systems

• Monitoring systems

• Cloud migrations become trivial

knife

knife with the Chef Server

• knife node

• create/delete/edit

• list

• knife cookbook ...

• knife role ...

• knife environment ...

knife bootstrap

knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/id_rsa

• SSH to the machine given existing credentials

• Install the Chef Client

• Register with the Chef Server

• Run the initial Run List

• Now managed with Chef!

knife ec2

$ knife ec2Available ec2 subcommands: (for details, knife SUB-COMMAND --help)

** EC2 COMMANDS **knife ec2 flavor list (options)knife ec2 instance data (options)knife ec2 server create (options)knife ec2 server delete SERVER [SERVER] (options)knife ec2 server list (options)

$ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'

knife openstack

$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)

** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)

$ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'

Chef for Infrastructure Portability

• knife ec2

• knife rackspace

• knife hp

• knife google

• knife azure

• knife cloudstack

• knife openstack

• knife vsphere

• ... and many others

The Chef Community

• Apache License, Version 2.0

• 850+ Individual contributors

• 150+ Corporate contributors

• HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more

• 550+ cookbooks

• http://community.opscode.com

• Vagrant

• VMware

• CloudStack

• Eucalyptus

• OpenStack

• bare metal

• AWS

• Rackspace

• HP

• Google

• Azure

• many others

Desktop, Virtualization, Private & Public Clouds

• Vagrant

• VMware

• CloudStack

• Eucalyptus

• OpenStack

• bare metal

Desktop, Virtualization, Private & Public Clouds

• AWS

• Rackspace

• HP

• Google

• Azure

• many others

Tale of the Tape

• Artur Bergman, CEO at Fastly

• It's All About Speed

• http://youtu.be/qRnTejOMbZU

• Jason Stowe, CEO at Cycle Computing

• CycleCloud + Chef = 50,000-core Utility Supercomputer for Science

• http://youtu.be/cEaQB6e7G0Q

Chef and Abstractions

• Resources and Providers

• Cookbooks (may) normalize deployment

• Knife treats APIs the same

• Chef strives to not be opinionated

Environments

• Lock down versions of cookbooks

• stable releases vs. development

• Enforce attributes for deployment

• Ports, addresses, etc.

• Different run lists based on environment

• Debugging enabled?

Environments

• Use the same infrastructure code for wherever you deploy

• Development, QA, Pre-Production, Prod

• Role-based Access Controls to restrict the promotion of deployment code

TL;DL

• Every infrastructure is a unique snowflake

• Understand the costs associated with the features of your platform(s) of choice.

• Chef enables Infrastructure Portability

• "Data Gravity" is the primary concern

Thanks!

Matt [email protected]

IRC/Twitter/GitHub: mattraywww.opscode.com