Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co

© Zühlke 2012

Torben Knerr

Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co

20. Juli 2012 Slide 1

© Zühlke 2012

Topics

Motivation

DevOps & Infrastructure-As-Code

Introduction to Chef

Terms & Concepts (with Examples)

Chef vs. Puppet

Cloud Deployment

Different Approaches for Cloud Provisioning

Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 2

© Zühlke 2012


Torben Knerr

Motivation

DevOps & Infrastructure-As-Code


© Zühlke 2012

Motivation DevOps & Infrastructure-As-Code

Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr

DevOps

Drivers: Cloud & Virtualisation Mass Server Management Agile & Continous Delivery

Philosophy: Collaboration between Devs and Ops Devs understand the needs of Ops Ops learn from Devs (Infrastructure-As-Code) Ops are part of the (SCRUM) team

Disciplines: Operations, Monitoring Automation Programming (Ruby) Configuration Management

Goals: Traceability, Auditability Maintainability Reproducability Speed / Time-To-Market

Infrastructure-As-Code


© Zühlke 2012


Torben Knerr

Introduction to Chef

Knife, Cookbooks, Recipes, etc…


© Zühlke 2012

Introduction to Chef What‘s in a Chef Repository?


chef-repo


© Zühlke 2012



chef-repo

cookbooks

databags

environments

roles


© Zühlke 2012



recipes

attributes

files

templates chef-repo

cookbooks

databags

environments

roles


© Zühlke 2012

Introduction to Chef Chef Solo vs. Chef Server


node

couchdb cookbooks

roles databags

nodes … Chef Server

node node node node

chef-client

poll for node’s run_list and converge node

node

cookbooks roles

databags solo.rb

chef-solo

read run_list from solo.rb and converge node

Chef Solo Runs chef-solo locally No server required Easier to get started with Good for few nodes, onetime

bootstraps

chef-repo

upload to node


© Zühlke 2012

Introduction to Chef Chef Solo vs. Chef Server


node

couchdb cookbooks

roles databags

nodes … Chef Server

node node node node

chef-client

poll for node’s run_list and converge node

node

cookbooks roles

databags solo.rb

chef-solo

read run_list from solo.rb and converge node

Chef Server chef-client polls regularly More complex workflow Additional features:

Search, WebUI Good for ongoing

operations of many nodes chef-repo

upload to chef server


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes

Introduction to Chef Terms & Concepts


The Chef Repository is the place where cookbooks, roles, config files and other artifacts for managing systems with Chef will live, typically stored in a version control system.


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Cookbooks are the fundamental units of distribution in Chef. They encapsulate all the resources you need to automate your infrastructure and are easily sharable with other Chef users.


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Attributes are node data such as the IP address, hostname, etc... New attributes can be defined on cookbook, environment, role and node level.

cookbooks/apache2/attributes/default.rb


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Recipes encapsulate collections of resources which are executed in order to configure the system.

cookbooks/apache2/recipes/default.rb


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Recipes encapsulate collections of resources which are executed in order to configure the system.

Resources

cookbooks/apache2/recipes/default.rb


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Resources are the cross platform abstraction of the thing you're configuring on the node. Examples:


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes





© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes





© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes





© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes





© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Resources are the cross platform abstraction of the thing you're configuring on the node. Examples: Many, many more in the Opscode Wiki:

http://wiki.opscode.com/display/chef/Resources

You can also write your own: http://wiki.opscode.com/display/chef/Lightweight+Resources+and+Providers+%28LWRP%29


http://wiki.opscode.com/display/chef/Resources

http://wiki.opscode.com/display/chef/Lightweight+Resources+and+Providers+(LWRP)

http://wiki.opscode.com/display/chef/Lightweight+Resources+and+Providers+(LWRP)

© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Data bags provide an arbitrary stores of globally available JSON data…

chef-repo/data_bags/users/bofh.json


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



…which are then used in Recipes:

cookbooks/users/recipes/default.rb


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



A role provides a means of grouping similar features of similar nodes, providing a mechanism for easily composing sets of functionality.

chef-repo/roles/webserver.rb


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Environments in Chef provide a mechanism for managing different environments such as production, staging, development, testing, etc

chef-repo/environments/dev.rb


© Zühlke 2012

Chef Repo

Cookbooks

Attributes

Recipes

Resources

Databags

Roles

Environments

Nodes



Nodes are the thing that Recipes and Roles are applied to. The primary features of a node, from Chef's point of view, are its Attributes and its run list.

chef-repo/nodes/web.example.com.rb


© Zühlke 2012

How to get started?

• If you are working on Windows use Bill’s Kitchen https://ikm.zuehlke.com/topics/Pages/Bill's%20Kitchen.aspx

• Start with Vagrant and Chef-Solo (you will love Vagrant…) http://vagrantup.com/v1/docs/provisioners/chef_solo.html

• Get familiar with cookbook development and testing https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md

• Set up a Chef Server using knife-server http://fnichol.github.com/knife-server/

• Use Vagrant with Chef-Server Provisioning http://vagrantup.com/v1/docs/provisioners/chef_server.html

• Get familiar with knife and use advanced features like search, bootstrap, etc.. http://wiki.opscode.com/display/chef/Knife

Introduction to Chef Getting Started with Chef Development


https://ikm.zuehlke.com/topics/Pages/Bill's Kitchen.aspx

http://vagrantup.com/v1/docs/provisioners/chef_solo.html

https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md



http://fnichol.github.com/knife-server/



http://vagrantup.com/v1/docs/provisioners/chef_server.html

http://wiki.opscode.com/display/chef/Knife

© Zühlke 2012

Introduction to Chef VirtualBox Automation with Vagrant


Vagranfile


© Zühlke 2012

Introduction to Chef Cookbook Dependency Management with Librarian


Cheffile


© Zühlke 2012

Introduction to Chef Test-Driven Infrastrucure!


Lint Checking

• foodcritic

Unit-Level Testing

• chefspec

• fauxhai

Smoke Tests (from the inside)

• chef-minitest-handler

Acceptance Testing (from the outside)

• cucumber-nagios


© Zühlke 2012

It doesn’t really matter (for us)

Same concepts, slightly different terminology

External vs. internal DSL (like Maven vs. Gradle)

Both have solo / standalone mode

There are more Puppet books than Chef books

Both have a very active community (mailing list, IRC, etc..)

Do you have in-house knowledge of Chef/Puppet?

Introduction to Chef Chef vs. Puppet


© Zühlke 2012

It doesn’t really matter (for us)

Same concepts, slightly different terminology

External vs. internal DSL (like Maven vs. Gradle)

Both have solo / standalone mode

There are more Puppet books than Chef books

Both have a very active community (mailing list, IRC, etc..)

Do you have in-house knowledge of Chef/Puppet?

Introduction to Chef Chef vs. Puppet


Use the tool that “operations” understands!


© Zühlke 2012


Torben Knerr

Cloud Deployment

Different Approaches for Provisioning Cloud Instances


© Zühlke 2012

Cloud Deployment The Basic Mechanism: EC2 User Data

You can pass arbitrary user data when launching an instance…

…and access it on the instance via the metadata service:

Notes:

User data is limited to 16K in raw form (before base64 encoding)


© Zühlke 2012

Cloud Deployment Option 1: Provisioning with Shell Scripts

Convention: if the user data script starts with a `#!` it will be run at first boot

Notes:

Might not be supported on all AMIs (for the official Ubuntu AMIs it is)


path/to/bootstrap.sh


© Zühlke 2012

Cloud Deployment Option 1: Provisioning with Shell Scripts

Convention: if the user data script starts with a `#!` it will be run at first boot

Notes:

Might not be supported on all AMIs (for the official Ubuntu AMIs it is)


path/to/bootstrap.sh

easiest method good if bootstrapping is simple one-off bootstrap (first boot)


© Zühlke 2012

VM instances are bootstrapped with Chef/Puppet via Cloud-init

The rest is just AWS CloudFormation Templates

Supports Chef Server, Chef Solo and Puppet Client Server

Cloud Deployment Option 2: AWS CloudFormation Templates


© Zühlke 2012

VM instances are bootstrapped with Chef/Puppet via Cloud-init

The rest is just AWS CloudFormation Templates

Supports Chef Server, Chef Solo and Puppet Client Server

Cloud Deployment Option 2: AWS CloudFormation Templates


good for more complex scenarios

official, AWS supported documentation available

not meant for humans to read 20. Juli 2012 Slide 38

© Zühlke 2012

Vagrant-like approach for managing VMs in the cloud

Supports Chef, Puppet and Shell provisioning

Currently focused on AWS but other providers possible

Supports stacks via AWS CloudFormation Templates

Cloud Deployment Option 3: Mccloud


© Zühlke 2012

Vagrant-like approach for managing VMs in the cloud

Supports Chef, Puppet and Shell provisioning

Currently focused on AWS but other providers possible

Supports stacks via AWS CloudFormation Templates

Cloud Deployment Option 3: Mccloud


Yay! :-) 20. Juli 2012 Slide 40

© Zühlke 2012

knife-ec2

knife plugin to let you launch, bootstrap and provision EC2 VMs with Chef

Works only with Chef Server – no Chef Solo support

Puppet Cloud Provisioner

Puppet extension to launch, bootstrap and provision EC2 VMs with Puppet

Works only with Puppet Master – no Puppet Standalone mode (?)

Cloud Deployment Option 4: Native Chef / Puppet Tools


© Zühlke 2012

knife-ec2

knife plugin to let you launch, bootstrap and provision EC2 VMs with Chef

Works only with Chef Server – no Chef Solo support

Puppet Cloud Provisioner

Puppet extension to launch, bootstrap and provision EC2 VMs with Puppet

Works only with Puppet Master – no Puppet Standalone mode (?)

Cloud Deployment Option 4: Native Chef / Puppet Tools


You can’t have both: it’s Chef OR Puppet Fallback if the generic tools (Option 2 & 3)

don’t work out as expected No Solo / Standalone mode


© Zühlke 2012

Vagrant – create and configure lightweight, reproducible, and portable development environments


Vagranfile


© Zühlke 2012

EC2 User Data Scripts http://alestic.com/2009/06/ec2-user-data-scripts https://help.ubuntu.com/community/CloudInit http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html

AWS CloudFormation Templates with Chef/Puppet https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithOpscodeChef.pdf https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithPuppet.pdf

Mccloud https://github.com/jedi4ever/mccloud https://github.com/jedi4ever/mccloud-demo https://gist.github.com/3175267

knife-ec2 http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife

Puppet Cloud Provisioner http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html

Cloud Deployment Overview & Resources


http://alestic.com/2009/06/ec2-user-data-scripts







https://help.ubuntu.com/community/CloudInit

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html





https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithOpscodeChef.pdf



https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithPuppet.pdf



https://github.com/jedi4ever/mccloud

https://github.com/jedi4ever/mccloud-demo



https://gist.github.com/3175267

http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife

http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html

© Zühlke 2012


Torben Knerr

Discussion Time!

Puppet, Mccloud, Foo, Bar, Baz

DevOps, NoOps, Blablabla

Vagrant, Chef, Deployment Stuff

Cloudy Cloud!


Thank You! And let’s continue the discussion here:

http://github.com/tknerr

http://linkedin.com/in/tknerr

https://twitter.com/tknerr_de

mailto:[email protected]

Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co

Software