Chef Cookbook Workflow

Chef Cookbook WorkflowTesting your infrastructure code

Alex Manly - @apmanly

Instructor

• Alex Manly• Solutions Architect, Chef• Developer (Java, oh and some Ruby)• @apmanly• [email protected]

What is Chef?

Short Answer: An Automation Framework for automating infrastructure and applications

Traditional Systems Management

Config 1

Config 2

Config 3

Traditional Systems Management

Config 1

Config 2

Config 3

Autom

ation

Idea of Services

Service

Config 1

Config 2

Config 3

Autom

ation

Abstraction of Services

Service

Config 1

Config 2

Config 3

Autom

ation

Chef is Infrastructure as Code

•Programmatically provision and configure components

http://www.flickr.com/photos/louisb/4555295187/



•Treat like any other code base




•Reconstruct business from code repository, data backup, and compute resources



Chef is Infrastructure as Code•Programmatically provision and configure components

•Treat like any other code base

•Reconstruct business from code repository, data backup, and compute resources



Building Blocks: What is a Resource?• A Resource is a system state you define

Example: Package installed, state of a service, configuration file existing• You declare what state you want the resource in.

Chef automatically determines HOW that state is achieved

On Linux based OSes: On Windows based OSes:

Resource Example#windowsdsc_resource 'webserver' do

resource :windowsfeature property :name, 'Web-Server' property :ensure, 'Present’

end

Resource Example#linuxpackage "httpd" do

action :installend

Building Blocks: What is a Recipe?• An abstraction of a Service that consists of a set of Resources to deliver that Service

• Resources are executed in the order they are listed.

Recipe Example#linuxpackage "httpd" do

action :installend

include_recipe "apache::fwrules”

service "httpd" doaction [ :enable, :start ]

end

Recipe Example#windowsinclude_recipe "fourthcoffee::dsc”include_recipe "iis::remove_default_site”remote_directory node['fourthcoffee']['install_path'] do

source 'fourthcoffee’ action :create

endiis_pool 'FourthCoffee' do

runtime_version "4.0" action :add

endiis_site 'FourthCoffee' do

protocol :http port 80 path node['fourthcoffee']['install_path'] application_pool 'FourthCoffee' action [:add,:start]

end

Cookbooks

•A Higher Level Abstraction of a Service

•A set of Recipes and Data Attributes required to deliver one or multiple Services

Define cookbook attribute#attributes.rbdefault['fourthcoffee']['port'] = 80

Consume cookbook attribute

iis_site 'FourthCoffee' do protocol :http port node['fourthcoffee']['port'] path node['fourthcoffee']

['install_path'] application_pool 'FourthCoffee' action [:add,:start]

end

Yes!

That’s cool but…• Things break• Chef is a language (based on Ruby)• How can you rapidly develop recipes and cookbooks?

Let’s step back…

Testing is Hard - FACT

Automated GUI Tests

Integration TestsUnit

Tests

Manual TestsAutomate

dGUI Test

sAutomated API

TestsAutomated Integration Tests

Automated Component Test

Automated Unit Tests

Manual Session Based Testing

Also known as…

You’ll never get to Continuous Deployment

clicking a GUI

Theory of testing…

Testing builds safety

Feedback loops…• Tell us we’re doing the right thing• At the right time• With the right results

Feedback loops…

Measurements we take to ensure the “experiment” is behaving as expected

Tests are essentially feedback loops

Remember…

Chef is “Infrastructure as Code”

Remember…

“Infrastructure as Code” should betreated like ANY other codebase.

Treated Any Other Codebase…• Stored in SCM• Testing Coverage• Part of your CI pipelines

Testing in Chef• Chef recipes need tested

Linting Static Analysis Unit Testing Functional Testing

DevOps is a Two-Way Street

• It’s great when developers care about

• Uptime!• Scaling!• Deployment!• Put them on call! etc.

etc. etc.

DevOps is a Two-Way Street

• Operations also has as much or more to learn from developers as well

Software Development Workflow

• Write some code• <ad-hoc verification here>• Go to pre-production• <ad-hoc verification here>• Go to production• Production failure

Software Development Workflow

• Write some code• Write and run some unit tests• Go to pre-production• Run some

integration/acceptance tests• Go to production• Lowered chance of

production failure

Old Chef Cookbook Workflow

• Write some cookbook code• <ad-hoc verification here>• Go to pre-production• <ad-hoc verification here• Go to production• Whoops, broke production

Chef Testing

• Did chef-client complete successfully?• Did the recipe put the node in the desired state?• Are the resources properly defined?• Does the code following our style guide?

New Chef Cookbook Workflow

• Write some cookbook code• Check for code correctness• Write and run some unit tests• Go to pre-production• Run some integration tests• Go to production

ChefDK: TDI In a Box

Rubocop Foodcritic

ChefSpec

Test Kitchen

Chef

Code Correctness

Unit Tests

Deploy sample environments

A wrapper to tie it all together

Not just syntactic correctness but quality gates too!

Provisioning

Rubocop Because Ruby

• Identify potential Ruby errors• Unclosed strings, etc.

• Identify style/convention that helps write better code• Single quotes vs. double quotes• This is useful for new Chefs, and helps make the code more readable

• Exclude rules that do not apply• Not everyone is the same – some tests won’t work for your organization or for Chef code

• Run against static code, so tests are very fast (<5 seconds to run)

Code Correctness

Rubocop Example

def badName if something test endend

test.rb:1:5: C: Use snake_case for methods and variables.def badName ^^^^^^^test.rb:2:3: C: Favor modifier if/unless usage when you have a single-line body. Another good alternative is the usage of control flow &&/||. if something ^^test.rb:4:5: W: end at 4, 4 is not aligned with if at 2, 2 end ^^^

1 file inspected, 3 offenses detected

Code Correctness

Food CriticTest Your “Chef Style”

• Flag problems that might cause your Chef cookbooks to fail on convergeFC010: Invalid search syntax

• Identify style/convention that has been adopted by the Chef communityFC004: Use a service resource to start and stop services

• Create custom rules for your own organization’s compliance/standardsCOMP001: Do not allow recipes to mount disk volumes

• Run against static code, so tests are very fast (<5 seconds to run)

Code Correctness

FoodCritic Examples• FC001: Use strings in preference to symbols to access node attributes• FC002: Avoid string interpolation where not required• FC003: Check whether you are running with chef server before using server-specific features

• FC004: Use a service resource to start and stop services• FC005: Avoid repetition of resource declarations• FC006: Mode should be quoted or fully specified when setting file permissions

• FC007: Ensure recipe dependencies are reflected in cookbook metadata

Code Correctness

ChefSpec Simulate Chef

• Did I send a properly formed piece of code to Chef?• Especially important if there is mutability in your code

• This can cover things like feature flags, or behavior that changes on a per-OS basis

• "If on Debian-based Linux I need to install package apache2, but on EL variants I need to install package httpd.”

• Tests are very fast to run• Even with simple code, it is useful for new Chefs to find simple errors

quickly• Useful for regression testing – to make sure new changes

don’t break stuff that worked before.

Unit Tests

ChefSpec Examplerequire 'chefspec'

describe 'example::default' do let(:chef_run) { ChefSpec::SoloRunner.converge(described_recipe) } let(:cheftest) { chef_run.node['cheftest'] }

it 'installs foo' do expect(chef_run).to install_package('foo') end

it 'sets the default attributes correctly' do expect(cheftest['package_name']).to eq('httpd') expect(cheftest['service_name']).to eq('httpd') expect(cheftest['web_root']).to eq('/var/www/html') endend

Unit Tests

Test KitchenLet’s do this (almost) for real• “Signal Out” testing for Chef code

• Just because I said something and it was interpreted correctly, doesn't mean that what I said is what I meant.

• Executes your Chef code on an actual, factual node• These nodes should be disposable (local virtualization, cloud instances,

etc.)• Use any number of “signal out” testing products to ensure expected

results• BATS• ServerSpec• Pester

• Can pull in other cookbook dependencies as well, and execute against a machine that looks the same as your standard image


Test Kitchen

it 'contains the default configuration settings' do file(File.join(node['chef_client']['conf_dir'], 'client.rb')).must_match('^chef_server_url') file(File.join(node['chef_client']['conf_dir'], 'client.rb')).must_match('^validation_client_name')end

it 'should configure the server with a started webserver' do expect(service('httpd')).to be_runningend

it 'should configure the server with an enabled webserver' do expect(service('httpd')).to be_enabledend

it 'should configure the server to listen on TCP port 80' do expect(port(80)).to be_listeningend


Testing for Compliance

Chef Analytics Ensures ComplianceSeparation of concerns is a thing• If my tests are failing, I can just re-write the tests, right?

• By using the Audit Mode of Chef Analytics in your pipeline, cookbooks have to pass your custom compliance rules in order to be published.

• These are the same compliance rules that run against existing nodes, so you don’t need to write separate tests.

• Compliance rules in Chef Analytics can (and should) be stored in a separate SCM repository than the code itself• Those who write the code, shouldn’t be able to write the rule check

Process and Workflow Enforces Standards• Chef enforces appropriate checks are being made and standards are being

enforced using FoodCritic.• This example will identify any code that tries to mount disk volumes. If code is

identified, it will be audited and then workflow can control the action of this deviation to standards.

Just because it’s published, doesn’t mean it’s used

• Chef Environments should be constrained to specific cookbook versions• If my Production environment is constrained to version 1.0.1 of the “MyApp” cookbook, it will

only use that version, even if a newer version is published to the Chef server• Incrementing the cookbook constraint can be done using your existing change

control mechanisms and separation of duties• A well-constructed pipeline will never overwrite an existing version of a cookbook

with new code/policy• BEST PRACTICE – have your pipeline auto-increment versions of cookbooks,

tied to build numbers, to have traceability all the way back to your SCM

Chef Workflow with Jenkins

Infrastructure Change Control PipelineChange/Feature

Request

Change Complete

Write/Modify Unit Test Write/Modify Cookbook Code Lint, Run Unit Tests Pass?

Push feature/hotfix branch to Git

Merge to Master, Release Artifact to Chef Server Update EnvironmentPass &

Approved?

Autobuild: Lint, Run Unit Tests, Run

Integration Tests

Dev

elop

er

Wor

ksta

tion

Con

tinuo

us In

tegr

atio

n &

G

it Sy

stem

s

Human Review of Pull Requests

Local Development

GitHub

Local Workstation

GitHubmaster

Clone Push

Develop Foodcritic ChefSpec Test KitchenCommit

GitHub

<USER>/<COOKBOOK>:master

DevOps/<COOKBOOK>:master

Fork

Pipeline Shape

We want to ensure that our changes are relevant and well tested before we deploy.

Verify Review Accept Ship

Verify Phase

During the verify phase, we want to do basic testing to build confidence that our changes are sane and at the very least, won’t break the build process


GitHub

Jenkins

GitHubmaster

Failed

Syntax

Foodcritic

ChefSpec

GitHub

<USER>/<COOKBOOK>:master


Pull Request

Pass

Fail

Fail

Pass

Pass

Pass

Test

Fail

Checkout

Verify Phase

Review Phase

Two heads are better than one! Ask your team members to double check your work.


Accept Phase

We need to test this in a production-like environment before we ship it.


Accept Job

GitHub

Jenkins

GitHubmaster

Notifications

Verify Job Tasks

Test Kitchen Serverspec


Fail

Pass

Pass

Test

Fail

Checkout

Update VersionUpload to Chef Server

Tag Release

Push

Ship Phase

The very last thing we need to do is start using the cookbook in our environments.


CHEF DELIVERY

VALIDATED IN OUR ENGAGEMENTS WITHENTERPRISE AND BIG WEB CUSTOMERS.

WE'VE IDENTIFIED A PROVEN PIPELINE

Stepsmanual automated

Workstation

Create NewChange1

Test ChangeLocally2

Build

Acceptance

Union

Rehearsal

Delivered

Verify

Chef Delivery

ApproveChange5

DeliverChange6

SubmitChange3

ReviewChange4

Stable Pipeline

Stepsmanual automated

VerifyLintSyntaxUnit

BuildMergeLintSyntaxUnitQualitySecurityPublish

ProvisionDeploySmokeFunctional

Acceptance

UnionProvisionDeploySmokeFunctional

RehearsalProvisionDeploySmokeFunctional

DeliveredProvisionDeploySmokeFunctional

Stages

customizable

Verify BuildAcceptance

UnionRehearsalDelivered

SubmitChange3

ReviewChange4

ApproveChange5

DeliverChange6

Chef Delivery

Create a new change1

Test ChangeLocally2

Workstation

Common Pipeline

BUILD COOKBOOK

├── recipes ├── default.rb

├── lint.rb ├── syntax.rb ├── unit.rb ├── quality.rb ├── security.rb ├── publish.rb ├── provision.rb

├── deploy.rb ├── smoke.rb

└── functional.rb

PHASE EXEC

log "Running unit"

repo = node['delivery_builder']['repo']

delivery_builder_exec “run my junit tests" do command "mvn test" cwd repoend

MANY ACCEPTANCE PIPELINES

ONE DELIVERY PIPELINE

ONE PIPELINEOne Pipeline

Delivery Pipeline

union rehearsal delivered

Acceptance Pipelines

review approve deliverChangeCookbook [A]

review approve deliverChangeCookbook [B]

review approve deliverChangeApplication [A]

review approve deliverChangeApplication [B]

Infrastructure & Applications

Cookbook WorkflowSupermarket

Chef Server

review approve deliverChangeCookbook

Node Node Node

Node Node Node

Node Node Node

Publis

h

Publish

Update

Application Workflow

review approve deliverChangeApplication

Node Node Node

Node Node Node

Node Node Node

Deploy

1 2 3

2 2 3

3 3 3

Chef Community Summit – LondonLondon, etc. Venues Monument – November 3rd & 4th

Why your participation matters • Influence the path of the Chef roadmap• Contribute to the formation of best practices and the avenues to best share them• Share your experiences transforming your business• Demonstrate your DevOps Kung Fu

Network with awesome engineers in the Community• Engage with a community of people actively using Chef to automate their workflow• Discuss “what keeps you up at night” with a passionate engaged audience• Meet with CHEF engineers IRL

**Use the code MEETUP and save 20% http://summit.chef.io

https://www.eventbrite.com/e/chef-community-summit-london-tickets-17402944676?discount=MEETUP

What Questions Can I Answer For You?Thank you!

@apmanly

Chef Cookbook Workflow

Business