Transcript
Chef Cookbook WorkflowTesting your infrastructure code
Alex Manly - @apmanly
Instructor
• Alex Manly• Solutions Architect, Chef• Developer (Java, oh and some Ruby)• @apmanly• am@chef.io
What is Chef?
Short Answer: An Automation Framework for automating infrastructure and applications
Traditional Systems Management
Config 1
Config 2
Config 3
Traditional Systems Management
Config 1
Config 2
Config 3
Autom
ation
Idea of Services
Service
Config 1
Config 2
Config 3
Autom
ation
Abstraction of Services
Service
Config 1
Config 2
Config 3
Autom
ation
Chef is Infrastructure as Code
•Programmatically provision and configure components
http://www.flickr.com/photos/louisb/4555295187/
Chef is Infrastructure as Code
•Treat like any other code base
http://www.flickr.com/photos/louisb/4555295187/
Chef is Infrastructure as Code
•Reconstruct business from code repository, data backup, and compute resources
http://www.flickr.com/photos/louisb/4555295187/
Chef is Infrastructure as Code•Programmatically provision and configure components
•Treat like any other code base
•Reconstruct business from code repository, data backup, and compute resources
http://www.flickr.com/photos/louisb/4555295187/
Building Blocks: What is a Resource?• A Resource is a system state you define
Example: Package installed, state of a service, configuration file existing• You declare what state you want the resource in.
Chef automatically determines HOW that state is achieved
On Linux based OSes: On Windows based OSes:
Resource Example#windowsdsc_resource 'webserver' do
resource :windowsfeature property :name, 'Web-Server' property :ensure, 'Present’
end
Resource Example#linuxpackage "httpd" do
action :installend
Building Blocks: What is a Recipe?• An abstraction of a Service that consists of a set of Resources to deliver that Service
• Resources are executed in the order they are listed.
Recipe Example#linuxpackage "httpd" do
action :installend
include_recipe "apache::fwrules”
service "httpd" doaction [ :enable, :start ]
end
Recipe Example#windowsinclude_recipe "fourthcoffee::dsc”include_recipe "iis::remove_default_site”remote_directory node['fourthcoffee']['install_path'] do
source 'fourthcoffee’ action :create
endiis_pool 'FourthCoffee' do
runtime_version "4.0" action :add
endiis_site 'FourthCoffee' do
protocol :http port 80 path node['fourthcoffee']['install_path'] application_pool 'FourthCoffee' action [:add,:start]
end
Cookbooks
•A Higher Level Abstraction of a Service
•A set of Recipes and Data Attributes required to deliver one or multiple Services
Define cookbook attribute#attributes.rbdefault['fourthcoffee']['port'] = 80
Consume cookbook attribute
iis_site 'FourthCoffee' do protocol :http port node['fourthcoffee']['port'] path node['fourthcoffee']
['install_path'] application_pool 'FourthCoffee' action [:add,:start]
end
Yes!
That’s cool but…• Things break• Chef is a language (based on Ruby)• How can you rapidly develop recipes and cookbooks?
Let’s step back…
Testing is Hard - FACT
Automated GUI Tests
Integration TestsUnit
Tests
Manual TestsAutomate
dGUI Test
sAutomated API
TestsAutomated Integration Tests
Automated Component Test
Automated Unit Tests
Manual Session Based Testing
Also known as…
You’ll never get to Continuous Deployment
clicking a GUI
Theory of testing…
Testing builds safety
Feedback loops…• Tell us we’re doing the right thing• At the right time• With the right results
Feedback loops…
Measurements we take to ensure the “experiment” is behaving as expected
Tests are essentially feedback loops
Remember…
Chef is “Infrastructure as Code”
Remember…
“Infrastructure as Code” should betreated like ANY other codebase.
Treated Any Other Codebase…• Stored in SCM• Testing Coverage• Part of your CI pipelines
Testing in Chef• Chef recipes need tested
Linting Static Analysis Unit Testing Functional Testing
DevOps is a Two-Way Street
• It’s great when developers care about
• Uptime!• Scaling!• Deployment!• Put them on call! etc.
etc. etc.
DevOps is a Two-Way Street
• Operations also has as much or more to learn from developers as well
Software Development Workflow
• Write some code• <ad-hoc verification here>• Go to pre-production• <ad-hoc verification here>• Go to production• Production failure
Software Development Workflow
• Write some code• Write and run some unit tests• Go to pre-production• Run some
integration/acceptance tests• Go to production• Lowered chance of
production failure
Old Chef Cookbook Workflow
• Write some cookbook code• <ad-hoc verification here>• Go to pre-production• <ad-hoc verification here• Go to production• Whoops, broke production
Chef Testing
• Did chef-client complete successfully?• Did the recipe put the node in the desired state?• Are the resources properly defined?• Does the code following our style guide?
New Chef Cookbook Workflow
• Write some cookbook code• Check for code correctness• Write and run some unit tests• Go to pre-production• Run some integration tests• Go to production
ChefDK: TDI In a Box
Rubocop Foodcritic
ChefSpec
Test Kitchen
Chef
Code Correctness
Unit Tests
Deploy sample environments
A wrapper to tie it all together
Not just syntactic correctness but quality gates too!
Provisioning
Rubocop Because Ruby
• Identify potential Ruby errors• Unclosed strings, etc.
• Identify style/convention that helps write better code• Single quotes vs. double quotes• This is useful for new Chefs, and helps make the code more readable
• Exclude rules that do not apply• Not everyone is the same – some tests won’t work for your organization or for Chef code
• Run against static code, so tests are very fast (<5 seconds to run)
Code Correctness
Rubocop Example
def badName if something test endend
test.rb:1:5: C: Use snake_case for methods and variables.def badName ^^^^^^^test.rb:2:3: C: Favor modifier if/unless usage when you have a single-line body. Another good alternative is the usage of control flow &&/||. if something ^^test.rb:4:5: W: end at 4, 4 is not aligned with if at 2, 2 end ^^^
1 file inspected, 3 offenses detected
Code Correctness
Food CriticTest Your “Chef Style”
• Flag problems that might cause your Chef cookbooks to fail on convergeFC010: Invalid search syntax
• Identify style/convention that has been adopted by the Chef communityFC004: Use a service resource to start and stop services
• Create custom rules for your own organization’s compliance/standardsCOMP001: Do not allow recipes to mount disk volumes
• Run against static code, so tests are very fast (<5 seconds to run)
Code Correctness
FoodCritic Examples• FC001: Use strings in preference to symbols to access node attributes• FC002: Avoid string interpolation where not required• FC003: Check whether you are running with chef server before using server-specific features
• FC004: Use a service resource to start and stop services• FC005: Avoid repetition of resource declarations• FC006: Mode should be quoted or fully specified when setting file permissions
• FC007: Ensure recipe dependencies are reflected in cookbook metadata
Code Correctness
ChefSpec Simulate Chef
• Did I send a properly formed piece of code to Chef?• Especially important if there is mutability in your code
• This can cover things like feature flags, or behavior that changes on a per-OS basis
• "If on Debian-based Linux I need to install package apache2, but on EL variants I need to install package httpd.”
• Tests are very fast to run• Even with simple code, it is useful for new Chefs to find simple errors
quickly• Useful for regression testing – to make sure new changes
don’t break stuff that worked before.
Unit Tests
ChefSpec Examplerequire 'chefspec'
describe 'example::default' do let(:chef_run) { ChefSpec::SoloRunner.converge(described_recipe) } let(:cheftest) { chef_run.node['cheftest'] }
it 'installs foo' do expect(chef_run).to install_package('foo') end
it 'sets the default attributes correctly' do expect(cheftest['package_name']).to eq('httpd') expect(cheftest['service_name']).to eq('httpd') expect(cheftest['web_root']).to eq('/var/www/html') endend
Unit Tests
Test KitchenLet’s do this (almost) for real• “Signal Out” testing for Chef code
• Just because I said something and it was interpreted correctly, doesn't mean that what I said is what I meant.
• Executes your Chef code on an actual, factual node• These nodes should be disposable (local virtualization, cloud instances,
etc.)• Use any number of “signal out” testing products to ensure expected
results• BATS• ServerSpec• Pester
• Can pull in other cookbook dependencies as well, and execute against a machine that looks the same as your standard image
Deploy sample environments
Test Kitchen
it 'contains the default configuration settings' do file(File.join(node['chef_client']['conf_dir'], 'client.rb')).must_match('^chef_server_url') file(File.join(node['chef_client']['conf_dir'], 'client.rb')).must_match('^validation_client_name')end
it 'should configure the server with a started webserver' do expect(service('httpd')).to be_runningend
it 'should configure the server with an enabled webserver' do expect(service('httpd')).to be_enabledend
it 'should configure the server to listen on TCP port 80' do expect(port(80)).to be_listeningend
Deploy sample environments
Testing for Compliance
Chef Analytics Ensures ComplianceSeparation of concerns is a thing• If my tests are failing, I can just re-write the tests, right?
• By using the Audit Mode of Chef Analytics in your pipeline, cookbooks have to pass your custom compliance rules in order to be published.
• These are the same compliance rules that run against existing nodes, so you don’t need to write separate tests.
• Compliance rules in Chef Analytics can (and should) be stored in a separate SCM repository than the code itself• Those who write the code, shouldn’t be able to write the rule check
Process and Workflow Enforces Standards• Chef enforces appropriate checks are being made and standards are being
enforced using FoodCritic.• This example will identify any code that tries to mount disk volumes. If code is
identified, it will be audited and then workflow can control the action of this deviation to standards.
Just because it’s published, doesn’t mean it’s used
• Chef Environments should be constrained to specific cookbook versions• If my Production environment is constrained to version 1.0.1 of the “MyApp” cookbook, it will
only use that version, even if a newer version is published to the Chef server• Incrementing the cookbook constraint can be done using your existing change
control mechanisms and separation of duties• A well-constructed pipeline will never overwrite an existing version of a cookbook
with new code/policy• BEST PRACTICE – have your pipeline auto-increment versions of cookbooks,
tied to build numbers, to have traceability all the way back to your SCM
Chef Workflow with Jenkins
Infrastructure Change Control PipelineChange/Feature
Request
Change Complete
Write/Modify Unit Test Write/Modify Cookbook Code Lint, Run Unit Tests Pass?
Push feature/hotfix branch to Git
Merge to Master, Release Artifact to Chef Server Update EnvironmentPass &
Approved?
Autobuild: Lint, Run Unit Tests, Run
Integration Tests
Dev
elop
er
Wor
ksta
tion
Con
tinuo
us In
tegr
atio
n &
G
it Sy
stem
s
Human Review of Pull Requests
Local Development
GitHub
Local Workstation
GitHubmaster
Clone Push
Develop Foodcritic ChefSpec Test KitchenCommit
GitHub
<USER>/<COOKBOOK>:master
DevOps/<COOKBOOK>:master
Fork
Pipeline Shape
We want to ensure that our changes are relevant and well tested before we deploy.
Verify Review Accept Ship
Verify Phase
During the verify phase, we want to do basic testing to build confidence that our changes are sane and at the very least, won’t break the build process
Verify Review Accept Ship
GitHub
Jenkins
GitHubmaster
Failed
Syntax
Foodcritic
ChefSpec
GitHub
<USER>/<COOKBOOK>:master
DevOps/<COOKBOOK>:master
Pull Request
Pass
Fail
Fail
Pass
Pass
Pass
Test
Fail
Checkout
Verify Phase
Review Phase
Two heads are better than one! Ask your team members to double check your work.
Verify Review Accept Ship
Accept Phase
We need to test this in a production-like environment before we ship it.
Verify Review Accept Ship
Accept Job
GitHub
Jenkins
GitHubmaster
Notifications
Verify Job Tasks
Test Kitchen Serverspec
DevOps/<COOKBOOK>:master
Fail
Pass
Pass
Test
Fail
Checkout
Update VersionUpload to Chef Server
Tag Release
Push
Ship Phase
The very last thing we need to do is start using the cookbook in our environments.
Verify Review Accept Ship
CHEF DELIVERY
VALIDATED IN OUR ENGAGEMENTS WITHENTERPRISE AND BIG WEB CUSTOMERS.
WE'VE IDENTIFIED A PROVEN PIPELINE
Stepsmanual automated
Workstation
Create NewChange1
Test ChangeLocally2
Build
Acceptance
Union
Rehearsal
Delivered
Verify
Chef Delivery
ApproveChange5
DeliverChange6
SubmitChange3
ReviewChange4
Stable Pipeline
Stepsmanual automated
VerifyLintSyntaxUnit
BuildMergeLintSyntaxUnitQualitySecurityPublish
ProvisionDeploySmokeFunctional
Acceptance
UnionProvisionDeploySmokeFunctional
RehearsalProvisionDeploySmokeFunctional
DeliveredProvisionDeploySmokeFunctional
Stages
customizable
Verify BuildAcceptance
UnionRehearsalDelivered
SubmitChange3
ReviewChange4
ApproveChange5
DeliverChange6
Chef Delivery
Create a new change1
Test ChangeLocally2
Workstation
Common Pipeline
BUILD COOKBOOK
├── recipes ├── default.rb
├── lint.rb ├── syntax.rb ├── unit.rb ├── quality.rb ├── security.rb ├── publish.rb ├── provision.rb
├── deploy.rb ├── smoke.rb
└── functional.rb
PHASE EXEC
log "Running unit"
repo = node['delivery_builder']['repo']
delivery_builder_exec “run my junit tests" do command "mvn test" cwd repoend
MANY ACCEPTANCE PIPELINES
ONE DELIVERY PIPELINE
ONE PIPELINEOne Pipeline
Delivery Pipeline
union rehearsal delivered
Acceptance Pipelines
review approve deliverChangeCookbook [A]
review approve deliverChangeCookbook [B]
review approve deliverChangeApplication [A]
review approve deliverChangeApplication [B]
Infrastructure & Applications
Cookbook WorkflowSupermarket
Chef Server
review approve deliverChangeCookbook
Node Node Node
Node Node Node
Node Node Node
Publis
h
Publish
Update
Application Workflow
review approve deliverChangeApplication
Node Node Node
Node Node Node
Node Node Node
Deploy
1 2 3
2 2 3
3 3 3
Chef Community Summit – LondonLondon, etc. Venues Monument – November 3rd & 4th
Why your participation matters • Influence the path of the Chef roadmap• Contribute to the formation of best practices and the avenues to best share them• Share your experiences transforming your business• Demonstrate your DevOps Kung Fu
Network with awesome engineers in the Community• Engage with a community of people actively using Chef to automate their workflow• Discuss “what keeps you up at night” with a passionate engaged audience• Meet with CHEF engineers IRL
**Use the code MEETUP and save 20% http://summit.chef.io
What Questions Can I Answer For You?Thank you!
@apmanly
top related