Chef Cookbook Design Patterns Eric Krupnik Software Engineer Copyright Clearance Center
05/02/20232
Who Am I?
• Software Engineer
• Platform Engineering at CCC
• Chef user since May, 2013
github.com/ekrupnik
05/02/20233
Who Are You?
• Who has heard of Chef?
• Who uses Chef?– Other configuration management tools?
05/02/20234
Agenda
• Introduction
• Chef overview
• Cookbook types
• Data bags vs attributes
• Run lists and cookbook management
• Q and A
05/02/20236
Issues To Address
• Large volume of applications– Number of apps
– Many environments
– Too much to handle manually!
• Error prone configuration during deployments
05/02/20237
Sheer Volume
• Many environments (columns)
• Many applications (rows)
• This leads to error prone deployments and inevitable configuration issues
• As of May, 2015 more than 180 deployable components over 6 major environments!
05/02/20239
What is Chef? Infrastructure as Code?
• Configuration management tool
• Infrastructure as code– Executable text
• Don’t write MS Word documents anymore
– Version control• Track how infrastructure changes over time• Shareable
– Idempotent• Write code to define how a machine should look• Run Chef often to prevent configuration drift
– Testable
05/02/202310
Chef Terms
• A cookbook is the versioned unit of distribution– We will cover cookbook types later
– Contains 1+ recipe(s)
• A recipe is where a user defines a set of resources Chef manages
• Recipe should manage a group of related resources
• A resource is “something” Chef manages and configures– Package, directory, file, service, user
– Can be configured with attributes
– Can create custom resources
05/02/202311
Chef Resource Example
directory "/tmp/folder" do
owner 'root'
group 'root'
.
.
mode node['folder_priv']
action :create
end
Declaration of resource
Hard coded attribute
Hard coded attribute
.
.
Node attribute
Action to perform (create, delete)
end
05/02/202313
Cookbook Types
• Application cookbook
• Library cookbook
• Wrapper cookbook– Run differently configured instances of application
– Won’t cover this today because it is rare
• Community cookbook– The community maintains many cookbooks you can use
– Won’t cover this today
05/02/202314
Application Cookbook
• One recipe per deployable component
• Define attributes and configurations needed for application– Attributes like application name
– Property file(s)
• Should be fairly lightweight
• Should depend on one or more library cookbooks
• Suggested naming conventions:– Cookbook – for the application it manages
– Recipes – for the individual deployable component
05/02/202315
Application Cookbook
cookbooks/myapp /attributes ui.rb rest_svc.rb /files/test ui_test.rb rest_svc_test.rb /recipes ui.rb rest_svc.rb /templates ui_properties.erb rest_svc_properties.erb
metadata.rb Berksfile .kitchen.yml
05/02/202316
Library Cookbook
• Defines a standard way to deploy an artifact
• Consumed by application cookbooks
• Can link multiple library cookbooks together
• Keeps application cookbooks DRY
• Contains any custom resources– Act on attributes defined by application cookbook
– Has defaults, can be overwritten by application cookbook
05/02/202317
Custom Resources
• This is called an LWRP (Lightweight Resource Provider)– Heavy weight resource providers also exist
• These should be the biggest worker in your cookbooks. It’s reusable, configurable, and defines how an application should be deployed.
• Two primary files– Resource: defines accepted attributes/configuration
– Provider: defines the “how” and does the work
– Each named after the resource you are defining
05/02/202318
Custom Resources (Resource)
lib_cookbook/resources/webapp.rb
actions :install, :delete
attribute :app_name, :kind_of => String, :required => trueattribute :port, :kind_of => String, :required => true
.
.
attribute :property_file_tempalte, :kind_of => String, :required => false
default_action :install
05/02/202319
Custom Resources (Provider)lib_cookbook/provider/webapp.rb
action :install do
app = new_resource.app_name java_version = Helpers.get_java_version(app)
java_install java_version do java_version java_version install_dir “/ccc/java” end
remote_file “#{app}” do source Helpers.get_artifactory_url(app) owner “root” end
template “/ccc/apps/#{app}/config/#{app}.properties” do source property_file_template mode 0644 variables( :data_bag_values => Helpers.get_data_bag_values(app) ) end
end
05/02/202321
Defining Configurations
• So far, we have covered setting attributes in application cookbooks– Pros:
• Forced standard in all environments• Versioned as part of a cookbook
– Cons:• Can’t change attribute values between environments• Hard to change “quickly and easily”
• Data bags are the answer!– Data bag contains 1 data bag item (JSON file) per environment
– 1:1 data bag to cookbook (same name even!)
– Live configuration data including application version
05/02/202322
Data Bag Example
data_bags/myapp/DEV.json
{ “id”: “DEV”, “myapp.welcomePage.title”: “Welcome to our app!”, “myapp.error.message”: “This is DEV – what did you expect! Of course it crashed!”, “myapp.error.email.recipient.list”: “[email protected], [email protected]” }
data_bags/myapp/QA.json
data_bags/myapp/UAT.json
data_bags/myapp/PRD.json
{ “id”: “PRD”, “myapp.welcomePage.title”: “Welcome to our app!”, “myapp.error.message”: “We’re sorry something went wrong.”, “myapp.error.email.recipient.list”: “[email protected]” }
05/02/202323
Data Bags in Action
• Remember this from the custom resource? template “/ccc/apps/#{app}/config/#{app}.properties” do source property_file_template variables( :data_bag_values => Helpers.get_data_bag_values(app) ) end
• Since the templates are erb files which allow deploy time variable replacement, we can use data bag values to generate a properties file template for the appropriate environment as we deploy.
05/02/202324
Data Bags in Action
• Properties file template: cookbooks/myapp/templates/myapp.properties.erb
myapp.welcomePage.title=<%= @data_bag_values[‘myapp.welcomPage.title’] %> myapp.error.message=<%= @data_bag_values[‘myapp.error.message’] %> myapp.error.email.recipient.list=<%= @data_bag_values[‘myapp.error.email.recipient.list’] %>
• Generated at deploy time by our custom resource we get: DEV: myapp.welcomePage.title=Welcome to our app! myapp.error.message=This is DEV – what did you expect! Of course it crashed! [email protected], [email protected]
PRD: myapp.welcomePage.title=Welcome to our app! myapp.error.message=We’re sorry something went wrong. [email protected]
05/02/202326
Run Lists
• A run list defines which recipes are needed to run during a Chef run.
• Run lists are stored on the Chef Server, and are specific to a node.
• Since library cookbooks only get called by application cookbooks, no library cookbook recipes should go into the run list
• Run lists are most commonly defined in either:– Chef role
– Chef environment file
05/02/202327
Roles vs Environment Files
• Both are unversioned, but roles tend to be more dangerous due to more global effect
• Both CAN define global or environment-wide attributes
• Environment files contain cookbook version to use
• Simple to keep cookbook versions and run lists in same place
• We do not use roles for these reasons!
05/02/202328
Environment Files Example
environments/DEV.rb name “DEV” cookbook_versions({ “myapp” => “= 3.1.0”, “newapp” => “= 1.4.0”, . . )} run_lists({ “ui-server” => “recipe[my_app::ui]”, “svc-server” => “recipe[my_app::rest_svc], recipe[newapp::rest_svc]” )}
environments/QA.rbenvironments/UAT.rb
environments/PRD.rb name “PRD” cookbook_versions({ “myapp” => “= 3.1.0”, . . )} run_lists({ “ui-server” => “recipe[my_app::ui]”, “svc-server” => “recipe[my_app::rest_svc]” )}
05/02/202329
Cookbook Dependencies
• Cookbooks can depend on other cookbooks– Application cookbook depends on library cookbook(s)
– Library cookbook depends on other library cookbook(s)
• Each cookbook has a metadata.rb file– Defines info on cookbook
– Defines cookbook dependencies
– Doesn’t matter if library or application cookbook
• There are tools like Berkshelf and Librarian– Berkshelf is now included in Chef-DK!
05/02/202330
Cookbook metadata.rb Example
name “myapp”
maintainer “Copyright Clearance Center”
maintainer_email “[email protected]”
description “App cookbook to install myapp”
version “3.1.0”
depends “library_cookbook_1”, “=5.0.0”
depends “library_cookbook_2”
depends “library_cookbook_3”, “~> 1.0”
depends “community_cookbook”
05/02/202331
Summary
• Recipes should be light weight and very focused
• Use library cookbooks with custom resources
• Library cookbooks can use other library cookbooks
• Use a mix between attributes and data bags– Standard configurations in attributes
– Environment specific configurations in data bags
• Use environment files for cookbook versions and run lists