CIRCUIT 2015 - AEM Infrastructure Automation with Chef Cookbooks

CIRCUIT – An Adobe Developer Event Presented by ICF Interactive

AEM DevOps – AEM Infrastructure

Automation with Chef Cookbooks

Drew Glass Hero Digital

AEM Practice Lead

•  Director, Engineering and AEM Practice Lead at Hero Digital http://herodigital.com/

•  San Francisco AEM Community Leader http://www.meetup.com/Adobe-Experience-Manager-Adobe-Marketing-Cloud/

•  MS in Computer Science from University of Illinois Urbana-Champaign

drew@herodigital.com

@drewglass

https://www.linkedin.com/in/drewglass1

Agenda

•  What is DevOps •  DevOps with Chef •  AEM Cookbook •  Deployment Scenarios with Chef

What Is DevOps

•  Use code and automation to increase collaboration between development and operations

•  Consistent, testable, repeatable infrastructure

The Value in DevOps

•  Reduce cost and risk •  Increase throughput and delivery speed

DevOps with Chef

•  Tool to automate infrastructure provisioning through code

•  Thin Domain Specific Language built on top of Ruby

•  Provides abstraction for infrastructure •  https://www.chef.io/chef/

Chef Basics

•  Develop cookbooks on Chef workstation using ChefDK

•  Knife Cookbooks and Policies to Chef Server

•  Chef client provisions nodes using cookbooks and policies and records that information on Chef Server

Chef Basics Illustrated

Chef  Server  Nodes,  cookbooks,  

policies  

Worksta/on    ChefDK,  Knife,  cookbooks  

Node  Chef-­‐client  

Node  Chef-­‐client  

Node  Chef-­‐client  

Chef Workstation

Chef  Server  Nodes,  cookbooks,  

policies  

Worksta/on    ChefDK,  Knife,  cookbooks  

Node  Chef-­‐client  

Node  Chef-­‐client  

Node  Chef-­‐client  

Chef Workstation

•  Develop cookbooks and policies •  Store cookbooks in chef-repo which is synced

with version control •  ChefDK - development kit which includes Kitchen,

ChefSpec, Berkshelf https://downloads.chef.io/chef-dk/

•  Knife – used to upload items such as cookbooks to Chef server and interact with nodes

•  Kitchen - test cookbooks across different platforms

•  ChefSpec - simulate convergence of resources on a node

•  Berkshelf - manage cookbook dependencies

Cookbooks

•  Fundamental unit of configuration •  Contains attributes, recipes, templates,

files, providers, and resources •  Kinds of cookbooks: base and wrapper

AEM Community Base Cookbook h6ps://github.com/tacitknowledge/aem-­‐cookbook    

AEM Community Base Cookbook

AEM Wrapper Cookbook

•  Depends on base cookbook •  Overrides base attributes and may include

new recipes, files, providers and templates

Example  metadata.rb  

Attributes in a Cookbook

•  Configuration data •  Defined in a cookbook and can override

attributes on a node •  When chef-client is run, attributes in

cookbook are compared against attributes on node

•  Attributes in default.rb are loaded first

Attributes in the AEM Cookbook default[:aem][:version] = nil default[:aem][:jvm_opts] = {} default[:aem][:jar_opts] = [] default[:aem][:enable_webdav] = false default[:aem][:jar_name] = nil default[:aem][:use_yum] = false default[:aem][:download_url] = nil default[:aem][:license_url] = nil default[:aem][:base_dir] = "/opt/aem" default[:aem][:cluster_name] = nil default[:aem][:author] = { :default_context => "/opt/aem/author", :port => "4502", :runnable_jar => "aem-author-p4502.jar", :base_dir => "/opt/aem/author/crx-quickstart", :jvm_opts => {}, :ldap => { :enabled => false, :options => {} },

:validation_urls => [ "http://localhost:4502/libs/cq/core/content/login.html", "http://localhost:4502/damadmin", "http://localhost:4502/miscadmin", "http://localhost:4502/system/console/bundles" ], :deploy_pkgs => [], #You changed these, right? :admin_user => "admin", :admin_password => "admin", :new_admin_password => nil, :replication_hosts => [], :find_replication_hosts_dynamically => false } default[:aem][:author][:startup][:max_attempts] = 20 default[:aem][:author][:startup][:wait_between_attempts] = 30

Attributes in the AEM Cookbook

Recipes in a Cookbook

•  Fundamental configuration element  •  Authored with Ruby and used to define

everything to configure part of system •  Executed in order specified in run list •  In AEM, we have recipes for author,

publish and dispatcher

Author Recipe in the AEM Cookbook

Author Recipe in an AEM Cookbook

Files and Templates in Cookbooks

•  Templates - file written in markup language to solve a more complex configuration scenario

•  Files - can be transferred from cookbooks to nodes

Dispatcher Attributes

Dispatcher Template in the AEM Cookbook

Dispatcher Template in the AEM Cookbook

Resources and Providers

•  Resources - package, service, etc. that tells chef-client which provider to use during a during a chef client run for various tasks

•  Provider  - defines steps to bring part of system into desired state

Author Recipe Calls Init Provider

Init Script Provider in the AEM Cookbook

Author Recipe Calls Replication Agent Provider

Replication Agent Provider in the AEM Cookbook

Chef Server

Chef  Server  Nodes,  cookbooks,  

policies  

Worksta/on    ChefDK,  Knife,  cookbooks  

Node  Chef-­‐client  

Node  Chef-­‐client  

Node  Chef-­‐client  

Chef Server

•  Hub of configuration data •  Stores cookbooks, node information and

policies •  Chef-client running on nodes queries Chef

server for configuration information like recipes

•  Chef-client does the actual configuration on the node

•  Allows for distribution of work •  Also includes search

Chef-client

Chef  Server  Nodes,  cookbooks,  

policies  

Worksta/on    ChefDK,  Knife,  cookbooks  

Node  Chef-­‐client  

Node  Chef-­‐client  

Node  Chef-­‐client  

Chef-client

•  Agent that runs on a node managed by Chef •  Chef-client is run to bring a node into a

desired state •  Typical chef-client run:

–  register and authenticate node with Chef server (RSA public key-pairs)

– builds the node object –  synchronizes cookbooks –  compiles resource collection by loading required

cookbooks (recipes, attributes, and all other dependencies)

– provisions node

Chef-client Run for Author Node in Dev Environment

Node

•  Physical, virtual or cloud machine that is configured and maintained by chef-client

•  Physical nodes are typically a server or virtual machine that has to have a network interface for chef-client to communicate with Chef server

•  Cloud node could be Amazon EC2, Google Compute Engine, Backspace, etc.

•  Chef-client can be used to deploy, configure and maintain

Nodes in AEM

AEM  Author  

AEM  Publish  

Apache  with  

Dispatcher  

Node Objects

•  Attributes – configuration data for a node – Describe the current state of a node – Can be defined by: cookbooks, roles,

environments •  Run List - ordered list of roles and/or

recipes that defines all of the information Chef needs to configure a node to a certain state

Runlist in AEM

Policy

•  Map business and operational requirements to settings and objects on Chef server

•  Roles - define server types – A role has zero or more attributes and  a run

list •  Environments

– Define attributes for environments - Integration, QA, Staging, Production

Roles in AEM

Environments in AEM

Deployment Scenarios

•  AEM sandbox using Vagrant, Chef and Berkshelf

•  Local AEM development environment using a AEM site wrapper cookbook

•  AEM demo environment in the cloud using Chef and Kitchen

•  Continuous delivery pipeline with Chef and Chef provisioning

Worksta/on  

Local Dev VM or Cloud Instance

VM  or  Cloud  Instance  

Vagrant  or    

Kitchen    

Cookbooks  and  Berkshelf  

Chef-­‐solo  

AEM  Author  

AEM  Publish  

Dispatcher  

Vagrant and Chef Demo

AEM Environment with Chef

Chef  Server  

with  Nodes,  Cookbooks,  and  Policies  

Node  

AEM  Author  

Chef-­‐  client  

Node  

AEM  Publish  

Chef-­‐client  

Node  

Dispat-­‐cher  

Chef-­‐client  

Chef Provisioning

•  Allows clusters to be managed with chef-client and Chef server using recipes

•  Collection of resources to enable creation of machines using chef-client

•  Bootstrap operations that are done against an infrastructure like VirtualBox, Amazon EC2, Docker, etc. using drivers

•  Machines – Chef provisioning runs multiple machine processes in-parallel in most situations

Scaling with Chef Provisioning

Chef  Server  

with  Nodes,  Cookbooks,  and  Policies  

Node  

AEM  Author  

Chef-­‐  client  

Dispatcher  Node  

Publish    Node  

Dispatcher  Node  

Publish    Node  

Scaling with Chef Provisioning

Chef  Server  

with  Nodes,  Cookbooks,  and  Policies  

Node  

AEM  Author  

Chef-­‐  client  

Dispatcher  Node  

Publish    Node  

Dispatcher  Node  

Publish    Node  

Dispatcher  Node  

Publish    Node  

Dispatcher  Node  

Publish    Node  

Wrapping Up

•  Infrastructure as code •  AEM DevOps with Chef •  Deploying AEM with Chef

Q & A

Send any questions to drew@herodigital.com