Chef patterns

Chef  Pa(erns  From  Building  Clusters  

Biju  Nair  Boston  DevOps  Meetup  

08-­‐July-­‐2015  

Background  

•  Automate  build  &  management  of  clusters    – Hadoop  – KaLa…  etc  

•  Pa(erns  which  can  be  used  elsewhere  

Movies  On  Demand  

Service  On  Demand  

•  Common  services  which  can  be  requested  – Copy  logs  from  applicaQons  to  a  centralized  locaQon  

– Service  available  on  all  the  nodes  – ApplicaQons  can  request  the  service  dynamically  

Service  On  Demand  

•  Node  A(ribute  to  store  service  requests  default['bcpc']['hadoop']['copylog'] = {}

{ 'app_id' => { 'logfile' => "/path/file_name_of_log_file", 'docopy' => true (or false) },... }

•  Data  Structure  to  make  service  requests  

Service  On  Demand  

•  ApplicaQon  recipes  make  service  requests  # # Updating node attributes to copy HBase master log file to HDFS # node.default['bcpc']['hadoop']['copylog']['hbase_master'] = { 'logfile' => "/var/log/hbase/hbase-master-#{node.hostname}.log", 'docopy' => true }

node.default['bcpc']['hadoop']['copylog']['hbase_master_out'] = { 'logfile' => "/var/log/hbase/hbase-master-#{node.hostname}.out", 'docopy' => true }

Service  On  Demand  •  Service  recipe  node['bcpc']['hadoop']['copylog'].each do |id,f| if f['docopy'] template "/etc/flume/conf/flume-#{id}.conf" do source "flume_flume-conf.erb” action :create ... variables(:agent_name => "#{id}", :log_location => "#{f['logfile']}" ) notifies :restart,"service[flume-agent-multi-#{id}]",:delayed end service "flume-agent-multi-#{id}" do supports :status => true, :restart => true, :reload => false service_name "flume-agent-multi" action :start start_command "service flume-agent-multi start #{id}" restart_command "service flume-agent-multi restart #{id}" status_command "service flume-agent-multi status #{id}" end

•  Separate  role  at  the  end  of  run  list    

Choices  

Pluggable  Alerts  

•  Single  source  for  monitored  stats  – Allows  users  to  visualize  stats  across  different  parameters  

– Didn’t  want  to  duplicate  the  stats  collecQon  by  alerQng  system  

– Need  to  feed  data  to  the  alerQng  system  to  generate  alerts  

Pluggable  Alerts  

•  A(ribute  where  users  can  define  alerts  default["bcpc"]["hadoop"]["graphite"]["queries"] = { 'hbase_master' => [ { 'type' => "jmx", 'query' => "memory.NonHeapMemoryUsage_committed", 'key' => "hbasenonheapmem", 'trigger_val' => "max(61,0)", 'trigger_cond' => "=0", 'trigger_name' => "HBaseMasterAvailability", 'trigger_dep' => ["NameNodeAvailability"], 'trigger_desc' => "HBase master seems to be down", 'severity' => 1 },{ 'type' => "jmx", 'query' => "memory.HeapMemoryUsage_committed", 'key' => "hbaseheapmem", ... },...], ’namenode' => [...] ...}

Pluggable  Alerts  

•  Recipes  and  templates  use  the  data  structure  – To  generate  queries  to  pull  data  from  staQsQcs  store  and  send  •  h(ps://github.com/bloomberg/chef-­‐bach/blob/master/cookbooks/bcpc-­‐hadoop/templates/default/graphite.query_graphite.config.erb  

– To  create  requested  trigger  related  objects  in  alarming  system  •  h(ps://github.com/bloomberg/chef-­‐bach/blob/master/cookbooks/bcpc-­‐hadoop/recipes/graphite_to_zabbix.rb  

Pluggable  Alerts  

•  Servers  Defined  in  role  is  used  by  recipes  "default_attributes" : { "jmxtrans": { "servers": [ { "type": "hbase_master", "service": "hbase-master", "service_cmd": "org.apache.hadoop.hbase.master.HMaster” }, { "type": "hbase_rs", "service": "hbase-regionserver", "service_cmd": "org.apache.hadoop.hbase.regionserver.HRegionServer" } ] } ...

Dependency  

Service  Restart  

•  We  use  jmxtrans  to  monitor  jmx  stats  – Services  to  be  monitored  varies  with  node  

– There  can  be  more  than  one  service  to  be  monitored  

– Monitored  service  restart  requires  JMXtrans  to  be  restarted**  

Service  Restart  

•  Data  structure  in  roles  to  define  the  services  "default_attributes" : { "jmxtrans": { "servers": [ { "type": "datanode", "service": "hadoop-hdfs-datanode", "service_cmd": "org.apache.hadoop.hdfs.server.datanode.DataNode" }, { "type": "hbase_rs", "service": "hbase-regionserver", "service_cmd": “org.apache.hadoop.hbase.regionserver.HRegionServer" } ] } ...

Service  Restart  

•  Jmxtrans  service  restart  logic  built  dynamically  jmx_services = Array.new jmx_srvc_cmds = Hash.new node['jmxtrans']['servers'].each do |server| jmx_services.push(server['service']) jmx_srvc_cmds[server['service']] = server['service_cmd'] end

service "restart jmxtrans on dependent service" do service_name "jmxtrans" supports :restart => true, :status => true, :reload => true action :restart jmx_services.each do |jmx_dep_service| subscribes :restart, "service[#{jmx_dep_service}]", :delayed end only_if {process_require_restart?("jmxtrans","jmxtrans-all.jar", jmx_srvc_cmds)} end

Service  Restart  

def process_require_restart?(process_name, process_cmd, dep_cmds) tgt_proces_pid = `pgrep -f #{process_cmd}` ... tgt_proces_stime = `ps --no-header -o start_time #{tgt_process_pid}` ... ret = false restarted_processes = Array.new dep_cmds.each do |dep_process, dep_cmd| dep_pids = `pgrep -f #{dep_cmd}` if dep_pids != "" dep_pids_arr = dep_pids.split("\n") dep_pids_arr.each do |dep_pid| dep_process_stime = `ps --no-header -o start_time #{dep_pid}` if DateTime.parse(tgt_proces_stime) < DateTime.parse(dep_process_stime) restarted_processes.push(dep_process) ret = true end ...

External  Dependency  

Rolling  Restart    

•  Changes  to  configuraQon  •  Availability  – Toxic  ConfiguraQon  

•  ContenQon  – Poll  &  Wait  – Fail  the  Run  – Simply  Skip  Service  Restart  and  Go  On  •  Store  the  state  and  need  for  restart  •  Breaks  assumpQons  of  Procedural  Chef  Runs  

Rolling  Restart    

•  ZooKeeper  – Service  specific  znode  as  lock  

•  Node  a(ribute  to  flag  restart  failures  

h(ps://github.com/bloomberg/chef-­‐bach/blob/rolling_restart/cookbooks/bcpc-­‐hadoop/definiQons/hadoop_service.rb  

Change  Course  

Logic  InjecQon  

•  We  use  Community  cookbooks  – Takes  care  of  standard  install,  enable  and  starQng  of  services  

•  Need  to  add  logic  to  cookbook  recipes  – Take  acQon  on  a  service  only  when  condiQons  are  saQsfied  

– Take  acQon  on  a  service  based  on  dependent  service  state  

Logic  InjecQon  

kafka_install node.kafka.version_install_dir do from kafka_target_path not_if { kafka_installed? } end

template ::File.join(node.kafka.config_dir, 'server.properties') do source 'server.properties.erb’ ... helpers(Kafka::Configuration) if restart_on_configuration_change? notifies :restart, 'service[kafka]', :delayed end end

service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true action kafka_service_actions end

Logic  InjecQon  

•  Changes  to  standard  cookbook  – Create  a  new  recipe  to  perform  service  acQon  •  Resource  to  intercept  noQficaQons  to  service  resource  •  Original  service  resource    • Add  node  attribute  which  stores  name  of  new  recipe  • Update  original  recipe  – Remove  the  service  resource  from  the  original  recipe  

– Replace  it  with  include_recipe  new_a(ribute  

Logic  InjecQon  

•  New  recipe  to  perform  service  acQons  – First  step  is  the  ruby_block  to  intercept  noQficaQons  

ruby_block 'coordinate-kafka-start' do block do Chef::Log.debug 'Default recipe to coordinate Kafka start is used' end action :nothing notifies :restart, 'service[kafka]', :delayed end

service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true action kafka_service_actions end

Logic  InjecQon  

•  A(ribute  to  set  the  recipe  for  service  acQons  # # Attribute to set the recipe to used to coordinate Kafka service star # if nothing is set the default recipe ”_coordinate" will be used # default.kafka.start_coordination.recipe = 'kafka::_coordinate'

Logic  InjecQon  

•  Changes  to  the  original  recipe  kafka_install node.kafka.version_install_dir do from kafka_target_path not_if { kafka_installed? } end

template ::File.join(node.kafka.config_dir, 'server.properties') do source 'server.properties.erb’ ... helpers(Kafka::Configuration) if restart_on_configuration_change? notifies :create,'ruby_block[coordinate-kafka-start]’,immediately end end

include_recipe node.kafka.start_coordination.recipe

Logic  InjecQon  

•  Changes  in  wrapper  cookbook  – Create  custom  recipe  in  wrapper  cookbook  •  NoQficaQon  interceptor  ruby_block  should  be  first  •  Logic  to  determine  service  restart  acQon  

•  service  resource  •  Any  clean-­‐up  logic  – Overwrite  a(ribute  with  custom  recipe  name  

Logic  InjecQon  

ruby_block 'coordinate-kafka-start' do block do Chef::Log.info 'Custom recipe to coordinate Kafka start/restart' end ...

ruby_block 'restart-coordination' do block do Chef::Log.info 'Implement the process to coordinate the restart' end ...

service 'kafka' do provider kafka_init_opts[:provider] supports start: true, stop: true, restart: true, status: true ...

ruby_block 'restart-coordination-cleanup' do block do Chef::Log.info 'Implement any cleanup logic required' end

Logic  InjecQon  

•  Overwrite  a(ribute  to  set  the  custom  recipe    # # Overwrite the community cookbook attribute with custom recipe name # default[:kafka][:start_coordination][:recipe] = 'kafka-bcpc::coordinate'

QuesQons  ?  

References    

•  h(ps://github.com/bloomberg/chef-­‐bach  •  h(p://blog.asquareb.com/blog/categories/chef-­‐pa(erns/  

Thank  You!!  [email protected]