Chef on Python and MongoDB

Rick  Copeland  @rick446  Arborian  Consulting,  LLC  

  Infrastructure  as  Code  

  Resources  &  Providers  

  Cookbooks,  Recipes,  Clients,  and  Nodes  

Chef  Server    (API)  

RabbitMQ  RabbitMQ  RabbitMQ  

CouchDB   Solr  

Solr  Indexer  

chef-­‐client  knife  Chef  Server  (Web  UI)  

HTTP  REST  API  

Build,  register,  and  authenticate  the  node  

Synchronize  cookbooks  

Build  resource  collection  (run  the  recipes  in  order)  

Configure  node  (“converge”)  

Run  notification  handlers  

Chef  Server    (API)  

RabbitMQ  RabbitMQ  RabbitMQ  

CouchDB   Solr  

Solr  Indexer  

chef-­‐client  knife  Chef  Server  (Web  UI)  

HTTP  REST  API  

Ruby  

Ruby  

Ruby  

  Chef  assumes  a  bootstrapped  node  exists  

  Chef  doesn’t  keep  release  notes  

  Code  and  infrastructure  are  versioned  differently  

  Solution:  Web  app  to  manage  deployments  &  generate  release  notes  

MongoDB  

Solr  

Solr  Indexer  

chef-­‐client  knife  

HTTP  REST  API  

MonQ  

Chef  Server    (API  +  web)  

Ruby  

Python  

  Reduce  #  of  processes  &  technologies  

  Don’t  know  Ruby  well  

  Keep  private  keys  out  of  the  system  

  Integrate  with  existing  authentication  

  Performance  

/clients  

/data  

/roles  

/nodes  

/cookbooks  

/environments  

/sandboxes  

/search  

 Mostly  JSON    almost  BSON    References  to  Ruby  files  stored  separately  

{      "name":  "allura-­‐0.0.1”,…      "json_class":  "Chef::CookbookVersion",      "attributes":  [          {              "name":  "default.rb",              "url":  "https://s3.amazonaws.com/opscode-­‐platform-­‐production-­‐data/…                                  "checksum":  "749a3a328d6c47a32d134b336183981f",              "path":  "attributes/default.rb",              "specificity":  "default”…  

Ruby  files  stored  on  S3  

role = collection( 'chef.role', doc_session, Field('_id', S.ObjectId()), Field('account_id', S.ObjectId(if_missing=None)), Field('name', str), Field('description', str), Field('default_attributes', str), Field('override_attributes', str), Field('run_list', [ str ] ), Field('env_run_lists', { str: [ str ]}), Index('account_id', 'name', unique=True))

MongoDB  Validator  

Shorthand  with  Python  Types  

Embedded  Documents  

Index  Definitions  

class Role(object):

def url(self): return request.relative_url( config.chef_api_root + '/roles/' + self.name)

def __json__(self): return dict( chef_type='role', json_class='Chef::Role',! … default_attributes=loads(self.default_attributes),! …)

def update(self, d): self.name = d['name'] … self.default_attributes = dumps(d['default_attributes']) self.override_attributes = dumps(d['override_attributes'])! …

Models  know  where  they  live  

Models  can  be  turned  into  dict  (to  

be  JSONified)  

Models  can  be  updated  from  dict  

class RoleSchema(JSONModelSchema):!

model_class=CM.role!

chef_type='role’!

json_class='Chef::Role’!

exclude_fields=['_id', 'account_id']

  /foo/bar/baz    root.foo.bar.baz  

  Lots  of  decorators    Validation  (params/body    **kwargs)    Authorization    Rendering  

  HTTP  method  lookup    GET  /foo/bar    root.foo.bar._get()  

class RolesController(RESTController):

@expose(template_engine='json') def _get(self):…

@expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _post(self, **kwargs):…

def __getattr__(self, name): return RoleController(name)

Returns  JSON  

Only  admins  can  access  

Convert  and  Validate  POST  

Continue  dotted  lookup  

class RoleController(RESTController):

@expose(template_engine='json') def _get(self): …

@expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _put(self, name, **kwargs): …

@expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)]) def _delete(self): …  

PUT  looks  just  like  a  POST  

class RoleController(RESTController):!

def __init__(self, name): self._role = CM.Role.query.get( account_id=c.account._id, name=name) if self._role is None: raise AttributeError, name!

@expose(template_engine='json') def _get(self): return self._role!

AttributeError      HTTP  404  Not  Found  

Auto-­‐JSONify  

class RoleController(RESTController):!

… @expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _put(self, name, **kwargs): assert name == self._role.name self._role.update(kwargs) return self._role!

@expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)]) def _delete(self): self._role.delete() return self._role

Update  model  from  kwargs  

  Don’t  trust  the  docs    Don’t  trust  the  docs  ▪  Don’t  trust  the  docs  

  Use  fat  models  

  Framework  support  for  REST  &  JSON  

  You’re  gonna  have  to  learn  some  Ruby  anyway  

  JSON  !=  BSON  

  Better  test  coverage  

  Search  support  (SOLR  /  ElasticSearch)  

 More  testing  with  real-­‐world  deployments  

  Finalize  integration  with  deployment  manager  

Rick  Copeland  @rick446  Arborian  Consulting,  LLC  

  http://openmymind.net/2011/10/28/CouchDB-­‐And-­‐MongoDB-­‐Performance/   MongoDB  is  14x  faster  

  http://www.snailinaturtleneck.com/blog/2009/06/29/couchdb-­‐vs-­‐mongodb-­‐benchmark/