Chef on MongoDB and Pyramid

Chef in Python and MongoDB

Rick Copeland @rick446Arborian Consulting, LLC

What is Chef?

Infrastructure as Code

Resources & Providers

Cookbooks, Recipes, Clients, and Nodes

Chef System Architecture

Chef Server (API)

RabbitMQRabbitMQRabbitMQ

CouchDB Solr

Solr Indexer

chef-clientknifeChef Server

(Web UI)

HTTP REST API

Chef Deployment (chef-client)

Build, register, and authenticate the node

Synchronize cookbooks

Build resource collection (run the recipes in order)

Configure node (“converge”)

Run notification handlers

Chef System Architecture

Chef Server (API)

RabbitMQRabbitMQRabbitMQ

CouchDB Solr

Solr Indexer

chef-clientknifeChef Server

(Web UI)

HTTP REST API

Ruby

RubyRub

y

So What’s the Problem?

Chef assumes a bootstrapped node exists

Chef doesn’t keep release notes

Code and infrastructure are versioned differently

Solution: Web app to manage deployments & generate release notes

Chef System Architecture (Revised)

MongoDB

Solr

Solr Indexer

chef-clientknife

HTTP REST API

MonQ

Chef Server (API + web)

Ruby

Python

Why Change It?

Reduce # of processes & technologies

Don’t know Ruby well

Keep private keys out of the system

Integrate with existing authentication

Performance

API Endpoints: Short and Sweet

/clients

/data

/roles

/nodes

/cookbooks

/environment

s

/sandboxes

/search

Data Models

Mostly JSON almost BSON References to Ruby files stored

separately{ "name": "allura-0.0.1”,… "json_class": "Chef::CookbookVersion", "attributes": [ { "name": "default.rb", "url": "https://s3.amazonaws.com/opscode-platform-production-data/… "checksum": "749a3a328d6c47a32d134b336183981f", "path": "attributes/default.rb", "specificity": "default”…

Ruby files stored on S3

Ming Models (Schema)

role = collection( 'chef.role', doc_session, Field('_id', S.ObjectId()), Field('account_id', S.ObjectId(if_missing=None)), Field('name', str), Field('description', str), Field('default_attributes', str), Field('override_attributes', str), Field('run_list', [ str ] ), Field('env_run_lists', { str: [ str ]}), Index('account_id', 'name', unique=True))  

MongoDB Validator

Shorthand with Python Types

Embedded Documents

Index Definitions

Ming Models (Classes)

class Role(ModelBase):

def __name__(self): return self.name

def __json__(self): return dict( chef_type='role', json_class='Chef::Role', … default_attributes=loads(self.default_attributes), …)

def update(self, d): self.name = d['name'] … self.default_attributes = dumps(d['default_attributes']) self.override_attributes = dumps(d['override_attributes']) …

Models know where they live

Models can be turned into dict

(to be JSONified)

Models can be updated from

dict

Declarative JSON Validation

class RoleSchema(JSONModelSchema):

model_class=CM.role

chef_type='role’

json_class='Chef::Role’

exclude_fields=['_id', 'account_id']

RESTful URL Traversal

Find resource /foo/bar/baz Root()[‘foo’][‘bar’][‘baz’]

(Resource, method) view @view_config(context=Resource,

request_method=‘GET’, …)

Authorization @view_config(…, permission=‘read’, …)

Rendering @view_config(…, renderer=‘json’)

RESTful URL Traversal

class ResourceCollection(object): __name__ = None __model__ = None key_property='name’

def allow_access(self, client, permission): return permission == 'read' or client.admin

def __getitem__(self, name): obj = self.account.get_object( self.__model__, **{ self.key_property: name }) if obj is None: raise exc.HTTPNotFound() obj.__parent__ = self return obj

Check auth

Traverse to find sub-objects

RESTful URL Views

@view_config(context=Roles, renderer='json', request_method='GET’, permission='read')def list_roles(context, request): return dict( (n.name, request.resource_url(n)) for n in context.find())

@view_config(context=Roles, renderer='json', request_method='POST’, permission='create')def create_role(context, request): n = context.new_object() value = V.RoleSchema().to_python( request.json, None) n.update(value) …

Resources can be located

Convert and validate JSON

input

Lessons Learned

Don’t trust the docs Don’t trust the docs▪ Don’t trust the docs

Use fat models

Framework support for REST & JSON

You’re gonna have to learn some Ruby anyway

JSON != BSON

Next Steps

Port from homegrown framework to Pyramid

Better test coverage

Search support (SOLR / ElasticSearch)

More testing with real-world deployments

Finalize integration with deployment manager

Questions?

Rick Copeland @rick446Arborian Consulting, LLC

github.com/rick446/MongoPyChef

Rick Copeland @rick446Arborian Consulting, LLC