Chef in Python and MongoDB
Rick Copeland @rick446Arborian Consulting, LLC
What is Chef?
Infrastructure as Code
Resources & Providers
Cookbooks, Recipes, Clients, and Nodes
Chef System Architecture
Chef Server (API)
RabbitMQRabbitMQRabbitMQ
CouchDB Solr
Solr Indexer
chef-clientknifeChef Server
(Web UI)
HTTP REST API
Chef Deployment (chef-client)
Build, register, and authenticate the node
Synchronize cookbooks
Build resource collection (run the recipes in order)
Configure node (“converge”)
Run notification handlers
Chef System Architecture
Chef Server (API)
RabbitMQRabbitMQRabbitMQ
CouchDB Solr
Solr Indexer
chef-clientknifeChef Server
(Web UI)
HTTP REST API
Ruby
RubyRub
y
So What’s the Problem?
Chef assumes a bootstrapped node exists
Chef doesn’t keep release notes
Code and infrastructure are versioned differently
Solution: Web app to manage deployments & generate release notes
Chef System Architecture (Revised)
MongoDB
Solr
Solr Indexer
chef-clientknife
HTTP REST API
MonQ
Chef Server (API + web)
Ruby
Python
Why Change It?
Reduce # of processes & technologies
Don’t know Ruby well
Keep private keys out of the system
Integrate with existing authentication
Performance
API Endpoints: Short and Sweet
/clients
/data
/roles
/nodes
/cookbooks
/environment
s
/sandboxes
/search
Data Models
Mostly JSON almost BSON References to Ruby files stored
separately{ "name": "allura-0.0.1”,… "json_class": "Chef::CookbookVersion", "attributes": [ { "name": "default.rb", "url": "https://s3.amazonaws.com/opscode-platform-production-data/… "checksum": "749a3a328d6c47a32d134b336183981f", "path": "attributes/default.rb", "specificity": "default”…
Ruby files stored on S3
Ming Models (Schema)
role = collection( 'chef.role', doc_session, Field('_id', S.ObjectId()), Field('account_id', S.ObjectId(if_missing=None)), Field('name', str), Field('description', str), Field('default_attributes', str), Field('override_attributes', str), Field('run_list', [ str ] ), Field('env_run_lists', { str: [ str ]}), Index('account_id', 'name', unique=True))
MongoDB Validator
Shorthand with Python Types
Embedded Documents
Index Definitions
Ming Models (Classes)
class Role(ModelBase):
def __name__(self): return self.name
def __json__(self): return dict( chef_type='role', json_class='Chef::Role', … default_attributes=loads(self.default_attributes), …)
def update(self, d): self.name = d['name'] … self.default_attributes = dumps(d['default_attributes']) self.override_attributes = dumps(d['override_attributes']) …
Models know where they live
Models can be turned into dict
(to be JSONified)
Models can be updated from
dict
Declarative JSON Validation
class RoleSchema(JSONModelSchema):
model_class=CM.role
chef_type='role’
json_class='Chef::Role’
exclude_fields=['_id', 'account_id']
RESTful URL Traversal
Find resource /foo/bar/baz Root()[‘foo’][‘bar’][‘baz’]
(Resource, method) view @view_config(context=Resource,
request_method=‘GET’, …)
Authorization @view_config(…, permission=‘read’, …)
Rendering @view_config(…, renderer=‘json’)
RESTful URL Traversal
class ResourceCollection(object): __name__ = None __model__ = None key_property='name’
def allow_access(self, client, permission): return permission == 'read' or client.admin
def __getitem__(self, name): obj = self.account.get_object( self.__model__, **{ self.key_property: name }) if obj is None: raise exc.HTTPNotFound() obj.__parent__ = self return obj
Check auth
Traverse to find sub-objects
RESTful URL Views
@view_config(context=Roles, renderer='json', request_method='GET’, permission='read')def list_roles(context, request): return dict( (n.name, request.resource_url(n)) for n in context.find())
@view_config(context=Roles, renderer='json', request_method='POST’, permission='create')def create_role(context, request): n = context.new_object() value = V.RoleSchema().to_python( request.json, None) n.update(value) …
Resources can be located
Convert and validate JSON
input
Lessons Learned
Don’t trust the docs Don’t trust the docs▪ Don’t trust the docs
Use fat models
Framework support for REST & JSON
You’re gonna have to learn some Ruby anyway
JSON != BSON
Next Steps
Port from homegrown framework to Pyramid
Better test coverage
Search support (SOLR / ElasticSearch)
More testing with real-world deployments
Finalize integration with deployment manager
Questions?
Rick Copeland @rick446Arborian Consulting, LLC
github.com/rick446/MongoPyChef
Rick Copeland @rick446Arborian Consulting, LLC
