Solving real world data problems with Jerakia

Solving real world data problems with JerakiaCraig Dunn, Config Management Camp, Ghent 2016

• Best practice

• Code base design

• Workflow mangement

• Scaling Puppet

• Installation and support

• Module writing

• Throughout Europe

www.enviatics.com

• Puppet user since 2008

• IT consultant for 15+ years

• Active community member

• The “Roles and Profiles” guy

• Problem solver

• Lives in Málaga, Spain.

• …. and hotels

• Daddy!

www.craigdunn.org

Craig Dunn

@crayfishx

A brief history of Puppet

In the beginning…

• Over complex code

• Unsharable modules

• Making simple changes required alot of skill.

The embedded data era

class ntp { if $env == ‘dev’ { $server = ‘dev.ntp.local’ } else { if $hostname == ‘gateway’ { $server = ‘pool.ntp.org’ } else { $server = ‘prod.ntp.local’ } } …}

And then…

HieraThe dawn of the data separation era

• Separation of data from code

• Module authors could write sharable re-usable code

• Code was less complex and more readable

• The Forge became useful

• Managing data became a lot easier

Hierarchical Search

Pluggable• Pluggable interchangable backends

• Data can be sourced from multiple formats

• hiera-eyaml

• hiera-mysql

• hiera-http

• hiera-redis

• hiera-consul

Managing our data is now a critical part of configuration management

Infrastructure grows and requirementsget more complex

• Different teams and customers require different hierarchies

• A particular application needs to source data from a different place

• Control access to sub-sets of data for teams within an organisation

• Dynamically generate the lookup hierarchy at runtime

• Group together application specific data into separate files

• Manage encrypted data from any data source

• Global hiera.yaml file creates restrictions

Introducing Jerakiajerakia.io

Jerakia• Data lookup tool

• Open source

• Extendable framework

• Solving the most complex edge cases

Jerakia• Can be used as a Hiera backend

• Can be wired directly into Puppet as a data binding terminus

• Drop in replacement for Hiera, or not.

Why Jerakia?

One design goal…

Flexibility

• Lookup behaviour written in Ruby DSL

• Almost everything is pluggable

• Inter-changable data sources

• Easy integration

• Hiera compatible*

$ gem install jerakia

$ puppet module install crayfishx/jerakia

• A request is received containing a key and a namespace

• A policy is chosen to perform the request

• One or more lookups are called to act on the request

• A response is sent back to the requestor

• Container for lookups

• Written in Ruby DSL

• Different policies for different apps

Policy File

An Example Jerakia Policy File

policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend

An Example Jerakia Policy File

policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend

An Example Jerakia Policy File

policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend

An Example Jerakia Policy File

policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend

• Lookups are contained within policies

• A policy can contain multiple lookups

• A lookup always contains at least a data source

Lookups

Scope

Handler

Request

Lookup

Plugins

Data Source

Output Filter

Response Data

Anatomy of a Jerakia lookup

Scope

Handler

Request

Lookup

Plugins

Data Source

Output Filter

Response Data

Anatomy of a Jerakia lookupRequest consists of a

lookup key, a namespace and some metadata

Scope

Handler

Request

Lookup

Plugins

Data Source

Output Filter

Response Data

Anatomy of a Jerakia lookupInformation to be

used in determining how data is looked up

Scope

Handler

Request

Lookup

Plugins

Data Source

Output Filter

Response Data

Anatomy of a Jerakia lookup

Lookup plugins can read and modify the scope and

request objects

Scope

Handler

Request

Lookup

Plugins

Data Source

Output Filter

Response Data

Anatomy of a Jerakia lookup

A pluggable data source is used to lookup data

Scope

Handler

Request

Lookup

Plugins

Data Source

Output Filter

Response Data

Anatomy of a Jerakia lookup

Data returned from the datasource is passed to a

pluggable output filter

Lookup methods

confine / exclude

Invalidates a lookup unless/if the criteria is met

confine request.namespsace[0], "apache"

confine request.namespsace[0], [ /website_.*/, "apache", "php" ]

Stop

Do not proceed to the next lookup if this lookup is valid

lookup :special do … confine request.namespsace[0], "apache" stopend

lookup :main do …

Datasources• Easily pluggable and extendable

• File and HTTP datasources shipped out-of-the-box

Datasources datasource :name, { :option => “value”… }

Datasource definitionlookup :main do

datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] }

end

/var/lib/jerakia/env/dev/apache.yaml

lookup :main do

datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] }

end

/var/lib/jerakia/env/dev/apache.yaml

Datasource definition

lookup :main do

datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] }

end

/var/lib/jerakia/env/dev/apache.yaml

Datasource definition

/var/lib/jerakia/env/dev/apache.yaml/var/lib/jerakia/env/dev/apache.d/www_corp_com.yaml/var/lib/jerakia/env/dev/apache.d/www_acme_net.yaml/var/lib/jerakia/env/dev/apache.d/www_fake_org.yaml

Fragments• Introduced in 0.4

• If a .d directory is found, files within are concatenated

• One document is returned

Data Layout :searchpath => [

"host/#{scope[:certname]}", "env/#{scope[:environment]}", ]

# cat /var/lib/jerakia/env/dev/apache.yaml—-port: 80

# cat /var/lib/jerakia/env/dev.yaml—-apache::port: 80

Hiera

Jerakia

Plugins• Access to request and scope

• Can read or modify on-the-fly

• Re-usable

• Cleaner code in policy files

class Jerakia::Lookup::Plugin module Mything def do_something … end endend

Writing plugins• Written as Ruby extensions

• Can be placed in the plugin dir

• Or shipped as rubygems

lookup :main, :use => :mything do plugin.mything.do_something …end

Using plugins• Plugins are loaded into the lookup

• Referenced as plugin.name.method

lookup :main, :use => [ :mything, :foo ] do …end

lookup :main, :use => :hiera do plugin.hiera.rewrite_lookup datasource :file, { :docroot => "/var/lib/jerakia", :format => :yaml, :searchpath => [ "env/#{scope[:environment]}", "common", ]end

The hiera plugin• Provides compatibility to hiera filesystem layouts

• Shipped with Jerakia

# cat /var/lib/jerakia/env/dev.yaml—-apache::port: 80

Output filters

• Pluggable

• Specified in the lookup

• Parses data returned from the datasource

Output filters

• Two are currently shipped

• Encryption (provided by eyaml*)

• Strsub

*https://github.com/TomPoulton/hiera-eyaml

Output filters

lookup :main do …

output_handler :encryptionend

Output filters

lookup :main do …

output_handler :encryptionend

Example User Story• Team in Ireland manage PHP/Apache

• Autonomous team that don’t manage infra

• Their optimal hierarchy is different from “ours”

• “We” need to service them from Puppet

• They must not modify infra services

• “We” also manage PHP/Apache for other clients

policy :default do lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end

Our main lookup is responsible for the entire

infrastructure

policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end

Lookup for the Ireland team added above the

main lookup with separate docroot and

searchpath

policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end

Only use this lookup if the requestor location is IE and the namespace is

apache or php

policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] stop end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end

If this lookup is valid then do not proceed to the

main lookup, even if data is not found.

Command line $ jerakia lookup port —namespace apache

$ jerakia help lookupUsage: jerakia lookup [KEY]

Options: c, [--config=CONFIG] # Configuration file p, [--policy=POLICY] # Lookup policy # Default: default n, [--namespace=NAMESPACE] # Lookup namespace t, [--type=TYPE] # Lookup type # Default: first s, [--scope=SCOPE] # Scope handler # Default: metadata [--scope-options=key:value] # Key/value pairs to be passed to the scope handler m, [--merge-type=MERGE_TYPE] # Merge type # Default: array l, [--log-level=LOG_LEVEL] # Log level v, [--verbose], [--no-verbose] # Print verbose information D, [--debug], [--no-debug] # Debug information to console, implies --log-level debug d, [--metadata=key:value] # Key/value pairs to be used as metadata for the lookup

Lookup [KEY] with Jerakia

Integration with Puppet—-:backends: - jerakia

[master] . . . data_binding_terminus = jerakia

Roadmap & Contributing

Upcoming in 0.5• Data Schemas

• Better REST client/server

• Deep merge behaviour

• Lookup plugin “load method”

Contributions wanted• Code maturity

• Caching

• Features

• Bugfixes

• Documentation

• #jerakia (freenode) Sponsored by

Jerakia 1.0

Thank youQuestions?

jerakia.io

@crayfishx