PuppetConf. 2016: External Data in Puppet 4 – R.I. Pienaar

R.I.Pienaar

PuppetConf 2016

External Data in Puppet 4

R.I.Pienaar | [email protected] | http://devco.net | @ripienaar

Who am I?

• Puppet User since 0.22

• Really into external data

• Author facts.d, extlookup, Hiera, MCollective

• Consultant - DevOps, Automation, Architect, Development

• Open Source @ github.com/ripienaar

• Volcane on IRC

mailto:[email protected]


External DataIs Configuration For

Your Code



Hierarchical Data

class monitor ( Array[Monitor::Plugin] $plugins = [] ) { … }



Hierarchical Data

class{“monitor”: plugins => [ “cpu", “write_graphite”, “df”, … ] }



Automatic Param Lookup

include monitor

# hieradb/nodes/host1.yaml monitor::plugins: - my_app

class{“monitor”: plugins => [ “my_app”, “cpu”, “df”, … ] }

+

=



Hierarchical Data

host1 host2 host3 host4 host5 host6

prod stage dev

dc1 dc2

Site wide common data



Hierarchical Data


prod stage dev

dc1 dc2




Hierarchical Data


prod stage dev

dc1 dc2




Hierarchical Data


prod stage dev

dc1 dc2


$trusted[“certname”]

$environment

$facts[“dc”]




include monitor



+

=



Lookup• lookup() function replaces hiera*()

• Automatic Param Lookup more useful

• environment and module aware

• Great CLI

• YAML/JSON out of the box, more to come

• Data compatible with Hiera, cohabits with Hiera

• Pluggable



Lookup - CLI

$ puppet lookup monitor::plugins —merge unique —— - my_app - write_graphite - cpu



$ puppet lookup monitor::plugins —merge unique —explain Merge strategy unique … Data Provider "Nodes" Path “/etc/…/production/data/nodes/dev3.devco.net.yaml” Original path: "nodes/%{trusted.certname}" Found key: "monitor::plugins" value: [ "my_app" ] … Data Provider "common" Path “/etc/…/data/common.yaml” Original path: "common" Found key: "monitor::plugins" value: [ "cpu", "write_graphite" ] … Merged result: [ "my_app", "cpu", "write_graphite" ]



$ puppet lookup monitor::plugins —merge unique —explain Merge strategy unique … Data Provider "Nodes" Path “/etc/…/production/data/nodes/dev3.devco.net.yaml” Original path: "nodes/%{trusted.certname}" Found key: "monitor::plugins" value: [ "my_app" ] … Data Provider "common" Path “/etc/…/data/common.yaml” Original path: "common" Found key: "monitor::plugins" value: [ "cpu", "write_graphite" ] … Merged result: [ "my_app", "cpu", "write_graphite" ]

Every tier

Final result



Lookup - Config# environments/production/hiera.yaml --- version: 4 datadir: data hierarchy: - name: "Nodes" backend: yaml path: "nodes/%{trusted.certname}"

- name: "common" backend: yaml



Lookup - function

$plugins = lookup(“monitor::plugins”)



Lookup - function

$plugins = lookup(“monitor::plugins”, { “value_type” => Array[Monitor::Plugin], “default_value” => [“cpu”, …]

})



Lookup - function

function monitor::params() { $result = { “monitor::plugins” => [ “cpu”, … ] } }

$plugins = lookup(“monitor::plugins”, { “default_value_hash” => monitor::params()

})



Lookup - function

$servers = lookup(“monitor::plugins”, { “merge” => “first”

})

Default behaviour



Lookup - function

$servers = lookup(“monitor::plugins”, { “merge” => “unique”

})

Array Merge (hiera_array())



Lookup - function

$servers = lookup(“monitor::plugins”, { “merge” => “hash”

})

Shallow Hash Merge (hiera_hash())



Lookup - function

$servers = lookup(“monitor::plugins”, { “merge” => “deep”,

})

Deep Hash Merge



Lookup - function

# common.yaml users::local: rip: email: “[email protected]" comment: “R.I.Pienaar” sudoer: true shell: “/bin/zsh”



Lookup - function

# nodes/dev3.devco.net users::local: rip: email: “[email protected]”



Lookup - function

$ puppet lookup users::local —merge deep --- rip: email: [email protected] comment: R.I.Pienaar sudoer: true shell: "/bin/zsh"




include monitor



+

=



Lookup - function

$lookup_options = { “merge” => {

“strategy” => “unique” } }

$plugins = lookup(“monitor::plugins”, $lookup_options)



Lookup - function

$plugins = lookup(“monitor::plugins”)

# hieradata/common.yaml lookup_options: monitor::plugins: merge: strategy: unique



include monitor

# …/production/hieradata/common.yaml lookup_options: monitor::plugins: strategy: unique

monitor::plugins: - cpu

- …

class monitor ( Array[Monitor::Plugin] $plugins = [] ) {}

Lookup - function



Lookup - Data In Modules# modules/monitor/hiera.yaml --- version: 4 datadir: "data" hierarchy: - name: "common" backend: "yaml"

# modules/monitor/data/common.yaml lookup_options: monitor::plugins: strategy: unique



include monitor

# hieradata/common.yaml monitor::plugins: - cpu

- write_graphite

Lookup - Data In Modules# hieradata/nodes/node1.yaml monitor::plugins: - my_app



Tips and Tricks



lookup(“classes”, {“merge” => “unique”}).include

hiera_include()



hiera_include()# common.yaml lookup_options: classes: merge: strategy: “deep” knockout_prefix: “-“

classes: - nagios

# nodes/node1.yaml classes: - -nagios - sensu



lookup(“classes”).include

hiera_include()

# common.yaml lookup_options: classes: merge: strategy: “deep” knockout_prefix: “-“



params.pp# modules/ntp/hiera.yaml --- version: 4 datadir: data hierarchy: - name: "Operating System Family" backend: yaml path: "%{facts.os.family}"

- name: "common" backend: yaml



params.pp

# modules/ntp/data/AIX.yaml --- ntp::driftfile: '/etc/ntp.drift' ntp::package_name: - 'bos.net.tcp.client' ntp::restrict: - 'default nomodify notrap nopeer noquery' - '127.0.0.1' ntp::service_name: 'xntpd'



params.pp

# modules/ntp/data/Debian.yaml ntp::restrict: - '-4 default kod nomodify notrap nopeer noquery' - '-6 default kod nomodify notrap nopeer noquery' - '127.0.0.1' - '::1' ntp::service_name: 'ntp' ntp::servers: - '0.debian.pool.ntp.org' - '1.debian.pool.ntp.org'



Future

• Hiera deprecated

• No more site wide hiera, only environment and modules

• Pluggable via functions in modules - native or ruby, much much simpler than old backends

• lookup.yaml or lookup.conf



Questions?

twitter: @ripienaar

email: [email protected]

blog: www.devco.net

github: ripienaarfreenode: Volcane

slack.puppet.com: ripienaar

https://www.devco.net/


PuppetConf. 2016: External Data in Puppet 4 – R.I. Pienaar

Technology