Puppet DB Higher-order Puppet Deepak Giridharagopal Lead Engineer @ Puppet Labs [email protected] grim_radical, #puppet Monday, May 21, 12
Puppet DB: Higher-Order Puppet - Deepak Giridharagopal - PuppetCamp LA '12
Apr 16, 2017
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
PuppetDBHigher-order Puppet
Deepak GiridharagopalLead Engineer @ Puppet Labs
[email protected]_radical, #puppet
Monday, May 21, 12
Let’s talk aboutdata
Monday, May 21, 12
Monday, May 21, 12
Data!Puppet generates a lot of it, in many delicious flavors!
Persisted, ephemeral, machine local, centralized, meticulously structured, totally free-form, human readable, machine optimized...
Monday, May 21, 12
Catalogs“The Graph”
Containment edges, dependency edges, classes, tags, resources, resource parameters, metadata
Monday, May 21, 12
target: &id063 !ruby/object:Puppet::Resource catalog: *id001 exported: false file: /etc/puppetlabs/puppet/manifests/site.pp line: 44 parameters: !ruby/sym content: This is a test !ruby/sym backup: main reference: "File[/tmp/foo]" tags: - file - node - default - class title: /tmp/foo type: File
file {“/tmp/foo”: content => “This is a test”}
Monday, May 21, 12
Relationships
Exec[broker_cert_bundle]
File[/etc/puppetlabs/activemq/broker.pem]
Exec[broker_cert_pkcs12]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
Service[pe-activemq]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
Anchor[pe_compliance::end]
File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]
File[/etc/puppetlabs/activemq/broker.ts]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Class[Settings] Class[Main]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.ssh]
Schedule[daily]
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]
Anchor[pe_accounts::begin]
Class[Pe_accounts::Groups]
Anchor[pe_accounts::end]
Filebucket[main]
File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
File[/etc/puppetlabs/activemq/broker.ks]
Cron[pe-mcollective-metadata]
Class[Pe_mcollective]
Class[Pe_mcollective::Plugins]
Anchor[pe_mcollective::end]
File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]
File[/etc/puppetlabs/activemq/activemq.xml]
File[/etc/puppetlabs/mcollective/client.cfg]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/etc/puppetlabs/activemq/activemq-wrapper.conf]
Schedule[never] Stage[main]Anchor[pe_mcollective::begin]
Class[Pe_mcollective::Posix]
Class[Pe_mcollective::Metadata]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[broker_cert_keystore]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Schedule[weekly]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Class[Pe_accounts]
Exec[broker_cert_truststore]
Schedule[hourly]
Class[Pe_compliance::Agent]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Schedule[monthly] Filebucket[puppet]
Pe_accounts::User[peadmin]
File[/etc/puppetlabs/activemq/broker.p12]
Node[default]
Pe_accounts::User[puppet-dashboard]
Class[Pe_compliance]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Monday, May 21, 12
Relationships
Exec[broker_cert_bundle]
File[/etc/puppetlabs/activemq/broker.pem]
Exec[broker_cert_pkcs12]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
Service[pe-activemq]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
Anchor[pe_compliance::end]
File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]
File[/etc/puppetlabs/activemq/broker.ts]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Class[Settings] Class[Main]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.ssh]
Schedule[daily]
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]
Anchor[pe_accounts::begin]
Class[Pe_accounts::Groups]
Anchor[pe_accounts::end]
Filebucket[main]
File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
File[/etc/puppetlabs/activemq/broker.ks]
Cron[pe-mcollective-metadata]
Class[Pe_mcollective]
Class[Pe_mcollective::Plugins]
Anchor[pe_mcollective::end]
File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]
File[/etc/puppetlabs/activemq/activemq.xml]
File[/etc/puppetlabs/mcollective/client.cfg]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/etc/puppetlabs/activemq/activemq-wrapper.conf]
Schedule[never] Stage[main]Anchor[pe_mcollective::begin]
Class[Pe_mcollective::Posix]
Class[Pe_mcollective::Metadata]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[broker_cert_keystore]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Schedule[weekly]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Class[Pe_accounts]
Exec[broker_cert_truststore]
Schedule[hourly]
Class[Pe_compliance::Agent]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Schedule[monthly] Filebucket[puppet]
Pe_accounts::User[peadmin]
File[/etc/puppetlabs/activemq/broker.p12]
Node[default]
Pe_accounts::User[puppet-dashboard]
Class[Pe_compliance]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Monday, May 21, 12
Relationships
Exec[broker_cert_bundle]
File[/etc/puppetlabs/activemq/broker.pem]
Exec[broker_cert_pkcs12]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
Service[pe-activemq]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
Anchor[pe_compliance::end]
File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]
File[/etc/puppetlabs/activemq/broker.ts]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Class[Settings] Class[Main]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.ssh]
Schedule[daily]
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]
Anchor[pe_accounts::begin]
Class[Pe_accounts::Groups]
Anchor[pe_accounts::end]
Filebucket[main]
File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
File[/etc/puppetlabs/activemq/broker.ks]
Cron[pe-mcollective-metadata]
Class[Pe_mcollective]
Class[Pe_mcollective::Plugins]
Anchor[pe_mcollective::end]
File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]
File[/etc/puppetlabs/activemq/activemq.xml]
File[/etc/puppetlabs/mcollective/client.cfg]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/etc/puppetlabs/activemq/activemq-wrapper.conf]
Schedule[never] Stage[main]Anchor[pe_mcollective::begin]
Class[Pe_mcollective::Posix]
Class[Pe_mcollective::Metadata]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[broker_cert_keystore]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Schedule[weekly]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Class[Pe_accounts]
Exec[broker_cert_truststore]
Schedule[hourly]
Class[Pe_compliance::Agent]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Schedule[monthly] Filebucket[puppet]
Pe_accounts::User[peadmin]
File[/etc/puppetlabs/activemq/broker.p12]
Node[default]
Pe_accounts::User[puppet-dashboard]
Class[Pe_compliance]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Monday, May 21, 12
FactsEverything detected by facter
Facts for hardware, software, networking, CPUs, memory, virtualization, manufacturer info, custom facts...
Coming soon: structured facts!
Monday, May 21, 12
netmask_lo: 255.0.0.0 augeasversion: 0.10.0 fqdn: pe-debian6.localdomain manufacturer: "VMware, Inc." processorcount: "1" productname: VMware Virtual Platform physicalprocessorcount: 1 facterversion: 1.6.7 boardproductname: 440BX Desktop Reference Platform kernelmajversion: "2.6" hardwareisa: unknown timezone: PDT puppetversion: 2.7.12 (Puppet Enterprise 2.5.1) lsbdistcodename: squeeze is_virtual: "true" operatingsystemrelease: 6.0.2 virtual: vmware type: Other domain: localdomain hostname: pe-debian6 selinux: "false" kernel: Linux
kernelrelease: 2.6.32-5-686 ipaddress: 172.16.245.128 processor0: Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz lsbdistrelease: 6.0.2 uniqueid: 007f0101 hardwaremodel: i686 kernelversion: 2.6.32 operatingsystem: Debian architecture: i386 lsbdistdescription: Debian GNU/Linux 6.0.2 (squeeze) lsbmajdistrelease: "6" interfaces: "eth0,lo" ipaddress_lo: 127.0.0.1 uptime_days: 0 lsbdistid: Debian rubysitedir: /opt/puppet/lib/site_ruby/1.8 rubyversion: 1.8.7 osfamily: Debian memorytotal: &id001 502.57 MB memorysize: *id001 boardmanufacturer: Intel CorporationMonday, May 21, 12
ReportsCatalogs say what you want, reports say what you got.
Desired state, actual state, events, duration, timestamps...
Monday, May 21, 12
"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status change_count: 1 changed: true evaluation_time: 0.001869 events: - !ruby/object:Puppet::Transaction::Event audited: false desired_value: !ruby/sym file historical_value: message: *id006 name: !ruby/sym file_created previous_value: !ruby/sym absent property: ensure status: success time: 2011-10-25 18:51:37.143970 -07:00 failed: false file: *id007 line: 44 out_of_sync: true out_of_sync_count: 1 resource: "File[/tmp/foo]" resource_type: File skipped: false tags: - file - node - default - class time: 2011-10-25 18:51:37.143396 -07:00 title: /tmp/foo
Monday, May 21, 12
Why bother?
Monday, May 21, 12
“There's a war out there, old friend. A world war. And it's not about who's got the most bullets. It's about who controls the information. What we see and hear, how we work, what we think... it's all about the information!”
-- Sneakers Monday, May 21, 12
StoreconfigsCentralized storage of the configuration of all your nodes.
All resources, all parameters, all classes, all tags, all stages...
Enables use of exported resources
Monday, May 21, 12
class exporter { @@file { "/var/lib/puppet/nodes/$fqdn": content => "$ipaddress\n", tag => "ip" }}
node "export1.daysofwonder.com" { include exporter}
node "export2.daysofwonder.com" { include exporter}
node "collector.daysofwonder.com" { File <<| tag == "ip" |>>}
http://www.masterzen.fr/2009/03/08/all-about-puppet-storeconfigs/
Monday, May 21, 12
public key distributionmonitoring checksclustered services
master/slave replicationload balancers
shared filesystemsfirewall rules
...Monday, May 21, 12
QueryInterrogation, investigation, correlation
Use Puppet-generated data in scripts or for integration with other tools
Monday, May 21, 12
Higher order
PuppetMonday, May 21, 12
VolumeEvery node, on every puppet run, generates data
We have customers generating over 750G of data a day. Even storing a small subset of that much information adds up...
Monday, May 21, 12
(demo)
Monday, May 21, 12
Slow = :(When data storage is slow, it makes baby Deepak cry!
Slows down catalog compilation,More quickly saturates a Puppetmaster,Thrashes disk,Bad news!
Monday, May 21, 12
APICurrent APIs are limited
Hard to get at the data, and performance concerns discourage use.
We need better ways of searching, filtering, and correlating data.
Monday, May 21, 12
ParadoxSeemingly contradictory goals
We want to store as much data as we can, and allow for better querying, but without slowing stuff down or reducing reliability.
Monday, May 21, 12
We needAn information clearinghouse
Something that evolves the Puppet Data Library. A scalable, safe place to store the information Puppet collects and generates.
This is a hard problem!
Monday, May 21, 12
PuppetDBDefinitely Better!
Monday, May 21, 12
Grayskull
Monday, May 21, 12
PuppetDB
Monday, May 21, 12
PuppetDB isFast storage of current catalogs and current facts,
100% compatible with storeconfigs and inventory service,
REST APIs for resource, fact, and node retrieval,
...and other things, even!
Monday, May 21, 12
science&
secret alien technology!
Monday, May 21, 12
Message Queue
"new catalog""new facts"
"delete node"
Storeconfigs, Catalogs, Facts
(SCF)Domain objects
Command HandlerParsing
Transformation
Validation
Query handling
REST
Puppetmaster
Compiler
Storeconfigs
Puppet Enterprise Console
CLI & Other Tools
"inventory query""interactive query"
"new catalog""new facts"
"delete node"
Monday, May 21, 12
(export)
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
F C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
F
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
(collection)
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
F
?
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F?
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F?
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
?
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
?
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
?
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
F
C
C
Monday, May 21, 12
(failure)
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
FCatalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
F C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
F
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalog Resrc
C
C
Monday, May 21, 12
PuppetDB Server
HTTP MQ
DB Workers
DLO
Monday, May 21, 12
PuppetDB Server
HTTP MQ
DBWorkers DLO
Monday, May 21, 12
PuppetDB Server
HTTP MQ
DBWorkers DLOHTTPProxy(SSL)
Monday, May 21, 12
(launch)
Monday, May 21, 12
Reliable!We work very hard to persist everything we accept
Acknowledgements with UUIDS,Checksums,Queueing,Automatic retry and reconnect,and the Dead Letter Office if all else fails!
Monday, May 21, 12
APIs!We don’t cheat
Anything Puppet does with PuppetDB, you can do to
Query your own resources, upload new fact sets, create catalogs, inspect facts...all part of the Puppet Data Library
Monday, May 21, 12
#> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"
{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}
Monday, May 21, 12
curl-H "Accept: application/json"
"http://puppetdb/facts/host.my.net"
Monday, May 21, 12
curl-H "Accept: application/json"
"http://puppetdb/resources?query=..."
Monday, May 21, 12
Transparent!We care about operational visibility
Ships with a real-time dashboard,Dozens of metrics and gauges,Correlate-able logs,Easy to integrate with monitoring systems
Monday, May 21, 12
Speedy!PuppetDB is much, *much* faster than the previous storeconfigs and inventory services
At Puppet Labs, we’ve seen huge reductions in compile times, resource collection times, time to persist catalogs and facts, etc.
Monday, May 21, 12
Design decisions
Monday, May 21, 12
Posit:Hosts are not
entirely unique snowflakes
Monday, May 21, 12
Therefore:A resource often
exists across multiple hosts
Monday, May 21, 12
Feature:Single-instance
resource storage
Monday, May 21, 12
Resource dedupeCompute unique hashes for resources
We quickly hash all the resources in a catalog, and use bulk operations to compare them to hashes stored.
Monday, May 21, 12
Resource dedupeSignificant speed improvement!
Internal to Puppet Labs, we see ~83% resource duplication; this number is consistent with what we’ve seen in most customer environments.
Monday, May 21, 12
Posit:Puppet runs
frequently, but catalogs change
infrequentlyMonday, May 21, 12
Therefore:We’ll often receive
the same catalog for a host
Monday, May 21, 12
Feature:Single-instance catalog storage
Monday, May 21, 12
Catalog dedupeCompute unique hashes for catalogs
We use a Merkle Tree approach (hash tree) for quick comparisons.
Puppet Labs sees ~88% catalog duplication
Big savings!
Monday, May 21, 12
Posit:You have more than
one core, though storeconfigs is
single-threadedMonday, May 21, 12
Therefore:Throughput is not
maximized
Monday, May 21, 12
Feature:Massively parallel
operation
Monday, May 21, 12
ParallelWe can pat our heads and rub our tummies at the same time
Database operations don’t block MQ operations don’t block HTTP operations don’t block hash computation operations don’t block metric calculations don’t block...
Dozens of threads, zero locks
Monday, May 21, 12
Monday, May 21, 12
science&
secret alien technology!
Monday, May 21, 12
The Future
Monday, May 21, 12
http://github.com/puppetlabs/
puppetdb
Monday, May 21, 12
http://docs.puppetlabs.com/
puppetdb
Monday, May 21, 12
Use it, and tell us about it!
Monday, May 21, 12
PuppetDBThanks for your time!
Deepak GiridharagopalLead Engineer @ Puppet Labs
[email protected]_radical, #puppet
Monday, May 21, 12
Related Documents
