Puppet Performance Profiling

R.I.Pienaar

NLUUG Fall Conference 2014

Puppet PerformanceProfiling

R.I.Pienaar | [email protected] | http://devco.net | @ripienaar

Who am I?

• Puppet user since 0.22.x

• Architect of MCollective

• Author of Extlookup and Hiera

• Independant Consultant

• Blog at http://devco.net

• Tweets at @ripienaar

• Volcane on IRC

mailto:[email protected]


Why?

• Unit and Integration testing

• Cloud based scaling

• Maintenance windows and fast feedback

• Resource usage on the master



Agent Life Cycle

• Agent gather facts using facter *

• Agent submit facts and request catalog

• Master compiles catalog, sends to agent *

• Agent stores catalog

• Agent applies catalog *

• Agent submits report

• Master processes report



Facter



Shows per fact timings

$ facter —timinglsbdistid: 56.06msoperatingsystem: 57.21msosfamily: 58.54msmacaddress: 0.29msipaddress6: 18.36msipaddress6_lo: 31.41msmtu_dummy0: 29.82msvlans: 0.20msselinux: 11.91msmtu_ip_vti0: 29.09ms

Timing Facts



Shows per fact timings…but only the setcode part

FREEGEOURL = “http://freegeoip.net/json/"

geoipinfo = JSON.parse(open(FREEGEOURL).read)

geoipinfo.each do |k, v| next if k == "ip" Facter.add("geo_#{k.downcase}") { setcode { v } }end

Timing Facts

geo_latitude: 0.02msgeo_region_name: 0.04msgeo_metro_code: 0.04msgeo_time_zone: 0.03msgeo_country_code: 0.02msgeo_country_name: 0.02msgeo_zip_code: 0.02msgeo_region_code: 0.04msgeo_city: 0.04msgeo_longitude: 0.03ms



Restructuring fact to do the slow work in setcode

geoipinfo = nil

["country_code", “country_name”…].each do |key| Facter.add(“geo_%s” % key.downcase) do setcode do geoipinfo ||= JSON.parse(open(FREEGEOURL).read)

["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key] end endend

Timing Facts

geo_zipcode: 7174.49msgeo_metro_code: 0.04msgeo_country_code: 0.03msgeo_region_name: 0.04msgeo_latitude: 0.05msgeo_area_code: 0.04msgeo_country_name: 0.03msgeo_city: 0.04msgeo_longitude: 0.04msgeo_region_code: 0.04ms

Fetch once per runand re-use



Add a local cache, now only the first invoke is slow

def geoip_cached_fetch store = PStore.new(File.join(Dir.tmpdir, “fgeoip_fact.pstore")) store.transaction do store[:freegeoip] ||= JSON.parse(open(FREEGEOURL).read) endend

geoipinfo = nil

["country_code", “country_name”…].each do |key| Facter.add(“geo_%s” % key.downcase) do setcode do geoipinfo ||= geoip_cached_fetch

["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key] end endend

Timing Facts



Catalog Compilation



class apache($version=present) { validate_string($version)

class{“::apache::package”: version => $version }}

Compilation



# /etc/puppet/modules/apache/init.ppclass apache($version=present) { validate_string($version)


Compilation

Needs to find the file



class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true}

Compilation

$ strace puppet parser validate test.pp 2>&1 |grep stat|wc -l10546

$ strace puppet parser validate test.pp 2>&1 |egrep '((stat|open).+modules)'|wc -l217





Compilation

hiera(“apache::version”, “present”)



:hierarchy: - 01_Nodes/%{::fqdn} - 02_Domain/%{::domain} - 03_Perimeter/%{::perimeter}/%{::datacenter} - 03_Perimeter/%{::perimeter} - 04_DC/%{::datacenter} - 05_OS/%{::operatingsystem}/%{::hardwareisa} - 05_OS/%{::operatingsystem} - 06_Manufacturer/%{::manufacturer} - 07_RealTime/%{::real_time} - 09_Mco_Teams/%{::team} - 50_Teams/%{team}/01_Nodes/%{::fqdn} - 50_Teams/HPCE/%{::hpce_env}/%{::hpce_type} - 50_Teams/HPCE/%{::hpce_env}/common - 50_Teams/%{::team}/common - common

Compilation




Compilation

$ puppet apply test.pp 2>&1|grep "Looking for data source"|wc -l3562





Compilation

ruby code, similar issues as the fact





Compilation

also speeds up finding the class internally





Compilation

No hiera lookup, authoritative value supplied




Profiling Compilation

$ puppet apply test.pp —profile



importing ‘..epel/manifests/init.pp’importing ‘..epel/manifests/params.pp’importing ‘..epel/manifests/rpm_gpg_key.pp’importing ‘..puppetdb/manifests/init.pp’importing ‘..puppetdb/manifests/params.pp’PROFILE [apply] 2.3.1 Called defined: took 0.0001 secondsPROFILE [apply] 2.3.2 Called downcase: took 0.0000 secondsPROFILE [apply] 2.3.3 Called validate_re: took 0.0000 secondsPROFILE [apply] 2.3.4 Called downcase: took 0.0000 secondsPROFILE [apply] 2.3.5 Called validate_re: took 0.0000 secondsPROFILE [apply] 2.3.6 Called downcase: took 0.0001 secondsPROFILE [apply] 2.3.7 Called validate_re: took 0.0000 secondsimporting ‘..puppetdb/manifests/server.pp’ in environment productionPROFILE [apply] 2.3.8 Called downcase: took 0.0000 secondsPROFILE [apply] 2.3.9 Called validate_re: took 0.0000 seconds

Profiling Compilationclass {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true}



Scope(Apache::Vhost[default]): Retrieving template apache/vhost.conf.erb

template[…vhost.conf.erb]: Bound template variables for …vhost.conf.erb in 0.00 seconds

template[…vhost.conf.erb]: Interpolated template …vhost.conf.erb in 0.05 seconds

Profiling Compilationdefine apache::vhost(…) { file { “${priority_real}-${filename}.conf": content => template(‘apache/vhost.conf.erb’) }}



AGGREGATE PROFILING RESULTS:compiler -> compile: 5.800916 ms (1 calls)compiler -> evaluate_resource: 1.53646 ms (79 calls)compiler -> evaluate_resource -> Apache::Vhost[…]: 0.148923 msfunctions: 1.892293 ms (562 calls)functions -> include: 1.091945 ms (41 calls)functions -> template: 0.754388 ms (137 calls)

Profiling Compilation



Catalog apply phase



# puppet agent —test —evaltraceClass[Postgresql::Server::Reload]: Starting to evaluateClass[Postgresql::Server::Reload]: Evaluated in 0.00 secondsClass[Postgresql::Server::Config]: Starting to evaluateClass[Postgresql::Server::Config]: Evaluated in 0.00 secondsClass[Postgresql::Server::Service]: Starting to evaluateClass[Postgresql::Server::Service]: Evaluated in 0.00 secondsApache::Vhost[puppet-localhost]: Starting to evaluateApache::Vhost[puppet-localhost]: Evaluated in 0.00 secondsPackage[mailcap]: Starting to evaluatePackage[mailcap]/ensure: createdPackage[mailcap]: Evaluated in 65.36 seconds

Profiling the Agent

Tracing the catalog apply phase



# report_print.rb —count 5 —report 201411101110.yamlReport for localhost at Wed Nov 10 11:10:54 +0000 2014

Report File: /var/…/last_run_report.yaml Report Kind: apply Puppet Version: 3.7.3 Report Format: 4 Configuration Version: 1416395447 UUID: d4a9fac9-…-7db61b0dfa89 Log Lines: 2 (show with —log)

Profiling the Agent

Processing reports for performance metrics



Report Metrics:

Changes: Total: 2 Events: Total: 2 Success: 2 Failure: 0 Resources: Total: 213 Out of sync: 2 Changed: 2 Scheduled: 0 Skipped: 0 Failed to restart: 0 Failed: 0 Restarted: 0

Profiling the Agent




Time: Total: 28.44 Package: 15.55 Config retrieval: 7.71 File: 1.43 Exec: 1.33 Puppetdb conn validator: 0.87 Service: 0.73 Postgresql psql: 0.66 Augeas: 0.07 Ini setting: 0.04

Profiling the Agent




Resources by resource type:

107 File 38 Ini_setting 13 Exec 12 Postgresql_psql 9 Package 6 Yumrepo 6 Anchor 6 Schedule 4 Service 3 Postgresql_conf

Profiling the Agent




Slowest 5 resources by evaluation time:

15.55 Package[mailcap] 0.88 Puppetdb_conn_validator[puppetdb_conn] 0.66 File[/usr…/validate_postgresql_connection.sh] 0.31 Exec[validate postgres connection] 0.30 Service[puppetmaster]

Profiling the Agent




5 largest managed files (only those that are readable)

3.85 KB /var/…/bin/concatfragments.sh 1.65 KB /etc/puppet/rack/config.ru 1.61 KB /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 1.23 KB /etc/puppet/puppet.conf 849.00 B /etc/httpd/conf/httpd.conf

Profiling the Agent




Questions?

twitter: @ripienaar

email: [email protected]

blog: www.devco.net

github: ripienaar

freenode: Volcane

geo fact: http://srt.ly/hxreport processor: http://srt.ly/gq


Puppet Performance Profiling

Software