Monitoring using Sensu

R.I.Pienaar

Malta DevOps December 2016

Monitoring using Sensu

R.I.Pienaar | [email protected] | http://devco.net | @ripienaar

Who am I?• Malta since December 2015

• Consultant for 20+ years

• Government, Finance, Health, Social Media, Fortune 50, Startups

• DevOps, Automation, Architect, Development

• Open Source @ github.com/ripienaar

• Linux since Kernel 99 alpha p11

mailto:[email protected]




Toolvs

Framework



Checks

API

MetricsDynamic

External InputsSecure

Events

Plugins

JSON



Architecture - Overview



Architecture - Schedule



Architecture - Event



Architecture - Integrate



{ "checks": { "procs_rhel7": { "command": "/usr/lib64/nagios/plugins/check_procs -w 50 -c 70 -k", "interval": 300, "subscribers": [ "RedHat-7" ] } } }

Config - Checks




Config - Checks

Name

Command to run

How often

Where



{ "client": { "name": "dev3.devco.net", "address": "139.162.163.118", "subscriptions": [ "all", "linode_fra", "de", "RedHat-7" ], "redact": [

], "socket": { "bind": "127.0.0.1", "port": 3030 }, "procs": { "total": { "warn": 80, "crit": 90 } } } }

Config - Clients

Local event sync

What am I?

Node Parameters



{ "client": { …, "procs": { "total": { "warn": 80, "crit": 90 } } } }

Config - Clients

Node Parameters




{ "client": { …, "procs": { "total": { "warn": 80, "crit": 90 } } } }

Config - Clients

Node Parameters

{ "checks": { "procs_rhel7": { "command": “…/check_procs -w :::procs.total.warn|50::: -c :::procs.total.crit|70::: -k", "interval": 300, "subscribers": [ "RedHat-7" ] } } }

Check Tokens



require ‘sensu-plugin/check/cli'

class CheckFile < Sensu::Plugin::Check::CLI option :file, long: “—file FILE”, description: “The file to check”, default: nil

def run unknown(“no file specified, please use —file”) unless config[:file]

if File.exist?(config[:file]) ok(“file %s exist” % [config[:file]]) else critical(“file %s does not exist” % [config[:file]]) end end end

Checks - Writing in Ruby



require ‘sensu-plugin/check/cli'







require “sensu-plugin/check/cli”







# yum install nagios-plugins-all # /usr/lib64/nagios/plugin/check_file_age —help … Usage: check_file_age [-w <secs>] [-c <secs>] [-W <size>] [-C <size>] [-i] -f <file> check_file_age [-h | --help] check_file_age [-V | --version]

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE UNKNOWN: No file specified 3

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE CRITICAL: File not found - /keepalive 2

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE WARNING: /keepalive is 353 seconds old and 0 bytes 1

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE OK: keepalive is 4 seconds old and 0 bytes 0

Checks - Reuse



# yum install nagios-plugins-all # /usr/lib64/nagios/plugin/check_file_age —help … Usage: check_file_age [-w <secs>] [-c <secs>] [-W <size>] [-C <size>] [-i] -f <file> check_file_age [-h | --help] check_file_age [-V | --version]

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE UNKNOWN: No file specified 3

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE CRITICAL: File not found - /keepalive 2

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE WARNING: /keepalive is 353 seconds old and 0 bytes 1

# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE OK: keepalive is 4 seconds old and 0 bytes 0

Checks - Reuse



Checks - Reuse

http://sensu-plugins.io/



$ cat <<EOF > /dev/tcp/localhost/3030 { "name":"my_awesome_check", "output":"everything is fine", "ttl":600, “status”:0 } EOF

Checks - Your own App



{ "mailer": { "mail_from": "[email protected]", "mail_to": "[email protected]", "smtp_address": "smtp.example.org", "smtp_port": "25", "smtp_domain": "example.org", "template": “/etc/sensu/templates/mail.erb”, "subscriptions": { "subscription_name": { "mail_to": "[email protected]" } } } }

Handlers - Email

https://github.com/sensu-plugins/sensu-plugins-mailer

/etc/sensu/handlers/mailer.json

Custom mail body



<%= output %> Admin GUI: <%= admin_gui %> Host: <%= @event['client']['name'] %> Timestamp: <%= Time.at(@event['check']['issued']) %> Address: <%= @event['client']['address'] %> Check Name: <%= @event['check']['name'] %> Command: <%= command %> Status: <%= status_to_string %> Occurrences: <%= @event['occurrences'] %>

Handlers - Email

https://github.com/sensu-plugins/sensu-plugins-mailer

/etc/sensu/templates/mail.erb



Handlers - Thirdparties



events = JSON.parse( RestClient.get(“http://example:4567/events").body )

statusmap = {0 => "OK", 1 => "WARNING", 2 => "CRITICAL", 3 => "UNKNOWN"}

puts "Found %d events" % [events.size]

events.each do |event| puts "%30s: %-10s %s" % [ event["client"]["name"], statusmap[event["check"]["status"]], event[“check"]["output"] ] end

API - JSON

Found 2 events puppet1.example.net: CRITICAL DISK CRITICAL - /run/docker/netns/default is not accessible: Permission denied dev1.example.net: WARNING PROCS WARNING: 183 processes



API - Uchiwa

https://uchiwa.io/



API - Uchiwa

https://uchiwa.io/



Linkshttps://sensuapp.org/

https://uchiwa.io/https://graphiteapp.org/

http://grafana.org/http://sensu-plugins.io/



Questions?

twitter: @ripienaaremail: [email protected]: www.devco.net

github: ripienaarfreenode: Volcane

slack.puppet.com: ripienaar

https://www.devco.net/


Monitoring using Sensu

Internet