Use Saltstack to deploy a full monitoring and supervision ...slides.logilab.fr/2018/cfgmgmtcamp_saltstack_monitoring.pdfSaltStack - huge toolkit to manage infrastructure Netdata -
Post on 17-May-2020
18 Views
Preview:
Transcript
Use Saltstack to deploy a fullmonitoring and supervisionstack#cfgmgmtcamp18Arthur Lutz ‐ Logilab
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
(part of) who am IArthur Lutz
Logilab
Python dev / Sysadmin / Debian / Devops
SaltStack Certi�ed Engineer
Paris Salt Meetup
Python Nantes Meetup / Nantes MonitoringMeetup
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Big picture ‐ scien�fic cloudapplica�on
Complex scaling parameters
salt-based auto-scaling, scale up and down basedon application logic
scaling is based on metrics collected by salt
automatically deploy application-orientedmonitoring
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
dashboards from diversedata
grafana provides data source diversity
we're looking for data origin diversity, and agileways to deploy them
system (internal) data
cloud (external) data
application-speci�c data
autoscaling state, etc...
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
agility by design
consistency by designeasily de�ne and add metrics
compare and test metrics from different points ofview
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Overview of componentsSaltStack - huge toolkit to manage infrastructure
Netdata - collect metrics, visualise "realtime"status of server
Sensu - devops oriented monitoring system
Graphite / Carbon - to collect and query timeseries data
Grafana - to build and display dashboards on thecollected data
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Components usedsalt pillars
salt scheduler
salt mine
salt returners
salt runners
salt reactor
salt-cloud
... "which ones did we not use?"
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Salt formulassalt-formula (what else?)
nginx-formula
nfs-formula
nagios-formula
sensu-formula
rabbitmq-formula
redis-formula
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Role based approach
Pillars (with pillar_merge_list: True ) :
base: '*': - role.sensu-client 'saltmaster.local': - role.sensu-server
States :
base: 'I@role:sensu-client': - sensu.client 'I@role:sensu-server': - sensu.server
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Salt minesend unique ID used by Netdata to stream clientsto central instance states :
{{ salt['mine.get']('role:netdata-master', 'network.interface_ip' expr_form='pillar').values()[0] }}
use to retrieve location of Rabbitmq server used by Sensu . pillars :
{% set rabbitmq = salt.saltutil.runner("mine.get", tgt='role:rabbitmq', fun="network.interface_ip", tgt_type='pillar').values() %}
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Salt returnersCarbon returner to push metrics from applicationand infrastructure
returner address is con�gured in pillars (cfextraction from mine)
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Salt custom modulesCustom modules are shipped by salt master from _modules folder
celery statistics (use app.inspect.stats )
core application metrics using HTTP API
sentry (error collector) metrics using the sentrypython binding (which uses the sentry HTTP API)
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Custom module ‐ celery inspectdef stats(): ''' Provide stats from celery tasks via app.inspect.stats() [snip] ''' ret = __salt__['redis.sentinel_get_master_ip']('name') r_url = 'redis://{}:{}/0' app = Celery(broker=r_url.format(ret['master_host'], ret['master_port'])) stats = app.control.inspect().stats() _clean(stats) return stats
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Salt‐cloudenables custom auto-scaling system
portable between cloud infrastructures
enables to run infrastructure on local openstack(or even devstack)
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Salt reactor ‐ event driven!react to new VM connecting
get info about VM (for example kernelversion )
upgrade kernel to linux-backports
reboot system
deploy monitoring
deploy application
Ended up being replaced by Packer generated AMI.
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Let's zoom in
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Visualising complexity
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
monitoring ‐ howlaunch a salt runner to pool the metrics
salt.execute runner is used to get infrastructureview of a given metric
cloud.profile runner is used to provision new VMs
cloud.destroy runner is used to scale down unusedVMs
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
monitoring ‐ howNetdata sends system metrics
central Netdata sends data to Graphite
Salt sends custom metrics to Graphite
Sensu executes checks (subscriber based model)
Sensu collects metrics and sends them to Graphite
Grafana uses the metrics stored in Graphite todisplay dashboards
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
MASTER
sensuserversensuapi
web
minion
storage
minion
client
com
pute
nod
e
minion
client
com
pute
nod
e
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Demo �me!install netdata
role based architecture
auto con�gure streaming by pushing the API id tothe mine
install graphite
deploy netdata graphite con�guration
install grafana
con�gure grafana datasource with salt
con�gure grafana dashboard with salt#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
Demo �me!install rabbitmq with rabbitmq-formula for sensu
install sensu with sensu-formula
use roles to con�gure subscribed sensu checks
con�gure uchiwa for web ui
con�gure sensu metrics to send data to graphite
update grafana dashboard with salt
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
What's next / Roadmapconvert schedulers to beacons
various optimisations
use reactor instead of polling
look into salt's thorium (complex reactor system)
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
EndThanks for your attention
Questions ?
Comments ?
Improvements ?
#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
sourceshttp://slides.logilab.fr/2018/cfgmgmtcamp_saltstack_monitoring.pdf
Demo : https://github.com/arthurlogilab/salt-vagrant-demo
Contact / Follow us : https://www.logilab.fr
@logilab
@arthurlutz
arthurlutz@social.logilab.org#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab
top related