PDX DevOps: Intro to Logstash

Logstash

MeSysadmin at NetXposure Inc.

“DevOps” because I sit next to the Java engineers (there wasn’t any

desk space next to the other sysadmin)

First grownup job

BS in Microbiology

https://github.com/nickchappell

[email protected]

Splunk

500MB/day limit with demo version

Whaddya do?

tail -f /var/log/**/* | grep somestuff ?

LogstashLog aggregator/router/processor

http://logstash.net/

Elasticsearch +

Logstash +

Kibana

The ELK stack

Logstash internalsWritten in Ruby, but uses JRuby for a runtime

(so it will actually use more than 1 core)

•Logs enter through inputs!•Each incoming log line gets represented internally as an event

with fields!•Filters process events (add, remove or transform fields and their

contents) •Outputs send events elsewhere

Codecs are like filters but can be used inline in inputs/outputs (decode an incoming JSON log line and turn the objects/hashes/arrays into individual event fields)

(show example logstash.conf in repo)

Getting logs into Logstash

rsyslog: http://www.rsyslog.com/ Logstash forwarder: https://github.com/elasticsearch/logstash-forwarder

(show rsyslog config and Puppet code)

...and configure your Logstash server with inputs

(show inpiut section of example logstash.conf)

• TCP/UDP ports for plain syslog • log4J/JMX inputs to take logs directly out of your JVM app • collectd/ganglia • SNMP traps

Set up your machines/devices to send logs....

Processing logs

(show example rsyslog config in VM)

Done with filters

Grok can do lots of heavy lifting!

(show example logstash.conf in repo)

Basic idea: use filters (grok, mutate, etc.) to pull bits of data out of the raw message and put

different pieces into the event's separate fields

You can add, rename and delete an event's fields

http://grokdebug.herokuapp.com/

Exporting logs from Logstash

Tons of different outputs:

One output is Elasticsearch...

•Email •Plain text files on the local machine Logstash is installed on •Nagios/Icinga •Graphite •fluentd/flume/riemann event stream processors •SaaS monitoring/metrics services like Librato, Boundary, PagerDuty

Elasticsearch

http://www.elasticsearch.org/overview/elasticsearch/

ElasticsearchWritten in Java, based on Apache Lucene

Main competitor for Lucene-based search engines is Solr

Can be used as a JSON-based document database

Has HTTP APIs for sending/receiving data

Has several client libraries for sending/receiving data of different languages (Ruby, Python, Java, PHP, etc.)

ElasticSearch

(show ElasticSearch kopf plugin page)

Really easy to cluster (multicast networking FTW)!

Individual documents are grouped into indexes

Indexes are broken into shards

Shards are shared with other nodes in the cluster

Masters coordinate reads/writes and shard replication

(show ElasticSearch node status JSON page)(show example logstash.conf in repo)

Kibana

http://www.elasticsearch.org/overview/kibana/

KibanaHTML5+JS+CSS dashboard/search tool for ElasticSearch

Really easy to deploy and configure to talk to Elasticsearch!

(show Kibana dashboard in browser)

Browser talks directly to Elasticsearch

(show example Kibana config.js)

Rashid Khan's Monitorama PDX 2014 workshop: http://vimeo.com/95297576

https://github.com/elasticsearch/kibana/blob/master/sample/nginx.conf

https://github.com/elasticsearch/kibana/blob/master/sample/apache_ldap.conf

That's cool and all, but so what?

Centralized system logging for admins is one use case...

Centralized application logs for your devs is a lot more compelling!

Demo repository

What the repo has: •1 Puppet master VM •1 Kibana VM •1 Logstash VM •2 Elasticsearch VMs •2 VMs running rsyslog to generate log dataWhat you'll need: •Vagrant •VirtualBox •An Ubuntu 13.10 base box

https://github.com/nickchappell/logstash_pdxdevops

Further instructions in README

Community#logstash on freenode

#elasticsearch on freenode

#kibana on freenode

https://github.com/elasticsearch/

https://github.com/elasticsearch/elasticsearch

https://github.com/elasticsearch/logstash

https://github.com/elasticsearch/kibana