Data Discovery and Systems Diagnostics with Elasticsearch, Logstash and Kibana

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com

Data Discovery and Systems Diagnostics with the ELK stack

Rittman Mead - BI Forum 2015, BrightonRobin Moffatt, Principal Consultant Rittman Mead


T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

•Principal Consultant with Rittman Mead ‣OBIEE & ODI ‣SysAdmin ‣Performance

•Previously … •OBIEE/DW developer at large UK retailer •SQL Server DBA, Business Objects, DB2, COBOL….

•Oracle ACE

•Frequent blogger for Rittman Mead : http://ritt.md/rmoff •Twitter: @rmoff • IRC: rmoff / #obihackers / freenode



About Me





About Rittman Mead

•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 and 2014 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA ‣ODIEE ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca





ELK





•Elasticsearch - schema-free, document-orientated, distributed data store

•Logstash - centralised data processing

•Kibana - analytics and visualisation

ELK



Getting started is easy!

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)


Kibana



Kibana



Kibana





Data Discovery



Information Management and Big Data - A Reference Architecture





Logstash

Elasticsearch

Kibana

twitter





DEMO!





logstashElasticsearch

Kibana

csv





DEMO!





Elasticsearch



Elasticsearch

•The core component of the ELK stack

•Based on Apache Lucene (same as Cloudera’s Solr)

•Distributed for scalability & resilience

•Near-realtime document indexing



Elasticsearch Uses - Search and Analytics

•Search ‣Soundcloud ‣GitHub

•Analytics ‣The Guardian’s Ophan application

https://www.elastic.co/assets/bltd061cc55096a5780/case-study-the-guardian.pdf

A quarter of a billion events per day … typically the lag before something shows up on the dashboard is somewhere between three to five seconds…http://tnw.to/s3NV5

https://www.elastic.co/assets/bltd061cc55096a5780/case-study-the-guardian.pdf

http://tnw.to/s3NV5



Elasticsearch

•Stores data as JSON documents within an index

•An index is made up of shards

•Shards are distributed around a cluster automatically ‣Resilience and scale-out are simple



Elasticsearch Administration

https://github.com/lmenezes/elasticsearch-kopf

https://github.com/lmenezes/elasticsearch-kopf



Elasticsearch REST API

$ curl -XPOST 'http://es:9200/viz/characters/' -d '{"name":"finbarr saunders"}'



$ curl -XPOST 'http://es:9200/viz/characters/' -d '{"name":"roger mellie”, "notes":"the man on the tele"}'




$ curl -XGET 'http://localhost:9200/viz/_search?q=roger' […] "hits" : { "total" : 1, "max_score" : 0.11506981, "hits" : [ { "_index" : "viz", "_type" : "characters", "_id" : "AUyyNUrTI0Rm5Pb-t8_l", "_score" : 0.11506981, "_source":{"name":"roger mellie" ,"notes":"the man on the tele"}




$ curl -XDELETE 'http://localhost:9200/viz'






Logstash



Logstash

inpu

tfil

ter

outp

ut elasticsearch email

kafka

nagios

pagerduty stdout file

grok geoip mutate drop

kafkalog csv

tsv

json

syslog tcp log4j

stdin

twitter



Logstash

•Does Logstash support <foo> …. yes, probably! ‣Vast number of supported input (and output) formats

couchdb_changes drupal_dblog elasticsearch exec eventlog file ganglia gelf generator graphite github heartbeat heroku irc imap

jmx kafka log4j lumberjack meetup pipe puppet_facter relp rss rackspace rabbitmq redis snmptrap stdin sqlite

s3 sqs stomp syslog tcp twitter unix udp varnishlog wmi websocket xmpp zenoss zeromq

Outputsboundary circonus csv cloudwatch datadog datadog_metrics email elasticsearch exec file google_bigquery google_cloud_storage ganglia gelf graphtastic

graphite hipchat http irc influxdb juggernaut jira kafka lumberjack librato loggly mongodb metriccatcher nagios null

nagios_nsca opentsdb pagerduty pipe riemann redmine rackspace rabbitmq redis riak s3 sqs stomp statsd solr_http

sns syslog stdout tcp udp websocket xmpp zabbix zeromq

Inputs

http://www.elastic.co/guide/en/logstash/master/input-plugins.html http://www.elastic.co/guide/en/logstash/master/output-plugins.html

http://www.elastic.co/guide/en/logstash/master/input-plugins.html

http://www.elastic.co/guide/en/logstash/master/output-plugins.html



Logstash Filters

•Powerful data processing ‣Extract fields from input (grok) ‣Enrich data (geoip, dns) ‣Reformat (split, multiline, json, xml)

alter anonymize collate csv cidr clone cipher checksum date dns drop elasticsearch extractnumbers environment elapsed

fingerprint geoip grok i18n json json_encode kv mutate metrics multiline metaevent prune punct ruby range

syslog_pri sleep split throttle translate uuid urldecode useragent xml zeromq

Filters

http://www.elastic.co/guide/en/logstash/master/filter-plugins.html

http://www.elastic.co/guide/en/logstash/master/filter-plugins.html



Grok — Time to get your RegEx on!

Input data

Grok pattern

Key/Value output

http://grokdebug.herokuapp.com/

http://grokdebug.herokuapp.com/



Logstash in Action

filter { grok { match => [ "message", "\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{DATA:Component}\] \[%{WORD:Severity} (:%{NUMBER:LogLevelNum})?\]

input {file { path => ["nqserver.log" ] }}

output { elasticsearch { host => "localhost" }}



Logstash -> Elasticsearch


•Two-way connector between Hadoop and Elasticsearch •Read/Write with Elasticsearch from Hive, Pig, Spark, etc



Elasticsearch-Hadoop

https://www.elastic.co/products/hadoop

Hive

MongoDB HDFS

Elasticsearch

Tweets Website logs Blog post metadata

Datasift Flume CSV

mongo-hadoop

elasticsearch-hadoop

Kibana


Other Elasticsearch Input Methods

•JDBC ‣“River” or “Feeder” method ‣Pull data from any Oracle, mysql, etc with schema intact

•Native libraries for common languages: ‣Perl / Python / Ruby / PHP / Groovy / Scala / .NET / R / etc etc





Systems Diagnostics

with ELK




System Diagnostics



System Monitoring



System Monitoring



Performance Diagnostics



Summary

Summary



Summary

•Data Data Discovery With ELK







Data Discovery



Summary System Diagnostics

With ELK












email

[email protected]

web

http://ritt.md/rmoff

twitter

@rmoff

irc

rmoff @ #obihackers

Interested? Data Discovery

http://ritt.md/go-elk-1

System Diagnostics & Monitoring http://ritt.md/go-elk-2 http://ritt.md/go-elk-3

mailto:[email protected]

http://ritt.md/rmoff

https://twitter.com/rmoff/




Data Discovery and Systems Diagnostics with Elasticsearch, Logstash and Kibana

Data & Analytics