Listen to the sounds of your application

Listen to the sounds of your application

Listen to the sounds of your application

!2

Krzysztof Ciesielski

@kpciesielski

softwaremill.com

Lublin Software Craftsmen

!3

Maciej Biłas

@maciejb

softwaremill.com

jbison.com

!4

• Monitoring, huh?

• Introducing Graphite

• Log analysis – the whys

• Logstash architecture & use cases

• Exploring logs with Kibana

Agenda

!5

Monitoring, huh?

!6

source: codeascraft.com

!7

Types of measurements

!7

Types of measurements

Network

!7

Types of measurements

Network

Machine

!7

Types of measurements

Network

Machine

Application

!8

source: codeascraft.com

!9

measurement > prediction

!10

!11

measurement >> prediction

!12

Our stack

!12

Our stack

Server

App

Server

App

…

!12

Our stack

Server

App

Server

App

…

Graphite Logstash

!12

Our stack

Server

App

Server

App

…

Graphite Logstash

!12

Our stack

Server

App

Server

App

…

Graphite Logstash

!12

Our stack

Server

App

Server

App

…

Graphite Logstash

!12

Our stack

Server

App

Server

App

…

Graphite Logstash

!12

Our stack

Server

App

Server

App

…

Graphite Logstash

Yammer Metrics

!13

Introducing Graphite

!14

http://graphite/render?target=server.web1.load&height=800&width=600

!15

Graphite

!16

PORT=2003 SERVER=graphite.your.org echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT}; 

!17

successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297

!17

successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297

=> successful.login.attempt = 1

!18

Yammer Metrics

!18

Yammer Metrics

private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }

!19

Metrics

!19

MetricsTypes:

!19

MetricsTypes:

•Gauges

!19

MetricsTypes:

•Gauges

•Counters

!19

MetricsTypes:

•Gauges

•Counters

•Meters

!19

MetricsTypes:

•Gauges

•Counters

•Meters

•Histograms

!19

MetricsTypes:

•Gauges

•Counters

•Meters

•Histograms

•Timers

!19

MetricsTypes:

•Gauges

•Counters

•Meters

•Histograms

•Timers

•Health Checks

!19

MetricsTypes:

•Gauges

•Counters

•Meters

•Histograms

•Timers

•Health Checks

Reporters:

•STDOUT

•CSV

•SLF4J

•JMX

•Graphite

•Ganglia

!20

Counters, meters… vs.

Gauges

!21

Aggregation

!22

source: codeascraft.com

!23

You can also check out• collectd https://collectd.org

• StatsD https://github.com/etsy/statsd/

• Riemman http://riemann.io/

• Twitter’s Ostrich https://github.com/twitter/ostrich

• Ganglia http://ganglia.sourceforge.net/

• Dashboards: http://shopify.github.io/dashing/ https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk

!24

Event Sourcing

Tracing and storing all the events

Current app state: replaying the stream

Great for data mining and analysis

!25

Log aggregation

Less invasive way to build an “event stream”

Logs are data with plenty of value

!26

Log aggregation

Log source

Log source

Log source

Central storage

Query engine

Web view

!27

Logstash

Open Source (written in JRuby)

!27

Logstash

Open Source (written in JRuby)

Gathers logs from various inputs

!27

Logstash

Open Source (written in JRuby)

Gathers logs from various inputs

Parses and extracts metadata

!27

Logstash

Open Source (written in JRuby)

Gathers logs from various inputs

Writes to various outputs

Parses and extracts metadata

!27

Logstash

Similar tools: !Splunk, Graylog, Fluentd, log.io

Open Source (written in JRuby)

Gathers logs from various inputs

Writes to various outputs

Parses and extracts metadata

!28

Logstash centralized architecture

!29

Logstash grok

2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

!29

Logstash grok

2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

time 2013-11-10 19:41:25.321

thread main

loglevel INFO

source o.a.camel.impl.DefaultCamelContext

message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds

!30

Logstash simple use cases

Output: e-mail alert

ExceptionsSecurity issuesRare and business-meaningful events

Grok:

!31

Kibana

!32

Kibana checking general app activity

!33

Kibana Spotting anomalies

!34

Kibana Checking regular events

!35

Kibana How often is this feature used?

!36

Kibana Displaying multiple query results

Query: exception Type: applog

Query: index Type: mongolog

!37

References• Graphite http://graphite.readthedocs.org/en/latest/

• Yammer Metrics http://metrics.codahale.com/

• Logstash http://logstash.net/

• Surfing the event stream by Sam Newman at Geecon http://www.slideshare.net/spnewman/surfing-the-event-stream

• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack

• Code as Craft http://codeascraft.com/

!38

Reach us at: @maciejb and @kpciesielski

softwaremill.comThank You