Application Logging With Logstash

Application Logging With Logstash

Ben Waine

• Worked With PHP For 5 Years

• Software Engineer -Sainsbury’s

• Dabbles in devops

https://joind.in/talk/view/13369

System Logs

Application Log

Debug Information - Errors (connections, uncaught exceptions, resource exhaustion)

Narrative Information - Methods Calls, Event Triggers

Business Events - Purchases, Logins, Registrations, Unsubscribes

Keeping Track Of All This....ssh [email protected] -f /var/log/nginx/my-site.access.logtail -f /var/log/my.application.log

ssh [email protected] -f /var/log/mysql/mysql.log

ssh [email protected] -f /var/log/rabbitmq/nodename.log

The Elk Stack

Visualizing Log Data

PHP Logging Tools

1) Monolog2) Everything else....

Basic Logging Examples

1) Monolog: Loggers And Handlers2) Monolog: Tags & Formatters3) Logging business events

use Monolog\Logger;use Monolog\Handler\FingersCrossedHandler;use Monolog\Handler\StreamHandler;

$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;

$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level);

$appLog−>pushHandler($fcHandler);$appLog−>debug('LOGGING!');

EG1: Loggers And Handlers

// Set A Log Level$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;

// Create A Logger$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);

$fcHandler= new FingersCrossedHandler($strHandler, $level);

// Create Handlers

$appLog−>pushHandler($fcHandler);

$appLog−>debug('Start Logging!');$appLog−>emergency('Something Terrible Happened');

// Push The Handler And Start Logging

EG 2: Tagging Formatting

$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "application");

$strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler));

$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])

$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");

// Create A Logger$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "app");

// Create A Handler & Formatter

// Set Formatter Onto Handler$strHandler−>setFormatter($formatter);

$appLog−>pushHandler($strHandler));

//Push Handler Onto Logger

$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");

// Capture A Unique Id, Create A Tag Processor, Push

Log Levels2009 - RFC 5424 - Syslog Protocol

Code / Severity

0 Emergency: system is unusable1 Alert: action must be taken immediately2 Critical: critical conditions3 Error: error conditions4 Warning: warning conditions5 Notice: normal but significant condition6 Informational: informational messages7 Debug: debug-level messages

https://tools.ietf.org/html/rfc5424

Log Levels2013 - PSR03 - PHP Logging Interface Standard

Phrase / Severity

emergency Emergency: system is unusablealert Alert: action must be taken immediatelycritical Critical: critical conditionserror Error: error conditionswarning Warning: warning conditionsnotice Notice: normal but significant conditioninfo Informational: informational messagesdebug Debug: debug-level messages

http://www.php-fig.org/psr/psr-3/

http://imgs.xkcd.com/comics/standards.png

EG 3: Event Logginguse Monolog\Logger;use Symfony\Component\EventDispatcher\EventDispatcher;

$dispatcher = new EventDispatcher();

$dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); });

$dispatcher−>dispatch("business.registration.post");

Logstash Architecture

1. Logstash Shipper ships logs to logstash

2. Logstash processes them

3. Logstash Inserts Into Elastic Search

4. Kibana exposes a web interface to Elastic Search data

Logstash Architecture

https://joind.in/talk/view/13369

Why not rate the talk now BEFORE the demo?

Logstash Demo

https://github.com/LoveSoftware/application-logging-with-logstash

Logstash Config

Logstash Collecting{ "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] }

Logstash Processing

input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key"}

}

Input

Logstash ProcessingFilteringfilter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } }

Logstash ProcessingOutput

output { elasticsearch { host => localhost }}

Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }

https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns

http://grokdebug.herokuapp.com/

55.3.244.1 GET /index.html 15824 0.043

%{IP:client}%{WORD:method}%{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

Logging IdeasRelease MarkerError rates of various applications over timeLatency in various percentiles of each application tierHTTP Responses: 400 series responsesHTTP Responses: 500 series responsesAuto git blame production errorsAuth and Syslogs

Go Forth And Log....BUT

Remember log rotation

Beware running out of space

Beware file logging on NFS

Questions?

https://joind.in/talk/view/13369