Top Banner
Centralized log management based on Logstash and Kibana - case study Dariusz Eliasz 20.05.2014 Atmosphere Conference
31

Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Mar 07, 2018

Download

Documents

hahanh
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Centralized log management based on

Logstash and Kibana - case study

Dariusz Eliasz

20.05.2014 Atmosphere Conference

Page 2: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

• What’s the problem ?

• Solutons

• Transport format

• Architecture

– Sender

– Log router

– Log collector

– Full text search engine

– GUI

• Use case

Agenda

Page 3: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

What’s the problem ??

Page 4: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 5: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 6: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Solutons ??

Page 7: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 8: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 9: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 10: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 11: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Transport format - syslog

• RFC3164 (BSD syslog )

• limited size - 1kB

• format of a syslog message:

– PRIORITY (calculated from severity and facility)

– HEADER (tmestamp + hostname or IP)

– MSG (tag + content)

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

Page 12: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Transport format - json

• JavaScript Object Notaton

• lightweight text-data interchange format

• language independent

• self-describing

Page 13: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Transport format - json

{

"LogType": "access_log",

"Vhost": "atmosphere-conference.com",

"HtpsOn": "false",

"Xrealip": "1.2.3.5",

"Clientp": "91.17.13.28",

"UserAgent": "Mozilla/4.0 (compatble; MSIE 6.0; Windows NT 5.1)",

}

Page 14: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Architecture

Page 15: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Sender

Page 16: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Sender

• nxlog (htp://nxlog-ce.sourceforge.net/)

• multple input types:

– tcp socket

– udp socket

– fle input

– unx socket

• multple parser types:

– bsd syslog

– json

Page 17: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Sender

Good practce:

• make as much as possible processing on sender site, eg: apache access logs in

json format

• automate confguraton management

Page 18: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log router

Page 19: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log router

nxlog

nxlog logstash

logstash

redis

redissyslog-ng

syslog-ng redis

Page 20: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log router

### JSON PARSER

parser p_json {

json-parser (prefx("_json."));

};

### FILTERS

flter f_someflter {

("${_json.SomeJsonField}” == ”abc.com”)

};

### INPUTS & OUTPUTS

### LOG PATHS

log {

source(s_network_json);

destnaton(d_udp_logstash);

destnaton(d_tcp_hadoop);

#fags(fow-control); # disabled to separate destnatons

};

Page 21: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log router

Good practce:

• good separaton of destnatons

• calculate enough redis size – it’s yor bufer

• batch events writes to redis

Page 22: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log collector

Page 23: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log collector

• Logstash htp://logstash.net/

• collectng, parsing and storing logs tool

• plugins:

Inputs

• fle

• gelf

• tcp

• log4j

• redis

• varnishlog

Codecs

• json

• line

• msgpack

• netlow

• multline

Filters

• grok

• alter

• cidr

• geoip

• grep

• mutate

Outputs

• elastcsearch

• graphite

• jira

• tcp

• zeromq

• zabbix

Page 24: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Log collector

Good practce:

• keep up2date version of java & logstash

• use batch & multthread read from redis

• read logs

• bulk writes to elastcsearch

Page 25: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Full text search engine

• Elastcsearch htp://www.elastcsearch.org/

• distributed, real-tme search and analytcs engine

• store documents as a JSON

• high availability

• schema free

• index mult-tenancy

• on top of Lucene

Page 26: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Full text search engine

• every index is replicated

• every index sharded

• index parttoning – tme based

• data retenton – tme based

Page 27: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Full text search engine

Good practce:

• half memory for ES (<30GB), half for system cache

• bootstrap.mlockall: true

• gateway.recover_afer_nodes

• indices.felddata.cache.size

• authorizaton via proxy

• curator

• Marvel plugin

Page 28: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

GUI

• Kibana 3 htp://www.elastcsearch.org/overview/kibana/

• search, graph & analyze logs

• JavaScript based (AngularJS)

• only simple htp server needed

Page 29: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -
Page 30: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Q&A

Page 31: Centralized log management based on Logstash and …data.proidea.org.pl/atmosphere/2014/presentations/DariuszEliasz... · Centralized log management based on Logstash and Kibana -

Sources

Images:

htp://www.datalife7.com/2014/01/einsteins-secret-to-problem-solving-and.html

htp://www.formengifs.com/victorinox-swiss-army-swiss-champ-multtool-knife/

htp://www.slashgear.com/google-data-center-hd-photos-hit-where-the-internet-lives-gallery-17252451/