ELK Stack

ELK

Eberhard Wolff - @ewolff

Docker •  No true virtualization

•  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems

Eberhard Wolff - @ewolff

Docker File Systems •  Read only base images •  +read/write image

•  Can be stacked

Eberhard Wolff - @ewolff

Docker •  Linux only

•  So: Run it in VM

•  Controlled by Vagrant

Docker Demo

Eberhard Wolff - @ewolff

Communication Between Docker Container

•  Via ports

•  Via data volumes

Docker is a Component

System

Eberhard Wolff - @ewolff

Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L

Eberhard Wolff - @ewolff

Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines)

•  Total: 93 lines, 9 files •  Including one RegExp L

Eberhard Wolff - @ewolff

Vagrant VM

Containeruser-registration

8080

8081

Port8081

Containerelasticsearch

Containerlogstash

9200

9200

Port9200

9200

link

Containerkibana

Port8080

8080

8080

Browser

Volume/log

Demo ELK Stack

Eberhard Wolff - @ewolff

date time level

1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : [email protected] geloescht

pid thread class logmessage

Eberhard Wolff - @ewolff

Scaling Elasticsearch •  More servers •  Shards •  Easy to set up

•  Index per day or week or month •  Smaller dataset

Eberhard Wolff - @ewolff

Ideas about Logstash •  No parsing •  Send JSON

Eberhard Wolff - @ewolff

Break Logstash apart •  Logstash does parsing and

communication •  Not too sensible in a distributed

sytem •  Separate process per responsibility

Eberhard Wolff - @ewolff

Shipper •  Ship data from logging system •  Logstash •  Beaver

https://github.com/josegonzalez/beaver

•  Woodchuck https://github.com/danryan/woodchuck

Eberhard Wolff - @ewolff

Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast

•  ...and then Parse (Logstash)

Eberhard Wolff - @ewolff

Links •  https://github.com/ewolff/user-

registration

Eberhard Wolff - @ewolff

Thank You!!

[email protected] @ewolff