Elk with Openstack

ELK with Openstack

Arun prasath SJune 16, 2016

Challenges in log analysis

• Multiple services • Multiple servers behind load balancers• Searching the logs (cat, tail, sed, grep, awk)• Finding logs in particular time in multiple servers• Finding fields (Instance ID, name, IP address) in multiple servers and

correlating them• Log analysis , summary, visualization

ELK user operation demo

• Performing a normal search• Filtering based on time, fields• Viewing document data• Viewing field data statistics• Visualize data• Dashboards

ELK Architecture

Logstash forwarder

Servers

Logstash(Indexer)

Storage and search

Dashboard

Broker

• Temporary buffer between logstash agents and central server• Enhance performance by providing caching buffer for log events• Adds resiliency

• Incase the indexing fails, the events are held in queue instead of getting lost

Logstash

• Runs on JVM• Multiple Input / Multiple output• Centralize logs• Collect• Parse• Store / Forward• 3 components – Input, Filter and Output

Logstash conf - /etc/logstash/conf.d

input {}

filter{}

output{}

Logstash - Input

• Input plugin enables a specific source of events to be read by Logstash.

• Some examples of input• Beats• File• Stdin• Eventlog• More here

Logstash - Filter

• A filter plugin performs intermediary processing on an event. Filters are often applied conditionally depending on the characteristics of the event.

• Some examples are• Csv• Date• Grok• Json• More here

Logstash - Output

• An output plugin sends event data to a particular destination.• Some examples are

• Csv• redis• elasticsearch• File• Jira, Nagios, pagerduty• stdout• More here

Logstash - codec

• A codec plugin changes the data representation of an event• Some examples are

• Collectd - Reads events from the collectd binary protocol using UDP• Graphite - Reads graphite formatted lines• Json - Reads JSON formatted content, creating one event per element in a JSON array• Plain - Reads plaintext with no delimiting between events• rubydebug - Applies the Ruby Awesome Print library to Logstash events

• More here

Example - 1 (stdin input / rubydebug output)

input {stdin {}

}filter{}output{stdout { codec => rubydebug }}

Example – 2 (Basic filter on syslog with stdin input and rubydebug output)

input {stdin {}

}filter{grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]add_field => [ "received_from", "%{host}" ]

}syslog_pri { }date {

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]}

}output{stdout { codec => rubydebug }}

Example – 3 (Openstack filters with file input and multiple output)

input {file {

path => ['/var/log/nova/nova-api-metadata.log']tags => ['nova', 'oslofmt', 'novametaapi']type => "nova"

}…}filter{…else if "novaapi" in [tags] {

if [module] == "nova.osapi_compute.wsgi.server" {mutate {

gsub => ['logmessage',"\"",""]}grok {

match => { "logmessage" => "\[req\-%{NOTSPACE:requestid} %{NOTSPACE:user_id} %{NOTSPACE:tenant}\] %{NOTSPACE:requesterip} %{NOTSPACE:method} %{NOTSPACE:url_path} %{NOTSPACE:http_ver} status\: %{NUMBER:response} len\: %{NUMBER:bytes} time\: %{NUMBER:seconds}" }

add_field => ["api", "nova"]add_tag => ["apimetrics"]

}…}output {

elasticsearch { hosts => ["localhost:9200"] }stdout { codec => rubydebug }

}

Ref: https://github.com/bingoarunprasath/logstash-openstack-filters

Elasticsearch

• Searching and storing of logs• Built on Apache lucene (https://lucene.apache.org/core/)• Massively distributed• High availability• Developer friendly , RESTful API

Kibana

• Dashboard• Provides various options to search data• Creates bar charts, pie charts and various other data visualizations. • Can create custom dashboard and add the saved visualizations. • Simple data export

Installation notes

• Install Java• Install elasticsearch and Kibana

• Install nginx for reverse proxy and basic AUTH• Install logstash, generate SSH certificates

• Configure one output to elasticsearch• Load Kibana dashboard• Setup filebeat / logstash in agent machines and output to logstash• Ansible role - https://galaxy.ansible.com/bingoarunprasath/elk/• Filters - https://github.com/bingoarunprasath/logstash-openstack-

filters