Advancing the Elastic Stack - It’s more than just log aggregation!
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Advancing the Elastic Stack -It’s more than just log aggregation!
Introduction
Mike ClarkeDevOps Engineer/SA
Mike KeithSenior Software Engineer
Agenda● Project/Problem Overview
○ Our environment and problem we were solving○ Initially to solve distributed log problem
● Elastic Stack Overview● Kibana and ElasticSearch Demo
Architecture Overview● Our Environment
○ Multiple Geographical Regions/Zones○ Ingest processing application○ Webservice application
■ Our webservice application logs tell us a lot about what is going on with customers sending us information.
○ Access logs for JBOSS○ Data archive application
JBossWebservice
JBossWebservice
JBossWebservice
JBossWebservice
JBossUI
JBossUI
JBossUI
JBossUI
Architecture Overview
RDBMS
NoSQL DB
Project / Problem● Log aggregation is hard● No historical reference, as logs age off● Obtaining stats was painful
○ Realistically when all your service stats are in your logs what do you do?● Cluster SSH only helps so much
Obtaining stats was painful ?!?!?!
cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc
host@me$: cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc
5234
host@me$: cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc...………
host@me$: cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc...………
host@me$: cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc
20host@me$: cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc1240
host@me$: cat log | grep "someword" | awk '{print $8}' | paste -sd+ | bc
650
Technical Overview● For the most part restricted to FOSS products● Needed to be easily obtainable● Available options
○ GrayLog○ Grafana○ Airbrake○ Splunk○ Elastic Stack
Elastic Stack (formerly ELK) Overview
Elasticsearch - Distributed, RESTful search and analytics engine
Logstash - Server-side data processing pipeline
Kibana - Powerful visualization UI
Beats - Single-purpose, lightweight data shippers
X-Pack - Powerful features which enhance the Elastic Stack
Elastic Stack (formerly ELK) Overview
Initial Solution - Log Aggregation● Single node servers● Installed Elastic Stack and began shipping all application server logs to a
centralized server.● Near Realtime● Raw log message transitioned into a fielded log message● Grok parsing (text pattern matching)● Filters etc.
Elasticsearch
Logstash
Filebeat
Filebeat
Filebeat
Filebeat
KibanaFilebeat
Filebeat
Filebeat
Filebeat
Architecture Overview
Filebeatfilebeat.prospectors:
- input_type: log
paths:- /data/logs/apache/*.log
fields:type: apache
fields_under_root: true
#----------------------------- Logstash output --------------------------------output.logstash:
hosts: ["localhost:5443"]bulk_max_size: 1024
Logstash - Input & Outputinput {
beats {port => 5443ssl => truessl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}}
output {elasticsearch {
hosts => ["localhost:9200"]index => "%{[@metadata][beat]}-%{[@metadata][type]}-%{+YYYY.MM.dd}"document_type => "%{type}"user => "elastic"password => "*******"
}}
Kibana - Discover
Kibana - Discover
Logstash - Filters
filter {grok {
match => { "message" => "%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code:int} %{NUMBER:bytes:int} "}
}mutate {
add_field => { "read_timestamp" => "%{@timestamp}" }}date {
match => [ "time", "dd/MMM/YYYY:H:m:s Z" ]remove_field => "time"
}}
Kibana
● We change from looking at who is talking to us, to what they are talking to us about.
○ We kept adding more to our logs just so we could see it in Kibana.○ Our data was already in Avro format, which made it easy to convert to JSON ○ Then we used the JSON Codec for logstash to input directly into elasticsearch.
● Considered Accumulo○ But there was just too much we had to build to get it to a usable state.
Evolution of the solution
Kibana Twitter Demo● Let’s take a look at some interesting things you can see in kibana● Counting very easily across different fields in your data (makes aggregating
and histograms very easy)● Data changes over time, sometimes you need to go back and update
something you already stored?○ State changes or updates of some kind to the original document.
Twitter Data DemoBasic twitter JSON:
{ screen_name, text, retweeted_status.user.screen_name, retweeted_status.retweet_count, retweeted_status.text, ... }
{ screen_name, text, retweeted_status.user.screen_name, retweeted_status.retweet_count, retweeted_status.text, ... }
Data Storage Elastic Stack Architecture
ElasticsearchData Node 1
Logstash Node 1
Kibana
Filebeat
Filebeat
Logstash Node 4
ElasticsearchData Node 20
... ElasticsearchClient Node
ElasticsearchMaster Node 1
ElasticsearchMaster Node 2
... ...
Conclusion & Takeaways● Low Barrier to Entry● Quickly Search Across Data● Horizontally Scalable● Easily Visualize Data
About Clarity Business Solutions● We are a team of Software and System Engineers● Customer focused and mission driven● For more about us, please visit: www.claritybizsol.com
● Follow us:
@claritybizsol
Related Documents
