HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved

Improved ODS Analysis with ELKOr how to quickly parse your log files

HighQSoft GmbH | www.highqsoft.de | 11.05.2016

1

Alexander Ziller / Constantin Badescu


2

Content

Traditional situationDefining an Solution Approach: What is ELK?Architecture of ELKIntegration with ODS systemsUse-Cases of ELK and ODS


3

Traditional situationLogs don’t come to you. You need to get to them.

RDP

Putty

Putty, VNC

Avalon CFS Avalon CFS

User

Avalon Apache HTTP Syslog …


Tomcat Importer Tomcat Importer

This might be just one system. You may want to cover multiple.

LinuxLinux

Windows


4

Defining an Solution ApproachWhat is ELK?

ElasticSearch – Logstash – Kibana

Elasticsearch: Indexer with scalability

Logstash: Logserver with parsing capabilities

Kibana: WEB Frontend for Logstash

Beats: Log-Forwarder for Logstash

FELK: Filebeat – Elasticsearch – Logstash – Kibana


5

Architecture of ELK



Tomcat Tomcat

Avalon … Avalon …

Beat Beat

Beat Beat

Logstash Logstash

ElasticSearch ElasticSearch

Kibana Kibana


6

Integration of ODS logsGathering of the log data

...20160426 13:43:01.293535 diff: 0 Thread 7f2dffea7700 Session 851 AoSession_GetContextByName()...


+Beat


+Beat


7

Integration with ODSParsing of the log data

20160426 13:43:01.293535 diff: 0 Thread 7f2dffea7700 Session 851 AoSession_GetContextByName()

Date Thread ID Session ID ODS Object+Method

Message

…+

Logstash Elasticsearch Kibana

…+



8

Integration with ODSProviding the log data

…+


…+



9

Integration with ODS

License AdministrationIntegration of multiple RLM servers

Availability of combined Reportlogs

Currently: Standard-Level

> License Utilization (License/time)

Analyzing the log data: Use-Cases of ELK and ODS …

+ Logstash Elasticsearch Kibana

…+



10

Use-Cases of ELK and ODS

Avalon Session CountingIntegration of multiple Avalon Server

Availability of ODS Logs

Currently: Debug-Level 3

>> Sessions by Avalon Instance?

>> Sessions by User(Group)?

>> Number of Log-Ins?



…+



11

Use-Cases of ELK and ODS

Further Use-Cases:Error searching

Message analysis

…



…+



12

Use-Cases of ELK and ODSAnalyzing the log data: Demonstration


13

What is the current statusInstalling and configuring setup

FELK stack available for Windows/LinuxPartial Repository support for LinuxConfiguration files: YAML, JSON

Parsing patterns depend on the log formattingCategorize logs for easy browsingFELK can be extended with “Shield” for SecurityAble to index multiple weeks/months of ODS logs


14

What is next for ELK and ODS?Expanding ELK to your needs

Simplification: Avalon as a BeatDirect integration into ELK, less configuration

Simplification: Avalon as a Log-Provider (with Avalon Service?)No more Logstash

Enhancement: Introduction to additional logging informationGeoIP in ODS Log?

Standardization: Standard ODS log patterns and parsersAny Log-Level SupportModelMapper Compatibility (own logging-rules)Avalon Suite 2017 Integration (integrated service)

Intelligent Analysis: Additional plug-ins for Kibana to allow business-log-icODS related information merged with log data


15

Thank you

We hope that the presentation helps you to manage your log-files!

HighQSoft GmbHSchloßborner Weg 6b61479 GlashüttenGermany

Alexander Ziller+49 6147 [email protected]

HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved

Documents