Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan.

Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana

Onno de Haan

Hier uw naam

Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana

Onno de Haan

Agenda

• About VanMeijel

• Why we need insight in Metacom (our ERP-system)• Requirements for monitoring tool• Developing the tool• Final solution

• Demo !

Please feel free to ask questions at any time

About VanMeijel

• Founded in 1987• Based in The Netherlands, Emmeloord• Satellite office in Belgium, Sint-Niklaas• Around 110 employees• Software is targeted at construction companies• 300+ customers, 8000 - 9000 users

Metacom – Our ERP system

• Contains modules for

- Estimation - Reporting - Planning - Invoicing - Cost Control - Procurement - Relation Management - Resource Planning - and many more

Metacom – Our ERP system

• Build in OpenEdge, currently version 11.5

- Millions of code lines - Around 20 – 25 years of age - Lots of different techniques (DB-access on client, client-server, procedural, OO) - Deployed via Round Table, recently moved to Subversion / Env / Jenkins

• About VanMeijel

• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution

• Demo

Please feel free to ask questions at any time

• Application usage is changing

- People are working mobile - People want / expect Apps - Customers don’t want big server parks: web interface

• Changing world• Integrations • Maintenance• Different license models• Move to webinterface

Why we need insight in Metacom

• Customers are integrating 3th party applications

• One webservice often replaces many “real” users

• Integrations weaken our license model

• Changing world• Integrations • Maintenance• Different license models• Move to webinterface

Why we need insight in Metacom

• What methods are called

• How often are they called

• How many business objects are being created / edited per method

• How many systems are connecting to Metacom

• Changing world• Integrations • Maintenance• Different license models• Move to webinterface

Why we need insight in Metacom

• System is complex

• Around 3000 – 4000 windows/dialogs

• Contains lots and lots of settings

• What is the performance of our webservice methods

• Changing world• Integrations • Maintenance• Different license models• Move to webinterface

Why we need insight in Metacom

• In the future we need to move to “pay for usage”

• How many business documents are created - For example how many invoices

• Are business documents updated frequently or not?

• How many reports are being generated

• Changing world• Maintenance• Integrations• Different license models• Move to webinterface

Why we need insight in Metacom

• How many users are active?

• On what time of the day / week?

• How about evenings, weekends?

• Changing world• Maintenance• Integrations• Different license models• Move to webinterface

Why we need insight in Metacom

• What functions / modules are used

• What is not used anymore

• Top 100 of most used functions

• Changing world• Maintenance• Integrations• Different license models• Move to webinterface

Why we need insight in Metacom

• About VanMeijel

• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution

• Demo

Please feel free to ask questions at any time

Requirements monitoring tool

• Web based

• Real time

• Easy to analyze

• Flexible, schemaless database

• Light-weight; JSON

• About VanMeijel

• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution

• Demo

Please feel free to ask questions at any time

• Events are send to our AppServer API (written in OpenEdge)

- Client - Triggers - Business Logic - Webservices

• AppServer API sends events to webserver

• Webserver stores events and hosts a viewer

• Architecture• First try, january 2014• Different license models

Developing the tool

• NodeJS as webserver

• MongoDB as database

• Visualization was a problem

• Architecture• First try, january 2014• Second try, march 2014

Developing the tool

• We replaced MongoDB with ElasticSearch

• Visualization is super easy with Kibana

• Migration from MongoDB to ElasticSearch was difficult

• Architecture• First try, january 2014• Second try, march 2014

Developing the tool

• About VanMeijel

• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution

• Demo

Please feel free to ask questions at any time

• Clients push events to AppServer

• AppServer start HTTP-session to REST-service in cloud

• Fires JSON event, simplified example: { “customerid”:”vma”, “datetime”:”2015-11-05T14:36:54”, “appid”: “mtc”, “appversion”: “9.2.2.1” “type”:”ui”, “function”: “wcustomers” }

• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation

Final solution

• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation

Final solution

var restify = require('restify');var elasticsearch = require('elasticsearch');

var server = restify.createServer( { name : "stats" }); server.listen(‘1234’,’127.0.0.1’, function()

{ console.log('%s listening at %s ', server.name , server.url);});

server.use(restify.queryParser());server.use(restify.bodyParser());server.use(restify.CORS());var client = new elasticsearch.Client({ host: 'localhost:9200' });

server.post({path : '/stats', version: '0.0.1'}, postNewMtcStat);

function postNewMtcStat(req , res , next){ var body = req.body; res.setHeader('Access-Control-Allow-Origin','*'); client.create({index: 'usage', type: 'usage', body: req.body }, function(err , success){ if(success){ res.send(201 , 'saved'); return next(); }else{ return next(err); } });}

• OpenSource, Apache License

• Based on Lucene

• Distributed database, very scalable

• Near real-time search

• Document oriented, schema changes not a problem!

• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation

Final solution

• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation

Final solution

• Cool

• Flexible

• Fast (but not on this laptop )

• Very easy to use

• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation

Final solution

DEMO

QUESTIONS

Thank you for your