The WSO2 Business Activity Monitor - Optimizing Enterprise Data Management

With WSO2 Business Activity Monitor 2.4.1

Inosh GoonewardenaAssociate Technical Lead

Gokul BalakrishnanSoftware Engineer

Optimizing Enterprise Data

Management

June 2014

**

About the Presenters

● Inosh Goonewardena

Inosh Goonewardena is an associate technical lead and is a part of the WSO2 Business Activity Monitor team. He holds a bachelor’s degree in Engineering, specializing in Computer Science & Engineering from University of Moratuwa, Sri Lanka.

● Gokul Balakrishnan

Gokul is a software engineer in the Data and APIs Technology Group focusing on WSO2 Business Activity Monitor. He has a bachelor’s degree in Engineering from University of Moratuwa, Sri Lanka, and has provided consultation services on customer engagements.

**

● Global enterprise, founded in 2005 by acknowledged leaders in XML, web services technologies, standards and open source

● Provides only open source platform-as-a-service for private, public and hybrid cloud deployments

● All WSO2 products are 100% open source and released under the Apache License Version 2.0.

● Is an Active Member of OASIS, Cloud Security Alliance, OSGi Alliance, AMQP Working Group, OpenID Foundation and W3C.

● Driven by Innovation

○ Launched first open source API Management solution in 2012

○ Launched App Factory in Q2 2013

○ Launched Enterprise Store and first open source Mobile solution in Q4 2013

About WSO2

**

What we deliver

Agenda

● Introduction to Business Activity Monitoring

● WSO2 Business Activity Monitor

● WSO2 BAM Components

● WSO2 BAM Toolboxes

● WSO2 BAM High Availability Distributed Deployment

● Product Demo

“The aggregation, analysis, and presentation of real-time information about activities inside organizations and involving customers and partners” - Gartner

Business Activity Monitoring

● Capturing data

● Data storage

● What data to capture?

Aggregation

● Data operations

● Building KPIs

● Operate on large amounts of historic data or new data

● Building BI

Analysis

● Visualizing KPIs/BI

● Custom Dashboards

● Visualization tools

● Not just dashboards!

Presentation

● Fully-open source solution for aggregating and analyzing data and presenting information about business activities.

● Cloud-enabled, lightweight, developer-friendly and easy-to-deploy

● High performance data capture framework

● Elastically-scalable data analysis powered by Apache Hadoop

● Pre-built Data Agents for WSO2 products

WSO2 Business Activity Monitor

Architecture

Data Agents

● Compatible with CEP/BAM

● Get data across to BAM

− Service monitoring feature – WSO2 AS, DSS, ESB, API Manager

− Mediation monitoring feature – BAM Mediator for WSO2 ESB

− Custom data-agents

● Asynchronous & non-blocking

● Thrift for high performance message throughput

Data Agents

● Data agents are meant to be installed/plugged into the server to be monitored

● A Java SDK is provided

● Through Thrift, different languages can be supported

Custom Agents

Data Receiver

● Receives data and stores it in Cassandra

○ Scalable, big data repository

● Asynchronous & non-blocking

○ Combination of Cassandra, Thrift and the non-blocking nature results in extremely fast writes

● Shared with WSO2 CEP for real time analysis

● Supports Thrift & REST API

● Supports plugging in of different receiver types

Data Receiver

● Data is sent over using strongly typed Data Streams{ 'name':'phone.retail.shop', 'version':'1.0.0', 'nickName': 'Phone_Retail_Shop', 'description': 'Phone Sales', 'metaData':[ {'name':'clientType','type':'STRING'} ], 'payloadData':[ {'name':'brand','type':'STRING'}, {'name':'quantity','type':'INT'}, {'name':'total','type':'INT'}, {'name':'user','type':'STRING'} ]}

● Data Streams are versioned

○ Allows for easier analysis

Data Model

The Analyzer Engine

● Powered by Apache Hadoop with querying/managing through Apache Hive

● Data transfer to and from data sources is handled through custom storage handlers

● Parallel, distributed processing through the MapReduce programming model

The Analyzer Engine

● Runs on local Hadoop node or delegates to Hadoop cluster

o Scalable analytics

o Cluster can range from a couple of nodes to 1000s

● Analysis is carried out based on analytics scripts

● Scripts are based on an easy-to-learn, SQL-like query language

INSERT OVERWRITE TABLE UserTable SELECT userName, COUNT(DISTINCT orderID),SUM(quantity) FROM PhoneSalesTable WHERE version= "1.0.0" GROUP BY userName;

The Analyzer Engine

● Scripts can be scheduled○ Ex: once a minute, every Wednesday at 4:15 p.m., every

30th at 12 midnight

● Polyglot data architectures are supported○ Write summarised information to any kind of datastore

○ Custom written Hive JDBC handler used by default for output to relational DBs

The Analyzer Engine

The Presentation Layer

● Gadget Portal for the dashboard

● Gadget Generation Wizard – Tool for generating custom

gadgets for the dashboard

● Activity Dashboard for correlating activities

● Message Console

● Plug in your own report server/ dashboard server

The Presentation Layer

BAM Dashboards

The Presentation Layer

Gadget Portal

The Presentation Layer

Activity Dashboard

The Presentation Layer

Activity Dashboard (cont'd)

Message Console (HL7)

The Presentation Layer

Message Console (cont'd)

● BAM Toolboxes are installable and hot deployable artefacts used for deploying functionalities to a BAM server

○ Stream definitions○ Analytics scripts○ Dashboards

● Supports plugging in of one or more of the above features

● Toolboxes for monitoring and auditing most WSO2 products are available OOTB

● Toolboxes for custom scenarios can be created easily

BAM Toolboxes

● WSO2 BAM can be clustered and deployed in a distributed manner to enable high-availability, fail-over scenarios

○ Partially distributed deployment

■ Cassandra (storage) and Hadoop (analyzer) nodes are clustered

○ Fully distributed deployment

■ All components of BAM are clustered (data receiver, storage, analyzer and presentation)

● Hazelcast in-memory data grids are used for clustering implementation

HA Distributed BAM

Partially Distributed Deployment

Fully Distributed Deployment

Shared Events with CEP

WSO2 BAM Demonstration

In this demo:

ESB mediation statistics monitoring with the mediation data agent

BAM Mediation stats toolbox

BAM dashboard

Gadget gen tool with SQL query and visualisation

● UES integration for the presentation layer

● Enhanced alerting and reporting capabilities

● Enhanced toolbox support for WSO2 product stack

● Abstract data storage layer

Upcoming Features

?

**

Business Model

Contact us !